feat(incremental): level-decision + manifest engine (PR 2)

[Jarrod S Doucette (jdoucett)]

PR 2 of the leveled-incremental plan (docs/RFC_incremental_v2.md §2–3), stacked on #8. The engine core — the level-decision logic + manifest + per-level catalogs — all LOCAL and fully unit-tested.

Gated + unwired by design. A new config key incremental (default false) is added and validated, but nothing in __main__ calls the engine yet, so the existing non-incremental archive path is byte-for-byte unchanged. The __main__ wiring (decide_level call site, NO-OP/deletions exit paths, level-aware resume guard, flock, --globus refusal) lands as PR 2b — it touches the live archive path and gets its own review + a Negishi staging run before incremental is enabled anywhere.

Adds to archive.py: classify_delta, decide_level, effective_catalog, load/update_manifest, write/read_level_catalog, make_zip_files(only_arcnames=…), the incremental config gate.

Hardened by an adversarial review workflow (3 dimensions → verify pass) before opening:

• per-level catalogs moved into _vault/ (were written to log_dir → would show as spurious FAILED entries in reclaim's --scan);
• update_manifest now preserves a corrupt manifest aside and refuses instead of silently truncating the chain; tmp+fsync+os.replace;
• decide_level re-baselines to a full L0 instead of crashing when the prior catalog is unreadable;
• removed the dead rebaseline branch (byte-sum is redundant at per-file granularity) and documented the real same-size+same-mtime blind spot honestly.

Tests: tests/test_incremental_engine.py. Full suite 87 passing (was 54).

🤖 Generated with Claude Code

[Jarrod S Doucette (jdoucett)]

PARKED / deferred-dormant — not abandoned.

PR #8 (catalog + BLONDE primitives) has merged to main, which auto-closed this PR (its base branch was deleted on merge). That's fine: this is the right state for now.

Why parked. The settled direction is size-routing (solo + content-addressed shards) + identity/coverage exclusion of git-versioned metadata, which kills the measured re-ship waste for the data shapes we actually have (live measurements: standoff churn = git-versioned metadata only; the bulk data is stable → archive-once-skip-forever holds). The leveled L0/L1/L2 engine in this PR is only the cheaper fix for a measured uniform-medium-churn shard — every shard changing every run — which the live data did not show.

Code preserved. The full leveled-engine implementation lives on branch feat/incremental-level-engine (tip c8db396), rebuilt on top of the now-merged primitives. It is gated behind the opt-in incremental config (default off) and was hardened by an adversarial review pass.

Revival trigger. Reopen (new PR from the branch onto main) only if a real asset is measured with uniform-medium-churn across shards — the one case whole-shard-reship can't do cheaply. The reship-monitor (size-routing) is the signal: if a run reships >25% of shards repeatedly, revisit this engine.