docs: RFC — staged-hybrid incremental backup (Phase 2 = size-routing/append)

[Jarrod S Doucette (jdoucett)]

Refreshed: the staged-hybrid RFC (review gate for Phase 2)

This rewrites docs/RFC_incremental_v2.md from the original leveled-only RFC into the
umbrella staged-hybrid design — matching what actually shipped (Phase 1) and what is
being built next (Phase 2). This PR is a review gate: no engine code lands until the
Phase-2 design here is approved.

What changed vs the old RFC

• Phase 1 — Exclusion [DONE]: recap + pointer to the authoritative docs/EXCLUSION_SPEC.md (merged, PRs feat(incremental): catalog + BLONDE primitives (PR 1) #8/docs: VOCABULARY.md — shared terms aligned to Bastion #10–feat(exclusion): resume-under-exclusion + post-hoc re-verifier (finish Phase 1) #17, suite 136 green).
• Phase 2 — Size-routing / Append [THIS RFC — READY TO BUILD]: the substantive new content.
• Phase 3 — Leveled L0/L1/L2 [DEFERRED — DORMANT]: the entire prior leveled RFC is preserved intact under a DEFERRED banner (§3) as the Phase-3 reference — Path A/B recommendation, restore, chain-aware reclaim, the Negishi spike. Nothing was lost.
• Appendix: the PR feat(incremental): catalog + BLONDE primitives (PR 1) #8 primitives (badge_of / make_blonde / enumerate_source_catalog / floor_s) reused across phases, with current archive.py line refs.

Phase 2 in one screen

Per target, the post-exclusion survivors are partitioned into independent Fortress
objects, each shipped through the unchanged make_zip_files → send_to_fortress
verify chain:

• Solo — one file ≥ T_small (256 MiB, floor 100 MB); its own object; skips if unchanged.
• Shard — files < T_small bucketed by k = hash(relpath) mod K (K frozen at baseline); whole-shard re-ship on any member change. No repack cascade.
• The append win: a growing target's new big files become new solos; every existing object skips — zero re-read of already-archived bytes. Skip-unchanged also doubles as resume (a half-shipped object simply isn't in the manifest yet).
• Per-target AGGREGATE reclaim (verify_target) — REQUIRED in the same slice as the first routed ship, or every routed target perpetually false-DRIFTs (reclaim dedups to the newest log = one object). Sums all objects; requires every object tar present; INV-E exclusion re-gate on the aggregate.
• Reship-monitor — [RESHIP-WATCH] email when shard reship ratio > ~25%: the uniform-medium-churn signature and the only trigger to activate Phase 3.
• Full restore (§2.9) is the capstone slice — object-model reassembly (disjoint union, so simpler than Phase-3 leveled restore: no overlay/deletion replay), completeness gate + per-object MD5 verify + mandatory reconciliation, --slot selective restore. It's what makes aggregate --delete trustworthy.
• Local-mode only — refuses --globus (routing + manifest are on the launching host).
• Manifest per target at {log_dir}/_vault/{badge}.manifest.json (dodges reclaim's non-recursive glob).

Design driver (settled, do not re-litigate)

The 2026-07-08 source_files measurement (809 logs, Negishi): standoff imagery is
stable (Phase-1 exclusion covers it); the data modalities GROW by appending
(modified≈0) — so append is the dominant Phase-2 win, which content-addressed
shards + solos deliver for free. Corrects the earlier du-based "omics re-analyzes
everything" read.

Build order (after approval) — same cadence as exclusion

1. Routing primitives (behavior-neutral) + only_arcnames on make_zip_files.
2. Archive wiring (the append engine) + config keys + --globus refusal + --fresh re-baseline.
3. Aggregate reclaim (verify_target, required with slice 2's first routed ship).
4. Reship-monitor.
5. Full restore (restore.py) — the capstone.

Tests ship with every slice; python3 -m unittest discover -s tests before merging.

🤖 Generated with Claude Code