docs: RFC — staged-hybrid incremental backup (Phase 2 = size-routing/append) #7
+1,344
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Jul 8, 2026
…uting/append) Rewrites docs/RFC_incremental_v2.md from the leveled-only RFC into the umbrella staged-hybrid RFC that matches what actually shipped and what is being built next: - Phase 1 (exclusion): DONE/merged; recap + pointer to docs/EXCLUSION_SPEC.md. - Phase 2 (size-routing/APPEND): the new build target, full design — solo objects >= T_small (256 MiB, floor 100 MB), content-addressed shards (hash(relpath) mod K, K frozen at baseline), skip-unchanged (the append win, doubles as resume), per-target AGGREGATE reclaim (verify_target — REQUIRED same slice or every routed target false-DRIFTs), reship-monitor (the Phase-3 trigger), local-mode only (refuses --globus). 4 reviewable slices with tests. - Phase 3 (leveled L0/L1/L2): DEFERRED-DORMANT banner; the prior leveled RFC content (Path A/B, restore, chain-aware reclaim, Negishi spike) preserved intact as the Phase-3 reference. - Appendix: the PR #8 primitives (badge_of/make_blonde/enumerate_source_catalog/ floor_s) reused across phases, with current line refs. Design driver = the 2026-07-08 source_files measurement: standoff imagery is stable (exclusion covers it); the data modalities GROW by appending -> append is the dominant Phase-2 win. Review gate before any engine code. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jarrod S Doucette (jdoucett)
force-pushed
the
docs/incremental-rfc-v2
branch
from
87238b4 to
34ad875
Compare
Jarrod S Doucette (jdoucett)
changed the title
…uting/append) Rewrites docs/RFC_incremental_v2.md from the leveled-only RFC into the umbrella staged-hybrid RFC that matches what actually shipped and what is being built next: - Phase 1 (exclusion): DONE/merged; recap + pointer to docs/EXCLUSION_SPEC.md. - Phase 2 (size-routing/APPEND): the new build target, full design — solo objects >= T_small (256 MiB, floor 100 MB), content-addressed shards (hash(relpath) mod K, K frozen at baseline), skip-unchanged (the append win, doubles as resume), per-target AGGREGATE reclaim (verify_target — REQUIRED same slice or every routed target false-DRIFTs), reship-monitor (the Phase-3 trigger), local-mode only (refuses --globus). 4 reviewable slices with tests. - Phase 3 (leveled L0/L1/L2): DEFERRED-DORMANT banner; the prior leveled RFC content (Path A/B, restore, chain-aware reclaim, Negishi spike) preserved intact as the Phase-3 reference. - Appendix: the PR #8 primitives (badge_of/make_blonde/enumerate_source_catalog/ floor_s) reused across phases, with current line refs. Design driver = the 2026-07-08 source_files measurement: standoff imagery is stable (exclusion covers it); the data modalities GROW by appending -> append is the dominant Phase-2 win. Review gate before any engine code. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jarrod S Doucette (jdoucett)
force-pushed
the
docs/incremental-rfc-v2
branch
from
34ad875 to
b617c58
Compare
…se-3 deferral) Full restore now §2.9 (object-model reassembly: disjoint union, so simpler than the deferred Phase-3 leveled restore — no overlay/deletion replay/chain fold) and slice 5 of the build order (§2.10). --slot selective restore subsumes single-file restore. Restore is what makes reclaim's aggregate --delete trustworthy (prove round-trip first). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… 1 DONE
- shard fingerprint = {members, byte_sum, digest} (shake_128 over sorted
(arcname, floor_s(mtime), size) triples) — closes the same-count/same-bytes
reshuffle hole the plain mtime_sum draft had.
- solo slot = 64-bit hex(shake_128(arcname)[:8]) (was b32[:4]); route() guards
the residual collision.
- §2.10 slice 1 marked [DONE] with the real helper names + test coverage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tral)
Adds the pure, additive size-routing helpers to archive.py — nothing wires them
into the archive or reclaim flow yet (behavior-neutral; full suite 136 -> 172 green):
- route(catalog, t_small, shard_count) -> [{slot, kind, arcnames}] partition:
files >= T_small become solo objects; smaller files bucket into content-addressed
shards k = shake_128(arcname) mod K (no repack cascade -> cheap APPEND).
- shard_of / solo_slot / shard_slot / derive_shard_count (K = ceil(small_bytes /
shard_target), frozen per badge at baseline).
- object_fingerprint (solo {mtime_s,size}; shard {members,byte_sum,digest}) +
object_unchanged (the skip/append signal, also the resume mechanism).
- clamp_t_small (100 MB floor), budget_warnings (warn-only max_solo/max_objects/
oversize-shard), vault_dir_for/manifest_path/load_manifest/write_manifest
(atomic tmp+os.replace, best-effort; _vault child dodges reclaim's glob).
- make_zip_files gains an optional only_arcnames filter (no caller yet) so a later
slice can ship exactly one routed object; keys stay arcname-relpath (invariant #2).
Tests: tests/test_routing_primitives.py (36 cases) — boundary partition, shard
stability, K frozen, fingerprint change detection incl. reshuffle, skip vs
shard-append, only_arcnames subset with matching checksum keys, manifest round-trip.
See docs/RFC_incremental_v2.md §2 (slice 1).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…scratch Jarrod review calls: - K = derived-and-frozen (confirmed; already in derive_shard_count). - Manifest location: it is durable cross-run state reclaim must find, so it MUST live on shared+durable storage (the deployment Depot logs dir that log_dir already points at), NEVER home (the current default trap) or scratch (purged -> reclaim can't reconstruct the aggregate -> every routed target false-DRIFTs). Add a slice-2 durability guard in load_config; tmp_dir (staging zips) stays in scratch. - --globus: reframed. Real invariant = manifest on a shared log_dir readable by both the run and reclaim, which Negishi (the only launch host) satisfies either way; the split only happens off-box, which we don't do. Globus is off on Negishi now, so size_routing stays local-only as a cheap guard (untested stateful path), not a deep limitation — allowable under --globus later with the durability guard in place. §2.4 manifest-location invariant + guard; §2.8 --globus reframe; §2.10 slice-2 guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign in
to join this conversation on GitHub.
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refreshed: the staged-hybrid RFC (review gate for Phase 2)
This rewrites
docs/RFC_incremental_v2.mdfrom the original leveled-only RFC into theumbrella staged-hybrid design — matching what actually shipped (Phase 1) and what is
being built next (Phase 2). This PR is a review gate: no engine code lands until the
Phase-2 design here is approved.
What changed vs the old RFC
[DONE]: recap + pointer to the authoritativedocs/EXCLUSION_SPEC.md(merged, PRs feat(incremental): catalog + BLONDE primitives (PR 1) #8/docs: VOCABULARY.md — shared terms aligned to Bastion #10–feat(exclusion): resume-under-exclusion + post-hoc re-verifier (finish Phase 1) #17, suite 136 green).[THIS RFC — READY TO BUILD]: the substantive new content.[DEFERRED — DORMANT]: the entire prior leveled RFC is preserved intact under a DEFERRED banner (§3) as the Phase-3 reference — Path A/B recommendation, restore, chain-aware reclaim, the Negishi spike. Nothing was lost.badge_of/make_blonde/enumerate_source_catalog/floor_s) reused across phases, with currentarchive.pyline refs.Phase 2 in one screen
Per target, the post-exclusion survivors are partitioned into independent Fortress
objects, each shipped through the unchanged
make_zip_files → send_to_fortressverify chain:
T_small(256 MiB, floor 100 MB); its own object; skips if unchanged.T_smallbucketed byk = hash(relpath) mod K(K frozen at baseline); whole-shard re-ship on any member change. No repack cascade.verify_target) — REQUIRED in the same slice as the first routed ship, or every routed target perpetually false-DRIFTs (reclaim dedups to the newest log = one object). Sums all objects; requires every object tar present; INV-E exclusion re-gate on the aggregate.[RESHIP-WATCH]email when shard reship ratio > ~25%: the uniform-medium-churn signature and the only trigger to activate Phase 3.--slotselective restore. It's what makes aggregate--deletetrustworthy.--globus(routing + manifest are on the launching host).{log_dir}/_vault/{badge}.manifest.json(dodges reclaim's non-recursive glob).Design driver (settled, do not re-litigate)
The 2026-07-08
source_filesmeasurement (809 logs, Negishi): standoff imagery isstable (Phase-1 exclusion covers it); the data modalities GROW by appending
(
modified≈0) — so append is the dominant Phase-2 win, which content-addressedshards + solos deliver for free. Corrects the earlier
du-based "omics re-analyzeseverything" read.
Build order (after approval) — same cadence as exclusion
only_arcnamesonmake_zip_files.--globusrefusal +--freshre-baseline.verify_target, required with slice 2's first routed ship).restore.py) — the capstone.Tests ship with every slice;
python3 -m unittest discover -s testsbefore merging.🤖 Generated with Claude Code