feat(routing): Phase-2 archive wiring — the append engine (slice 2) #19
+633
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Wires the slice-1 primitives into an opt-in size-routing archive path. Default-off,
so an unset config is byte-for-byte the whole-target behavior (full suite 172 -> 192).
Config (load_config): size_routing (bool), t_small (int, floored 100 MB), shard_count
(null=derive), shard_target. DURABILITY GUARD (RFC §2.4): when size_routing is on,
refuse a log_dir under $HOME / scratch / the ~/globus_archive_tmp default — the
per-target manifest at {log_dir}/_vault/ must be shared+durable or reclaim can't find
it (routing_log_dir_ok). config.example.json annotated.
Engine:
- enumerate_source_catalog_excluded: one walk -> (survivor catalog, exclusion report),
the routing data source (exclusion applied once, before partitioning).
- route_and_ship (LOCAL): partition survivors -> objects; skip-unchanged (the append
win + resume); ship changed/new objects via the SAME make_zip_files ->
send_to_fortress chain (only_arcnames per object, exclude_spec=None); K frozen from
the manifest after baseline; drop deleted slots; manifest written atomically AFTER
EACH shipped object (crash-safe resume). object_project_name = per-object stem
(invariant #4).
- __main__: refuse size_routing + --globus up front (RFC §2.8); size-routing branch
replaces whole-target Step 1/2 and emails a [SUCCESS] summary.
Tests: tests/test_routing_wiring.py (20) — durability guard, config validation,
survivor enumerator (incl. exclusion drop), baseline/append/solo-change/shard-change/
delete/K-frozen/--fresh/resume-after-crash, exclusion recorded in manifest.
See docs/RFC_incremental_v2.md §2.5 (slice 2). Aggregate reclaim (slice 3) is REQUIRED
before a routed target is reclaimed, else verify() false-DRIFTs each object log.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign in
to join this conversation on GitHub.
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Phase 2, Slice 2 — archive wiring (the append engine)
Wires the slice-1 primitives into an opt-in
size_routingarchive path. Default-off,so an unset config is byte-for-byte the current whole-target behavior. Full suite
172 → 192 green.
Config (
load_config)size_routing(bool),t_small(int, floored to 100 MB),shard_count(
null= derive K at baseline),shard_target. All validated;config.example.jsonannotated.
size_routingis on, refuse alog_dirunder$HOME, a scratch path, or the~/globus_archive_tmpdefault — the per-target manifestat
{log_dir}/_vault/must be shared + durable or reclaim can't find it (a purgedscratch manifest → every routed target false-DRIFTs).
routing_log_dir_ok().Engine
enumerate_source_catalog_excluded— one walk →(survivor catalog, exclusion report); exclusion is applied once, before partitioning (routing partitions thesurvivors).
route_and_ship(LOCAL) — partition survivors → objects; skip-unchanged (theappend win, also the resume mechanism); ship only changed/new objects through the
unchanged
make_zip_files → send_to_fortresschain (only_arcnamesper object,exclude_spec=None); K frozen from the manifest after baseline; drop deleted slots;manifest written atomically after each shipped object (crash-safe resume).
__main__— refusesize_routing+--globusup front (RFC §2.8); the size-routingbranch replaces the whole-target Step 1/2 and emails a
[SUCCESS]per-object summary.object_project_name— per-object log/zip/tar stem (invariant RFC: incremental, split-source Fortress archiving (design only) #4).Tests —
tests/test_routing_wiring.py(20)Durability guard (home/default/scratch rejected, Depot accepted), config validation,
survivor enumerator incl. exclusion drop, and
route_and_ship: baseline ships all •append ships only the new solo • solo change re-ships just that solo • shard member
change re-ships the whole shard • deleted big file drops its slot • K frozen across runs •
--freshre-ships all • resume after a mid-run crash (persisted object skips onre-run) • exclusion recorded in the manifest.
python3 -m unittest discover -s tests→Ran 192 tests … OK.Important sequencing
Slice 3 (aggregate reclaim,
verify_target) is REQUIRED before any routed target isreclaimed — reclaim dedups to the newest log = one object, so today a routed target
would false-DRIFT. Do not enable
size_routingon a production config until #18 + this +slice 3 are all in. (Full restore is the slice-5 capstone.)
🤖 Generated with Claude Code