feat(routing): Phase-2 archive wiring — the append engine (slice 2) #19
+817
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Wires the slice-1 primitives into an opt-in size-routing archive path. Default-off,
so an unset config is byte-for-byte the whole-target behavior (full suite 172 -> 192).
Config (load_config): size_routing (bool), t_small (int, floored 100 MB), shard_count
(null=derive), shard_target. DURABILITY GUARD (RFC §2.4): when size_routing is on,
refuse a log_dir under $HOME / scratch / the ~/globus_archive_tmp default — the
per-target manifest at {log_dir}/_vault/ must be shared+durable or reclaim can't find
it (routing_log_dir_ok). config.example.json annotated.
Engine:
- enumerate_source_catalog_excluded: one walk -> (survivor catalog, exclusion report),
the routing data source (exclusion applied once, before partitioning).
- route_and_ship (LOCAL): partition survivors -> objects; skip-unchanged (the append
win + resume); ship changed/new objects via the SAME make_zip_files ->
send_to_fortress chain (only_arcnames per object, exclude_spec=None); K frozen from
the manifest after baseline; drop deleted slots; manifest written atomically AFTER
EACH shipped object (crash-safe resume). object_project_name = per-object stem
(invariant #4).
- __main__: refuse size_routing + --globus up front (RFC §2.8); size-routing branch
replaces whole-target Step 1/2 and emails a [SUCCESS] summary.
Tests: tests/test_routing_wiring.py (20) — durability guard, config validation,
survivor enumerator (incl. exclusion drop), baseline/append/solo-change/shard-change/
delete/K-frozen/--fresh/resume-after-crash, exclusion recorded in manifest.
See docs/RFC_incremental_v2.md §2.5 (slice 2). Aggregate reclaim (slice 3) is REQUIRED
before a routed target is reclaimed, else verify() false-DRIFTs each object log.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…zip)
Reworks the routed object transport per Jarrod's decision, verified live on Negishi
(HPSS 10.3.0.p3): tar each object's members DIRECTLY from source (drop the zip for
routed objects), choosing the tool by max member size.
- HTAR_MEMBER_LIMIT = 2^36 (64 GiB) — the empirically-locked htar per-file cutoff
(2^36-1 accepted, 2^36 refused; htar fail-fasts on oversize with no partial on tape).
- choose_transport(arcnames, catalog): "htar" iff every member < 2^36 else "htar_large".
Pure + tested (boundary, max-member, missing-member).
- ship_object(): LOCAL transport — htar -P -cvf -L memberlist (indexed .idx, random-
access restore) or htar_large -cvf <paths> (>=64 GiB solo); per-file MD5 from source;
round-trip verify every member off tape (htar -x for indexed, hsi get+tar xf for large);
writes the object's {stem}_{ts}.txt/.json run-log (routed_object marker for slice-3
reclaim). Command strings all verified on Negishi.
- route_and_ship manifest entry now records transport + idx (dropped zip_checksum);
__main__ routed ship_one calls ship_object. Whole-target zip->htar_large path untouched.
Tests 192 -> 196 (choose_transport + manifest-records-transport). ship_object itself is
tape-touching (not unit-testable locally) — pending Negishi end-to-end validation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign in
to join this conversation on GitHub.
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Phase 2, Slice 2 — archive wiring (the append engine) + size-routed transport
Opt-in
size_routingarchive path (default-off → an unset config is byte-for-byte thewhole-target behavior). Stacked on #18 (base =
feat/routing-primitives); merge #18first, then I'll rebase. Full suite 136 → 196 green.
The append engine (
route_and_ship, LOCAL)size_routing/t_small/shard_count/shard_target+ durability guard(§2.4: refuse a
log_dirunder$HOME/scratch/the home default — the manifest must beshared+durable).
config.example.jsonannotated.resume mechanism) → ship only changed/new objects. K frozen from the manifest after
baseline; deleted slots dropped; manifest written atomically after each shipped object.
size_routing+--globus;--freshre-baseline.Transport (
ship_object) — Jarrod's rule, verified live on NegishiEach object is tarred directly from source (no zip), tool chosen by max member size:
htarwhen every member < 2³⁶ (64 GiB) →{stem}.tar+.idx: fast listing +random-access single-member restore. Covers all shards + most solos.
htar_largewhen a member ≥ 2³⁶ → plain tar (only a huge single-file solo).Cutoff
HTAR_MEMBER_LIMIT = 2³⁶locked empirically (2³⁶−1 accepted, 2³⁶ refused; htarfail-fasts on oversize with no partial on tape).
ship_objectcomputes per-file MD5 fromsource, round-trip-verifies every member off tape (
htar -xfor indexed,hsi get+tar xffor large), and writes each object's own run-log (routed_objectmarker forslice-3 reclaim). Manifest records
transport+idx(nozip_checksum— tars of source,not zips). The whole-target Phase-1 path keeps
zip → htar_largeuntouched.Tests (
test_routing_primitives.py36 +test_routing_wiring.py24)Durability guard, config validation, survivor enumerator (incl. exclusion drop),
choose_transport(64 GiB boundary / max-member / missing), baseline / append / solo-change/ shard-change / delete / K-frozen /
--fresh/ resume-after-crash, manifest recordstransport.
ship_objectis tape-touching (not unit-testable locally).Negishi end-to-end validation (real Fortress, HPSS 10.3.0.p3 — all artifacts cleaned up)
Scratch worktree off the prod clone (prod untouched). 1 solo (120 MB) + 5 small files, K=2:
htar, each with.idx; round-trip MD5 verified;manifest on the durable Depot
log_dir;hsi rm -Rcleanup.Sequencing
Slice 3 (aggregate reclaim,
verify_target) is REQUIRED before any routed target isreclaimed (reclaim dedups to the newest log = one object → would false-DRIFT). Slice 3 is
transport-agnostic (reads the manifest). Full restore is the slice-5 capstone (uses
htar -xrandom access for indexed objects).
🤖 Generated with Claude Code