feat(routing): land slice 2 in main (re-target — stacked merge missed it) #20
+817
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Wires the slice-1 primitives into an opt-in size-routing archive path. Default-off,
so an unset config is byte-for-byte the whole-target behavior (full suite 172 -> 192).
Config (load_config): size_routing (bool), t_small (int, floored 100 MB), shard_count
(null=derive), shard_target. DURABILITY GUARD (RFC §2.4): when size_routing is on,
refuse a log_dir under $HOME / scratch / the ~/globus_archive_tmp default — the
per-target manifest at {log_dir}/_vault/ must be shared+durable or reclaim can't find
it (routing_log_dir_ok). config.example.json annotated.
Engine:
- enumerate_source_catalog_excluded: one walk -> (survivor catalog, exclusion report),
the routing data source (exclusion applied once, before partitioning).
- route_and_ship (LOCAL): partition survivors -> objects; skip-unchanged (the append
win + resume); ship changed/new objects via the SAME make_zip_files ->
send_to_fortress chain (only_arcnames per object, exclude_spec=None); K frozen from
the manifest after baseline; drop deleted slots; manifest written atomically AFTER
EACH shipped object (crash-safe resume). object_project_name = per-object stem
(invariant #4).
- __main__: refuse size_routing + --globus up front (RFC §2.8); size-routing branch
replaces whole-target Step 1/2 and emails a [SUCCESS] summary.
Tests: tests/test_routing_wiring.py (20) — durability guard, config validation,
survivor enumerator (incl. exclusion drop), baseline/append/solo-change/shard-change/
delete/K-frozen/--fresh/resume-after-crash, exclusion recorded in manifest.
See docs/RFC_incremental_v2.md §2.5 (slice 2). Aggregate reclaim (slice 3) is REQUIRED
before a routed target is reclaimed, else verify() false-DRIFTs each object log.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…zip)
Reworks the routed object transport per Jarrod's decision, verified live on Negishi
(HPSS 10.3.0.p3): tar each object's members DIRECTLY from source (drop the zip for
routed objects), choosing the tool by max member size.
- HTAR_MEMBER_LIMIT = 2^36 (64 GiB) — the empirically-locked htar per-file cutoff
(2^36-1 accepted, 2^36 refused; htar fail-fasts on oversize with no partial on tape).
- choose_transport(arcnames, catalog): "htar" iff every member < 2^36 else "htar_large".
Pure + tested (boundary, max-member, missing-member).
- ship_object(): LOCAL transport — htar -P -cvf -L memberlist (indexed .idx, random-
access restore) or htar_large -cvf <paths> (>=64 GiB solo); per-file MD5 from source;
round-trip verify every member off tape (htar -x for indexed, hsi get+tar xf for large);
writes the object's {stem}_{ts}.txt/.json run-log (routed_object marker for slice-3
reclaim). Command strings all verified on Negishi.
- route_and_ship manifest entry now records transport + idx (dropped zip_checksum);
__main__ routed ship_one calls ship_object. Whole-target zip->htar_large path untouched.
Tests 192 -> 196 (choose_transport + manifest-records-transport). ship_object itself is
tape-touching (not unit-testable locally) — pending Negishi end-to-end validation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign in
to join this conversation on GitHub.
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Re-land slice 2 into
main(stacked-merge dropped it)#19 was merged into its stacked base
feat/routing-primitives, notmain— and sincefeat/routing-primitiveshad already merged tomainvia #18 before #19 merged, slice 2'scode never reached
main. Confirmed: none ofroute_and_ship/ship_object/choose_transportare inorigin/main. (Same class of drop as the #13→#14 incident.)This PR re-targets the same, already-reviewed
feat/routing-wiringhead atmainso thecontent actually lands. Diff = exactly the 3 slice-2 files (merge-base is slice 1, which is
already in
main):archive.py—size_routingconfig + durability guard,route_and_ship,ship_object(size-routed
htar/htar_largeof source, per-file MD5 + round-trip verify),choose_transport,enumerate_source_catalog_excluded.config.example.json— routing keys.tests/test_routing_wiring.py— 24 tests.No code changes vs what you reviewed in #19 — it's the identical commits (
c195fbf,2d482fe), just merging to the right base. Suite 196 green; the transport was validatedend-to-end on Fortress (see #19).
🤖 Generated with Claude Code