Skip to content

Conversation

@jdoucett
Copy link
Contributor

Phase 2, Slice 1 — size-routing primitives (behavior-neutral)

First slice of Phase 2 (size-routing / append) per the RFC in #7 (§2, §2.10). Pure,
additive helpers — nothing wires them into the archive or reclaim flow yet, so this is
behavior-neutral.
Full suite 136 → 172 green. Safe to land ahead of the full RFC
review (Jarrod OK'd proceeding with slice 1).

What lands (all in archive.py, reusing the merged PR #8 primitives)

  • route(catalog, t_small, shard_count)[{slot, kind, arcnames}] — partition:
    files ≥ T_small become solo objects; smaller files bucket into content-addressed
    shards k = shake_128(arcname) mod K. Deterministic order; empty buckets omitted;
    disjoint union of the catalog.
  • shard_of / solo_slot / shard_slot / derive_shard_countK = ceil(small_bytes / shard_target), meant to be frozen per badge at baseline (a content-addressed map
    must be stable).
  • object_fingerprint — solo {mtime_s, size}; shard {members, byte_sum, digest}
    where digest = shake_128 over sorted (arcname, floor_s(mtime), size) triples (the
    authoritative change signal; catches add/remove/modify/size and a same-count/
    same-bytes reshuffle). object_unchanged — the skip/append signal (also the resume
    mechanism in slice 2).
  • clamp_t_small (100 MB floor), budget_warnings (warn-only: max_solo,
    max_objects_per_target, oversize shard), vault_dir_for / manifest_path /
    load_manifest / write_manifest
    (atomic tmp+os.replace, best-effort; the
    _vault/ child dodges reclaim.find_logs' non-recursive glob).
  • make_zip_files gains an optional only_arcnames filter (no caller yet) so a later
    slice can ship exactly one routed object. Keys stay os.path.relpath arcnames
    (invariant Canonical-root invariant: paths reclaim can never delete #2); an empty object still raises the clear error.

Tests — tests/test_routing_primitives.py (36 cases)

Boundary partition (≥ T_small = solo), shard stability (same arcname → same k), K derived

  • frozen, fingerprint change detection incl. reshuffle, object_unchanged skip vs
    shard-append, only_arcnames narrows to the subset with matching checksum keys, manifest
    round-trip + no leftover .tmp + _vault/ placement. Existing 136 unchanged.

python3 -m unittest discover -s testsRan 172 tests … OK.

Not in this slice (gated on the #7 review)

Archive wiring (route_and_ship, config keys, --globus refusal, --fresh), aggregate
reclaim (verify_target), reship-monitor, and restore.py — slices 2–5.

🤖 Generated with Claude Code

…tral)

Adds the pure, additive size-routing helpers to archive.py — nothing wires them
into the archive or reclaim flow yet (behavior-neutral; full suite 136 -> 172 green):

- route(catalog, t_small, shard_count) -> [{slot, kind, arcnames}] partition:
  files >= T_small become solo objects; smaller files bucket into content-addressed
  shards k = shake_128(arcname) mod K (no repack cascade -> cheap APPEND).
- shard_of / solo_slot / shard_slot / derive_shard_count (K = ceil(small_bytes /
  shard_target), frozen per badge at baseline).
- object_fingerprint (solo {mtime_s,size}; shard {members,byte_sum,digest}) +
  object_unchanged (the skip/append signal, also the resume mechanism).
- clamp_t_small (100 MB floor), budget_warnings (warn-only max_solo/max_objects/
  oversize-shard), vault_dir_for/manifest_path/load_manifest/write_manifest
  (atomic tmp+os.replace, best-effort; _vault child dodges reclaim's glob).
- make_zip_files gains an optional only_arcnames filter (no caller yet) so a later
  slice can ship exactly one routed object; keys stay arcname-relpath (invariant #2).

Tests: tests/test_routing_primitives.py (36 cases) — boundary partition, shard
stability, K frozen, fingerprint change detection incl. reshuffle, skip vs
shard-append, only_arcnames subset with matching checksum keys, manifest round-trip.

See docs/RFC_incremental_v2.md §2 (slice 1).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign in to join this conversation on GitHub.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant