From b617c5806a3eb1c136943cdbdc88bce1b1081cbb Mon Sep 17 00:00:00 2001 From: "Doucette, Jarrod S" Date: Wed, 8 Jul 2026 09:12:49 -0400 Subject: [PATCH 1/5] docs: refresh RFC #7 into the staged-hybrid design (Phase 2 = size-routing/append) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rewrites docs/RFC_incremental_v2.md from the leveled-only RFC into the umbrella staged-hybrid RFC that matches what actually shipped and what is being built next: - Phase 1 (exclusion): DONE/merged; recap + pointer to docs/EXCLUSION_SPEC.md. - Phase 2 (size-routing/APPEND): the new build target, full design — solo objects >= T_small (256 MiB, floor 100 MB), content-addressed shards (hash(relpath) mod K, K frozen at baseline), skip-unchanged (the append win, doubles as resume), per-target AGGREGATE reclaim (verify_target — REQUIRED same slice or every routed target false-DRIFTs), reship-monitor (the Phase-3 trigger), local-mode only (refuses --globus). 4 reviewable slices with tests. - Phase 3 (leveled L0/L1/L2): DEFERRED-DORMANT banner; the prior leveled RFC content (Path A/B, restore, chain-aware reclaim, Negishi spike) preserved intact as the Phase-3 reference. - Appendix: the PR #8 primitives (badge_of/make_blonde/enumerate_source_catalog/ floor_s) reused across phases, with current line refs. Design driver = the 2026-07-08 source_files measurement: standoff imagery is stable (exclusion covers it); the data modalities GROW by appending -> append is the dominant Phase-2 win. Review gate before any engine code. Co-Authored-By: Claude Opus 4.8 --- docs/RFC_incremental_v2.md | 697 +++++++++++++++++++++++++++++++++++++ 1 file changed, 697 insertions(+) create mode 100644 docs/RFC_incremental_v2.md diff --git a/docs/RFC_incremental_v2.md b/docs/RFC_incremental_v2.md new file mode 100644 index 0000000..b4a3e65 --- /dev/null +++ b/docs/RFC_incremental_v2.md @@ -0,0 +1,697 @@ +# RFC: Staged-Hybrid Incremental Backup for fortress-archive + +**Status:** Phase 1 DONE (merged). **Phase 2 (size-routing / append) — this RFC's build target, ready for review.** Phase 3 (leveled) DEFERRED. +**Base:** `main` @ `f7e505a` (suite 136 passing). Reuses the PR #8 primitives already merged. +**Supersedes:** the original leveled-only RFC (this file's prior revision, still the Phase-3 reference — see §3) and the closed design-only RFC (former PR #4). + +> **Read `docs/EXCLUSION_SPEC.md` first for Phase 1.** This RFC is the umbrella +> design; the exclusion spec is authoritative for the shipped Phase-1 mechanism +> and is not restated here. + +--- + +## Background — the measured waste, in two rounds + +`archive.py` re-ships a **whole target** whenever *any* matched file changes: it +re-zips, re-MD5s, and re-`htar_large`s every byte, even for a two-file delta. +Two rounds of live measurement on FloraSense/Negishi data drove the staging: + +**Round 1 (2026-06-29) — the standoff signature.** `repository_X1D_2_spectral-standoff` +(36,186 files, ~2.2 TB) re-shipped **2.35 TB** because exactly **2** files changed +(`*_status.json` + `*_parse-warnings.json`). Across the fleet: ~15 TB re-shipped +over 180 events, ~11 TB of it triggered by ≤5 changed files. Those two files are +**git-versioned regenerable metadata** → **Phase 1 (exclusion)** drops them and the +stable imagery ships once, then skips forever. + +**Round 2 (2026-07-08) — the append signature.** Diffing `source_files` across 809 +consecutive archive logs split the significant (large / big-file) targets into two +clean populations: + +- **Standoff imagery (12k–47k files): STABLE, metadata-only churn.** X0C_2 + 31978→31978 net 0, X1D_2 36186→36186 net 0, X0F_2 23572 net 0. → **Phase 1 + exclusion fully handles these; size-routing is not needed for standoff.** +- **Data modalities GROW by APPENDING** (files *added*, `modified=0`): omics-analysis + X0E_4 2638→7225 (+4587, 0 mod), X0D_4 657→4203, X0F_4 2633→4184, X0H_4 5→4863; + metabolomics/transcriptomics X1D_3 15→8816, X1B_3/X1C_3 16→7233; visual-injury & + spectral-contact X1C_1 234→623, X0E_1 450→569, X1B_2_contact 547→1317, + X0H_2_contact 1470→2094. Whole-target re-ship re-ships the **entire accumulated + target** on each growth. **Exclusion does NOT cover this — it is data, not + git-metadata.** Of 87 significant re-ship events, essentially all are growth or + partial-data, not all-files re-analysis. + + > **Correction to an earlier note.** A prior `du`-based read claimed omics + > re-ships had "~all files changed (legit re-analysis, size-routing no help)." + > The `source_files` diff refutes that: omics/metabolomics/etc. **GROW (append), + > `modified≈0`.** So the dominant Phase-2 win is **APPEND** (grow → ship only the + > new objects), which content-addressed shards + solo objects give for free — + > more than "solo big-file skip-unchanged." + +**NET:** Phase 1 (exclusion) fixes the stable-imagery-with-metadata-churn case +(standoff). Phase 2 (size-routing, **centered on append**) fixes the growing data +modalities. They are complementary, and both are measurement-justified. + +An earlier investigation also confirmed there is **no user-facing HPSS-native +versioned/dedup backup**: htar `-n` re-ships whole changed files into a new tar; +Globus `--sync-level` is unversioned transfer that drops tar aggregation + per-file +MD5; GNU `tar --listed-incremental` is the only real native incremental and needs +`htar`→`hsi put` (see the Phase-3 spike, §3.9). So the engine owns leveling itself. + +--- + +## 0. The staged hybrid at a glance + +Effort is spent where a live measurement justified it. Three phases, built in order: + +| Phase | Mechanism | Kills | Status | +|---|---|---|---| +| **1 — Exclusion** | Identity + git-coverage subtractive filter; drop git-versioned regenerable metadata from the backup | Standoff-style metadata churn on a stable bulk target (2.35 TB for 2 files) | **DONE — merged** (`docs/EXCLUSION_SPEC.md`; PRs #8/#10–#17; suite 136 green) | +| **2 — Size-routing / append** | Per target: files ≥ `T_small` ship as their own Fortress object (**solo**, skip-if-unchanged); smaller files bundle into K **content-addressed shards** (`shard = hash(relpath) mod K`, whole-shard re-ship on change). New file → new object; unchanged objects skip. Per-target **aggregate** reclaim. Reship-monitor. | Whole-target re-ship on **growth (append)** and on a single big file changing | **THIS RFC — ready to build** | +| **3 — Leveled L0/L1/L2** | Per-target full-anchor + differential/incremental delta tars, chain-aware reclaim, restore | **Uniform-medium-churn**: a large tier of small, non-git-redundant DATA files changing often — the one case exclusion can't drop and routing would re-ship whole shards | **DEFERRED — dormant** (§3; PR #9 branch `feat/incremental-level-engine` preserved). Activated per-asset **only** when the reship-monitor measures the signature. | + +The phases share one spine (adopted from Nathan Denny's Bastion; see +`bastion-architecture` memory and §3): a **badge** identity per target, a per-target +BLONDE-named object/tar scheme, skip-unchanged, and policy-gated escalation. Phase 2 +extends that spine with size-routing + content-addressed sharding + aggregate reclaim +— deliberate, measurement-justified extensions Bastion lacks. Bastion itself *is* the +Phase-3 leveled mechanism, which is why Phase 3 follows its drift/cadence/genus/thread +model when it is finally built. + +--- + +## 1. Phase 1 — Exclusion `[DONE]` + +Merged and running end-to-end (archive → reclaim (INV-E) → post-hoc audit), +`--globus`-capable, default-safe. **`docs/EXCLUSION_SPEC.md` is authoritative;** this +is a one-paragraph recap so the umbrella RFC is self-contained. + +After `file_pattern`, a **second subtractive filter** drops a file **iff** (a) it is +classified `EXCLUDE` (or `OPTIONAL` opted-out) by a **role-bearing relpath regex** — +never by extension (a bare `*.json` rule is rejected at compile time) — **and** (b) it +is proven git-`COVERED` (tracked + committed-at-HEAD + clean + blob on a fresh-fetched +remote off `/depot` + real bytes not an LFS pointer + no EOL filter). Every other +verdict → back it up. `evaluate_exclusions()` **never raises**; any problem degrades to +"back up everything" with a recorded reason (banner + run-log `exclusion` block + +email). INV-C (companion guard) suppresses exclusion run-wide if a directory would ship +zero members. **INV-E**: reclaim re-records the exact excluded arcname set **and +re-runs the git gate** per file, so a metadata-only change reads SAFE (no false DRIFT → +the re-ship is skipped — the actual byte saving) and an excluded-but-no-longer-COVERED +file reads DRIFTED (never a silent `--delete`). `verify_exclusions.py` re-confirms +months later. Runs under `--globus` (the spec PATH is read where the zip is built). + +Config keys: `exclude_spec` (path), `exclude_optional` (list). + +**Phase 2 composes with Phase 1, does not replace it:** exclusion is evaluated first, +over the full matched set; **routing partitions the survivors** (§2.5). INV-E's reclaim +re-gate operates on the Phase-2 aggregate (§2.6). + +--- + +## 2. Phase 2 — Size-routing / Append `[THIS RFC — READY TO BUILD]` + +### 2.1 Model — solos + content-addressed shards + +A logical target (one `badge`, §Appendix) no longer ships as one whole-target zip/tar. +Its **survivors** (post-`file_pattern`, post-exclusion) are partitioned into a set of +independent **Fortress objects**, each shipped through the *unchanged* +`make_zip_files → send_to_fortress` chain (its own zip → `htar_large` → `hsi get` → +`tar xvf` → zip-MD5 → per-file MD5). Two kinds: + +- **Solo object** — one file **≥ `T_small`**. Its own object; **skips if unchanged** + (fingerprint = floored-mtime + size; periodic re-hash for the same-size+same-mtime + blind spot, §2.3). Single-file restore. This is the "big file changed → ship only + it" win. +- **Shard object** — a bucket of files **< `T_small`**, assigned by + `k = int.from_bytes(shake_128(arcname)[:8]) % K`. Shard `k` re-ships **whole** when + any member changes (fingerprint = member arcname set + floored mtimes + byte-sum). + K is **fixed for the life of the badge** (recorded in the manifest at baseline, §2.4) + — a content-addressed map must be stable, or one file moving buckets reshuffles + everything. + +**Why content-addressed (not size-bin-packing):** a file's bucket is a pure function of +its relpath, identical every run. Adding a file lands it in a deterministic bucket; +only that bucket moves. Changing a file re-ships only its bucket. There is **no repack +cascade** (bin-packing shifts every downstream file when one is inserted). This is +exactly what makes **APPEND cheap**. + +**Where the append win comes from (Round-2 data):** the growing modalities add *big* +files → each new file becomes a **new solo**; every existing solo and every untouched +shard **skips**. A target that grows 2638→7225 with 4587 new big files ships ~4587 new +small objects and re-reads **zero** of the 2638 already-archived. The blunt +whole-target re-ship read all 7225 every time. + +> **Small-file growth is the honest edge.** A target that grows by thousands of *small* +> files (e.g. metabolomics 15→8816) hashes them across the K shards → many/most shards +> re-ship. But such targets are small in **bytes** (the expensive re-ships were the +> multi-TB imagery/omics targets), so whole-shard re-ship is cheap in absolute terms. If +> a *large* small-file tier ever churns repeatedly, that is the **uniform-medium-churn** +> signature the reship-monitor (§2.7) watches for — and the sole trigger for Phase 3. + +### 2.2 Thresholds, K, and budgets (measurement-tuned; all warn-only) + +| Knob | Default | Basis | +|---|---|---| +| `T_small` | **256 MiB** (floor **100 MB**) | Fortress auto-selects Class of Service by size (COS 10 ≤ small → disk-cache-fronted-tape; COS 11 large → tape). Sub-100 MB solo lands in COS 10 disk cache — *tape-fronted, not unsafe, just inefficient*, so 100 MB is a floor, not a safety line. **256 MiB → 16 solos** on the clean 2.2 TB imagery target; 100 MiB → 535 (over budget). See `fortress-hpss10-json-output` memory. | +| `shard_target` | **256 GiB** | Target uncompressed shard size. K derived at baseline: `K = max(1, ceil(small_tier_bytes / shard_target))`, then **frozen in the manifest**. Warn (do not fail) if a live shard exceeds ~2× this — the signal to raise `shard_target` and re-baseline (`--fresh`). | +| `max_solo` | **500** | Warn if a target's solo count exceeds this (object-count pressure on the vault). | +| `max_objects_per_target` | **1000** | Warn if solos + shards exceeds this. | +| `solo_rehash_days` | **30** | A solo unchanged by (mtime,size) but not re-hashed within this window is re-hashed (or re-shipped) to catch same-size+same-mtime rewrites (§2.3). | + +`shard_count` (K) may be pinned explicitly in config to override the derivation; once a +badge has a manifest, its recorded K is immutable (changing it requires `--fresh`). + +### 2.3 Fingerprints, skip-unchanged, and the one blind spot + +Skip-unchanged is the whole point; it is also the **resume mechanism** (§2.8). Before +shipping any object, its **live fingerprint** is computed from +`enumerate_source_catalog` (the merged `{arcname:(mtime_ns,size)}` primitive) and +compared to the fingerprint the manifest recorded for that slot: + +- **Solo** — `(floor_s(mtime_ns), size)`. Match → skip. +- **Shard** — `(sorted member arcnames, [floor_s(mtime) per member], Σ size)`. Any + change (add / remove / modify / size change) → re-ship the whole shard. + +A slot's fingerprint is written to the manifest **only after that object's transfer + +round-trip verify succeeds** — so a half-shipped object is simply absent/stale in the +manifest and re-ships next run (this is why skip-unchanged doubles as resume). + +**The blind spot (must land in CLAUDE.md invariants):** a same-size, same-mtime in-place +rewrite defeats both fingerprints. `reclaim.verify()` already lives with this exact +stale-but-safe blind spot; the byte-sum guard catches *size-changing* rewrites only. +Mitigations, day one: (1) `solo_rehash_days` periodic re-hash; (2) a `--full` / +`--fresh` re-baseline is the operational safety valve; (3) documented as a hard +limitation, not a footnote. The true fix is re-hashing every file, which is the precise +cost this feature exists to avoid. + +### 2.4 Manifest — per-target object index (reclaim-ready, no future migration) + +One JSON per badge at **`{log_dir}/_vault/{badge}.manifest.json`** — invisible to +`reclaim.find_logs` (its `glob.glob` is non-recursive; a `_vault/` child never matches), +so routed run logs and the manifest never collide with legacy scanning. Written LOCAL in +`__main__` after each object's transfer succeeds, atomically (`tmp` + `os.replace`), +**best-effort** (a manifest write must never fail an archive already verified on tape). +The manifest is the **aggregate index**; per-file MD5 stays in each object's own run-log +JSON (invariant #2/#4 preserved per object). + +```json +{ + "schema_version": 1, + "badge": "k7m3xq9p", + "project": "repository_X0E_4_omics-analysis", + "source_folder": "/depot/florasense/data/repository/X0E/4/omics-analysis", + "file_pattern": ".*", + "routing": { "t_small": 268435456, "shard_count": 8, "shard_target": 274877906944 }, + "objects": [ + { + "slot": "solo:9f2c1ab7", + "kind": "solo", + "arcnames": ["aligned/sample_042.bam"], + "blonde": "k7m3xq9p.lq4r8.s.k7m", + "fortress_tar": "/…/repository_X0E_4_omics-analysis__solo-9f2c1ab7_20260707_031500.tar", + "run_timestamp": "repository_X0E_4_omics-analysis__solo-9f2c1ab7_20260707_031500", + "zip_checksum": "ab12…md5", + "fingerprint": { "mtime_s": 1751857200, "size": 3221225472 }, + "n_files": 1, "source_bytes": 3221225472, "tar_bytes": 3222000000, + "rehashed_at": 1751857200 + }, + { + "slot": "shard:3", + "kind": "shard", + "arcnames": ["logs/run.log", "meta/params.txt", "…"], + "blonde": "k7m3xq9p.lq6t1.h.k7m", + "fortress_tar": "/…/repository_X0E_4_omics-analysis__shard-3of8_20260707_031500.tar", + "run_timestamp": "repository_X0E_4_omics-analysis__shard-3of8_20260707_031500", + "zip_checksum": "cd34…md5", + "fingerprint": { "members": 214, "mtime_sum": 375112340000, "byte_sum": 88123456 }, + "n_files": 214, "source_bytes": 88123456, "tar_bytes": 61230000 + } + ] +} +``` + +- **`slot`** is the stable logical key within the target: `solo:` + (stable per file path) or `shard:` (0..K−1). A solo whose content changes re-ships + to a **new** `fortress_tar` under the **same** slot; the manifest updates that slot in + place. A deleted big file → its solo slot is dropped from `objects` (its tape tar + lingers until Phase-3 tape lifecycle reclaims it; source-reclaim ignores it because it + is no longer in the current object set). +- **`arcnames`** per object makes the aggregate reconstructable for reclaim (§2.6): the + target's expected member set = ⋃ objects' arcnames (a **disjoint** union — each live + member is in exactly one current object). +- **`tar_bytes`** (on-tape) is recorded distinctly from `source_bytes` (uncompressed) so + a future `hsi ls -j` size audit compares against the right field. +- **BLONDE** reuses the merged `make_blonde(badge, run_epoch, detail)`; Phase 2 uses + `detail='s'` for solo, `'h'` for shard (display/ordering only — `slot` is the key). + +### 2.5 Archive orchestration (LOCAL only) + +Routing is decided **LOCAL in `__main__`** (it reads `log_dir`/manifest, which exist on +the launching host) and is **refused under `--globus`** (§2.8). Sequence when +`size_routing: true`: + +```text +route_and_ship(config): # LOCAL, __main__ + survivors, exclusion = run(enumerate_source_excluded, source, pattern, + exclude_spec, exclude_optional) # Phase 1 first, ONCE + catalog = run(enumerate_source_catalog, source, pattern) # {arcname:(mtime_ns,size)} + catalog = {a: v for a, v in catalog.items() if a in survivors} # exclusion applied + + badge = badge_of(project, source, pattern) + manifest = load_manifest(vault_dir, badge) # or None → baseline + K = manifest.routing.shard_count if manifest else derive_K(catalog, shard_target) + + objects = partition(catalog, T_small, K) # solos (≥T) + K shards (2×target + + shipped, skipped = [], [] + for obj in objects: + if manifest and fingerprint(obj, catalog) == manifest.slot(obj.slot).fingerprint: + skipped.append(obj); continue # THE APPEND WIN — unchanged, skip + # ship exactly this object's members; exclusion already applied → skip re-gate + zip_path, zip_ck, file_ck, members, src_bytes, _ = run( + make_zip_files, source, pattern, tmp_dir, obj_stem(project, obj), + compression, allow_empty_files, None, (), only_arcnames=obj.arcnames) + tar, txt, jsn = run(send_to_fortress, zip_path, zip_ck, file_ck, members, + fortress_base_dir, emails, obj_stem(project, obj), log_dir) + update_manifest(vault_dir, badge, obj, tar, zip_ck, fingerprint(obj, catalog)) # LOCAL, atomic + shipped.append(obj) + + drop_missing_slots(manifest, objects) # deleted big files → drop solo slots + reship_monitor(shipped, skipped, objects, emails) # §2.7 + send_completion_email(shipped, skipped, exclusion, budget_warnings) +``` + +- **`only_arcnames`** is a new optional param on `make_zip_files` (planned in the + Phase-3 RFC too, §3.6): after the pattern match, keep only arcnames in the set. It + MD5s **only** the object's members — unchanged bytes are never re-read. +- **Exclusion runs once** over the full survivor set; per-object `make_zip_files` calls + pass `exclude_spec=None` because their `only_arcnames` are *already* survivors — the + git gate is not re-run per object (it ran locally once; local-mode only). +- **Empty object guard:** `make_zip_files` raises on an empty match, so a shard whose + members all vanished mid-run is skipped (dropped from the manifest), not shipped. +- **`file_pattern` scoping** and every arcname stay `os.path.relpath(full, + normpath(source))` — identical in `enumerate_source_catalog`, `make_zip_files`, and + the round-trip verify (invariant #2). Solos and shards partition on that exact key. + +The default (`size_routing: false`) path is byte-for-byte the current whole-target +behavior — Phase 2 is opt-in and default-neutral. + +### 2.6 Aggregate reclaim — REQUIRED, same slice as the first routed ship + +**This is the load-bearing reclaim change and is not optional or a fast-follow.** A +routed target has **many** objects (many run logs) but reclaim today dedups to the +*newest log per project* and compares live-vs-that-one-log +(`reclaim.py:816-821` scan loop → `verify()` `reclaim.py:441/519/522`). For a routed +target the newest log is one object → `archived` = that object's few files → **every +routed target perpetually false-DRIFTs.** So `verify()` must become **per-target +aggregate**: + +1. In the scan loop, before per-log `verify()`, detect routed targets: a log whose + `badge` (recompute `badge_of`, or read it from the log) has a + `_vault/{badge}.manifest.json`. Group all such logs by badge; **route the group to a + new `verify_target(manifest, source_folder, holds, …)`** and do **not** feed the + individual object logs to the legacy `verify()`. +2. `verify_target`: expected member set = ⋃ objects' `arcnames`; expected count = Σ + `n_files`; expected bytes = Σ `source_bytes`. `scan_source` the live source **once** + (pattern-scoped), with INV-E exclusion re-gate subtracted (unchanged from Phase 1; + `recheck_exclusions` runs against the aggregate). Compare live count/bytes vs the + **aggregate** (never a single object). Require **every** object tar present + (`hsi_exists` per object; later `hsi ls -j` tape-size audit against `tar_bytes`). + Newer-than-run uses each object's own `run_timestamp` (a file is "newer" only if + newer than the run of the object that currently owns it). Verdict spans the whole + target: `SAFE | HOLD | DRIFTED | FAILED | GONE`, same semantics as today. +3. **Legacy, pattern-scoped, and whole-folder (non-routed) logs keep flowing through the + existing `verify()` unchanged** — this is a new parallel path opted into only when a + `_vault/` manifest exists. The 136-test suite stays green. +4. Source-space `--delete` semantics (snapshot / path / holds / canonical guards) are + unchanged; they gate on the aggregate verdict. + +This mirrors the deferred Phase-3 `verify_chain` (§3.5.2) but is **simpler**: objects +partition the *current* member set (a disjoint union), with no deltas to fold and no +level ordering — reclaim just sums the current objects. + +### 2.7 Reship-monitor (the Phase-3 trigger) + +After each routed run, compute what fraction re-shipped and email a warn-only summary: + +- **shard reship ratio** = shards shipped / K, and **byte reship ratio** = Σ shipped + `source_bytes` / Σ target `source_bytes`. +- If shard reship ratio **> ~25%** for a run — or a target trips the threshold on N + consecutive runs — email `[RESHIP-WATCH]`: this is the **uniform-medium-churn** + signature (a large small-file tier changing repeatedly, which routing re-ships as whole + shards and exclusion can't drop). It is the **only** trigger to activate Phase 3 for + that asset. Growth-by-append does **not** trip it (new big files are new solos, ~0% + shard reship). + +The monitor is the measurement discipline that keeps Phase 3 dormant until a real asset +demands it — matching how Phase 2 itself was gated on Round-2 data. + +### 2.8 Interactions + invariants + +- **`--globus` — refused when `size_routing: true`.** Routing + manifest are LOCAL + (`log_dir`/manifest on the launching host); under `--globus` `log_dir` may be on a + different host and manifest/logs diverge (the Phase-3 §3.5.3 concern). Exit with a + clear error naming the config key. (Phase-1 exclusion still runs under `--globus` + unchanged — only routing is local-only.) +- **Resume = skip-unchanged (no new machinery).** A routed target does **not** use + `find_existing_zip`/the stale-zip guard. A failed run re-runs; objects already on tape + (fingerprint recorded in the manifest) skip; only the unfinished remainder ships. A + changed `exclude_spec` or `T_small`/`K` shifts the survivor/partition set → those + objects' fingerprints miss → they re-ship (correct). `--fresh` forces a full + re-baseline (re-derive K, re-ship all). +- **Invariant #1 (dill):** all routing/partition/manifest/monitor logic is LOCAL in + `__main__`. Remote-eligible workers (`make_zip_files`, `send_to_fortress`, + `enumerate_source_catalog`, `enumerate_source_excluded`) receive only plain args + (the `only_arcnames` list). Moot in practice since routing refuses `--globus`, but kept + true so the workers stay serializable. +- **Invariant #2 (arcname↔checksum keys):** the catalog, `only_arcnames`, each object's + `file_checksums`, and the round-trip verify all key on the same + `os.path.relpath(full, normpath(source))`. Solo/shard assignment hashes that key. +- **Invariant #4 (fresh log pair):** every object writes its own fresh + `{obj_stem}_{ts}.txt/.json`; never appended. A skipped object writes no log. +- **Invariant #6 (htar stdout):** unchanged — each object is the existing + `htar_large -cvf` write + `hsi get`/`tar xvf` retrieve. No new transfer/verify surface + (like Phase-3 Path A). + +### 2.9 Build order (reviewable slices, same cadence as exclusion) + +Every slice ships its tests in the same change; `python3 -m unittest discover -s tests` +before merging (suite 136 → growing). Tests slot beside the existing ones +(`sys.path.insert` + `import archive, reclaim`; mock tape with +`reclaim.hsi_exists = lambda tar: True`). + +- **Slice 1 — routing primitives (behavior-neutral).** `route(catalog, T_small, K)` → + solos + shards; `slot_of`, `shard_of(arcname, K)`, `derive_K`; solo/shard + `fingerprint`; manifest read/write (atomic, best-effort); budget warnings. `only_arcnames` + param added to `make_zip_files` (no caller yet). **No archive/reclaim behavior change.** + Tests: partition correctness at the T_small boundary, shard stability across runs + (same arcname → same k), K-frozen-from-manifest, fingerprint change detection + (add/remove/modify/size), `only_arcnames` MD5s only the subset with keys matching + `source_files`, manifest round-trip + atomicity. +- **Slice 2 — archive wiring (the append engine).** `size_routing`/`t_small`/ + `shard_count`/`shard_target` config keys; `route_and_ship` in `__main__`; skip-unchanged; + drop-missing-slots; `--globus` refusal; `--fresh` re-baseline; run-log/manifest/email + reporting. Tests: baseline ships all objects; append ships only new solos (existing + skip); one big file changed → one solo re-ships; shard member change → whole shard + re-ships; deleted big file → slot dropped; `--globus` refused; `--fresh` re-baselines. +- **Slice 3 — aggregate reclaim (REQUIRED with the first routed ship, §2.6).** + `verify_target` + badge grouping in the scan loop; object logs excluded from legacy + `verify()`; INV-E re-gate on the aggregate; every-tar-present gate. Tests: a routed + target verifies SAFE (not DRIFTED); aggregate count/bytes match; a partial-GONE object + → FAILED; an INV-E no-longer-covered exclusion → DRIFTED; **the existing suite stays + green**. +- **Slice 4 — reship-monitor (§2.7).** shard/byte reship ratios; `[RESHIP-WATCH]` email; + run-log `routing` block. Tests: append run → ~0% (no watch); a wide shard churn → + watch fires; ratio math. + +A **single-file restore** utility (extract one solo, or one shard, by slot) is a natural +Phase-2.5 follow-up but is not required to ship the append win; the full multi-level +`restore.py` is Phase 3 (§3.4). + +--- + +## 3. Phase 3 — Leveled L0 / L1 / L2 `[DEFERRED — DORMANT]` + +> **DEFERRED.** Everything below is the **Phase-3 reference design**, preserved intact +> from this RFC's prior revision. It is **not being built now.** It is activated +> **per-asset only** when the Phase-2 reship-monitor (§2.7) measures the +> **uniform-medium-churn** signature — a large tier of small, non-git-redundant DATA +> files changing often, the one case exclusion can't drop and size-routing re-ships as +> whole shards. That signature has **not** appeared in any FloraSense asset. The code +> exists and is gated on branch `feat/incremental-level-engine` (parked PR #9). When +> built, it adopts Bastion's drift/cadence/genus/thread model (see `bastion-architecture` +> memory). The chain-aware reclaim (retain-N-L0, prune deltas, re-anchor, +> reconstruct-from-L0+all-deltas) is the expensive/risky part and is built here, not +> before. Line references below are to the pre-Phase-1 tree and are indicative, not +> current. + +### 3.0 Recommendation — Path A now, Path B only behind a passing spike + +**Build Path A (Python-on-zip deltas, keep `htar_large` for every level). Treat Path B (GNU `tar --listed-incremental` + `hsi put`) as a deferred, spike-gated option for L0 throughput only.** + +Why: + +1. **Zero new transfer/verify surface.** Path A shrinks only the *member set* fed to the existing `make_zip_files` → `send_to_fortress` chain. The verify chain (`htar_large -cvf` → `hsi get` → `tar xvf` → zip-MD5 → per-file MD5) stays byte-for-byte. Invariant #6 holds for free. (This is the same property Phase 2 relies on.) + +2. **Path B has an unresolved correctness defect in the hybrid.** A zip-derived L0 tree has no GNU `.snar` dumpdir baseline, so the first `--listed-incremental` delta restore has no state to reconcile L0→L1 deletions against. A unified Path B (L0 also via `hsi put`) fixes that but eats `htar_large`'s parallel-mover throughput on multi-TB L0s. Neither is acceptable to ship blind. + +3. **The measured waste is killed by Path A alone.** `repository_X1D_2_spectral-standoff` re-shipped 2.2 TB for 2 of 36,186 changed files. A Path-A L1 zips+MD5s+htars only those 2 files. (In the staged hybrid, Phase 1 already kills this specific case — Phase 3's niche is *non-git-redundant* small-file churn.) + +4. **`htar_large` is genuinely better for the L0.** Deltas are tiny by construction, so even if Path B is later adopted for deltas, single-stream `hsi put` is fine there — but the L0 should stay on `htar_large`. Path A keeps everything on it. + +**Phased A→B is the long-run shape:** ship Path A leveling + leveled reclaim, run the spike (§3.9), and adopt Path B for deltas *only if* it wins on retrieval+restore correctness AND the L0-baseline-deletion problem is closed (unified Path B, or engine-owned deletion replay). The manifest schema (§3.3) is designed so Path B drops in with **no schema migration** — the per-level node already carries `transfer:"zip"|"tar"` and a `snar_fortress_path` slot. + +### 3.1 Hard limitation accepted up front (must land in CLAUDE.md invariants) + +**Same-size, same-mtime in-place rewrite is invisible to the cheap-diff and will NOT be captured in any delta.** The byte-sum guard (reused from `reclaim.py`) catches size-*changing* rewrites; a content edit preserving count+mtime+total-bytes defeats mtime, count, and byte-sum simultaneously. `reclaim.verify()` already lives with this exact blind spot (a stale-but-safe verdict), but the *engine* skipping a real change is strictly worse — it under-ships. The only true fix is re-hashing, which is the precise cost this feature exists to avoid. (This is the same blind spot Phase 2 documents in §2.3.) + +Mitigation, day one: the NO-OP path emails loudly — *"detected zero metadata change; same-size in-place content edits are NOT captured. Use `--full` to force a fresh L0 if you rewrote files preserving mtime+size."* — and this is documented as a hard invariant, not a footnote. A periodic full re-baseline (`--full`) is the operational safety valve. + +### 3.2 Level decision — exact local-side algorithm + +Runs **LOCAL in `__main__`** (invariant #1: it reads `log_dir`/manifest, which exist only on the launching host). The remote workers receive only a precomputed `list[str]` of arcnames. + +**Where the (mtime,size) catalog is persisted:** in **BOTH**, with distinct roles. +- **Authoritative diff source:** the manifest's active-L0 node carries the full `arcname → [mtime_ns, size]` catalog. `decide_level` reads ONE local JSON to diff. (For very large L0s the catalog is written to a sibling `{stem}.catalog.json` next to the run log and the manifest stores its path + a hash, not the inline 36k-entry dict.) +- **Rebuild source:** the run-log JSON / sibling catalog makes the manifest rebuildable-from-logs (manifest is best-effort + rebuildable). + +**Catalog primitive:** `enumerate_source_catalog` (merged, PR #8) returns `{arcname: (mtime_ns, size)}`. Use `st_mtime_ns` as the single catalog precision everywhere; **floor ns → whole seconds** (`floor_s`, merged) before any compare against a whole-second `zip_ts`. + +```text +decide_level(badge, live_catalog, manifest, force_full): + if force_full or manifest has no live+present L0: + return Level(detail='0', parent=None, members=ALL matched arcnames) + + prior = manifest.active_l0_effective_catalog # L0 ∪ all deltas to date (reconstructed) + added = {a for a in live_catalog if a not in prior} + removed = {a for a in prior if a not in live_catalog} + modified = {a for a in live_catalog + if a in prior + and (floor_s(live[a].mtime_ns) != floor_s(prior[a].mtime_ns) + or live[a].size != prior[a].size)} + + # in-place rewrite catch (size-changing only) — reuses reclaim byte-sum logic + if not (added or removed or modified): + if sum(c.size for c in live_catalog.values()) != prior_total_bytes: + return Level(next_detail, parent=l0_blonde, members=ALL) # forced re-baseline + return NOOP # ship nothing; write a noop log + manifest event (LOUD email per §3.1) + + members = sorted(added | modified) # files that go INTO this delta + return Level(next_detail, parent=l0_blonde, members=members, deleted=sorted(removed)) +``` + +- **MD5 only the delta members.** `members` (added ∪ modified) is handed to `make_zip_files(..., only_arcnames=members)`; only those files are walked, zipped, and MD5'd. (Same `only_arcnames` primitive Phase 2 introduces.) +- **Deletions** are metadata only: recorded in the manifest level node's `deleted` list. A **deletions-only** level ships **no tar**. +- **NO-OP** writes a `status:"noop"` log (recognizable so reclaim skips it — §3.5) and a manifest event; ships nothing. + +**Concurrency (day one, not deferred):** acquire an advisory `fcntl.flock` on `{log_dir}/_vault/{badge}.lock` around `decide_level` + the manifest write. Two runs of one target otherwise race the manifest read-modify-write and last-writer-wins orphans a delta's tar. (Phase 2's manifest writes need the same lock.) + +### 3.3 BLONDE id + manifest level-tree (reclaim-ready, no future migration) + +**BLONDE — `badge.quaver.detail.genus`** (merged as `make_blonde`/`decode_blonde`, PR #8): +- `badge` = base32 of 40-bit `shake_128(project\0source_folder\0file_pattern)` — the **manifest key**. +- `quaver` = base36(run epoch seconds) — **lossless**, sequences same-level siblings off one L0 by run time. +- `detail` = 1 char **level** — `'0'`/`'F'` = full, `'1'`,`'2'`… = deltas. **The load-bearing leveling field.** (Phase 2 repurposes `detail` as an object-kind tag; the two phases never share a manifest.) +- `genus` = `badge[:3]` — **display only, NEVER a key.** + +**Topology is a STAR, not a linked list** (important for retain-N): every delta carries `parent_blonde = current_l0_blonde` (one hop to its L0). Deltas do **not** point at the prior delta. Run-time (`quaver`) is the only sequencer for restore ordering. Reclaim must treat **the whole badge's delta set per L0 as one indivisible chain**. + +**Manifest** at `{log_dir}/_vault/{badge}.manifest.json` (invisible to `reclaim.find_logs`' non-recursive glob). Written LOCAL in `__main__` after transfer succeeds, atomic (`tmp` + `os.replace`), best-effort. + +```json +{ + "schema_version": 1, + "badge": "k7m3xq9p2a", + "project": "repository_X1D_2_spectral-standoff", + "source_folder": "/depot/florasense/data/repository/X1D/2/spectral-standoff", + "file_pattern": ".*", + "current_l0_blonde": "k7m3xq9p2a.lq4r8.0.k7m", + "levels": [ + { + "blonde": "k7m3xq9p2a.lq4r8.0.k7m", + "level": 0, + "parent_blonde": null, + "run_timestamp": "repository_X1D_2_spectral-standoff_20260620_030000", + "transfer": "zip", + "fortress_tar": "/path/on/fortress/..._20260620_030000.tar", + "snar_fortress_path": null, + "zip_checksum": "ab12...md5", + "tar_bytes": 2310123456, + "n_files": 36186, + "source_bytes": 2199023255552, + "catalog_ref": "repository_X1D_2_spectral-standoff_20260620_030000.catalog.json", + "catalog_sha256": "ff00...", + "deleted": [] + }, + { + "blonde": "k7m3xq9p2a.lq6t1.1.k7m", + "level": 1, + "parent_blonde": "k7m3xq9p2a.lq4r8.0.k7m", + "run_timestamp": "repository_X1D_2_spectral-standoff_20260621_030000", + "transfer": "zip", + "fortress_tar": "/path/on/fortress/..._20260621_030000.tar", + "snar_fortress_path": null, + "zip_checksum": "cd34...md5", + "tar_bytes": 4096512, + "n_files": 2, + "source_bytes": 2400000000, + "catalog_ref": "repository_X1D_2_spectral-standoff_20260621_030000.catalog.json", + "catalog_sha256": "ab99...", + "deleted": [] + } + ] +} +``` + +Reclaim-readiness fields present **from day one**: `level`, `parent_blonde`, `current_l0_blonde`, `n_files`, per-level `tar_bytes` (**on-tape**, distinct from uncompressed `source_bytes`), `fortress_tar`, `snar_fortress_path` (null on Path A), `transfer`. `catalog_ref`/`catalog_sha256` point at the sibling catalog so the manifest stays lean. + +### 3.4 Restore — NEW (no restore path exists today) + +A new `restore.py` (or `archive.py restore` subcommand). Runs on Negishi. **Always restores into an empty scratch dir, never over live data.** + +```text +restore(badge, to_timestamp=None, dest): + m = load_manifest(badge) + l0 = node where level==0 and blonde==m.current_l0_blonde + chain = [l0] + sorted([d for d in m.levels if d.parent_blonde==l0.blonde + and d.level>0 + and quaver(d) <= quaver(to_timestamp or now)], + key=quaver) # run-time order — the only sequencer + + # 4a. chain completeness gate (BEFORE any transfer): every referenced tar must exist + for node in chain: + if node.fortress_tar is None: # deletions-only level — no tape artifact + continue + if not hsi_exists(node.fortress_tar): + ABORT "chain is broken — level {node.blonde} tar GONE; not restorable" + + os.makedirs(dest) # MUST be empty + for node in chain: # strict L0 → L1 → … order + if node.fortress_tar is not None: + local_tar = hsi_get(node.fortress_tar) + verify md5(local_tar segment) ... # reuse the zip-MD5 verify logic + tar xf local_tar -> unzip the inner zip -> OVERLAY members onto dest + for member in node: verify md5(member) == node.catalog[member] + for d in node.deleted: # apply this level's deletions to dest + remove dest/d + # 4b. final reconciliation (MANDATORY, not advisory) + reconstructed = fold(L0.catalog, deltas...) # apply add/modify/delete in order + assert set(walk(dest)) == set(reconstructed) else ABORT "restored tree != manifest" +``` + +**Restore-specific correctness points:** +- **Overlay semantics:** because every level is a self-contained zip (Path A), restore is *file-overlay* — extract each level's members on top of the accumulated tree, then apply that level's `deleted` list. Deletions come from the **manifest** — exactly why Path A avoids the L0-baseline-deletion defect that breaks Path B's hybrid. +- **Deletions-only node:** `fortress_tar is null` → skip `hsi_get`, apply `deleted` only. Special-cased (a naive `hsi_get(None)` would crash). +- **Strict ordering:** out-of-order replay yields a wrong tree. `quaver` orders the star's siblings. +- **Partial-GONE chain = un-restorable, full stop.** Any missing intermediate tar → FAILED/abort, never "SAFE minus one." + +### 3.5 Interactions + fixes + +#### 3.5.1 Stale-zip resume guard — misfires on deltas, must be level-aware +As written it compares full-live count to zip member count and flags any source newer than the zip — **both true by construction for a delta**. Every delta-resume would abort with a false `[STALE]` email. **Fix:** the guard's "zip is a complete snapshot of live source" invariant is **L0-only**. For a delta-resume, validate `set(members) == expected_delta_arcname_set` and no delta member's mtime is newer than `zip_ts` (floored). (Phase 2 sidesteps this entirely by using skip-unchanged as resume, §2.8.) + +#### 3.5.2 reclaim `verify()` drift on deltas — REQUIRED reclaim change, SAME PR +After an L1 run the newest log for that project IS the delta log, handed straight to `verify()`; it DRIFTs three ways (delta count vs full live; delta bytes vs full live; NO-OP `source_files=={}` trips `live != 0`). **Every leveled/no-op target would be perpetually false-DRIFTED.** **Fix (additive):** detect leveled logs (badge has a `_vault/{badge}.manifest.json`), group by badge, route to `verify_chain(manifest, …)` — reconstruct the expected full catalog from the level-tree (L0 ∪ deltas, applying add/modify/delete), `scan_source` once, compare live vs the *reconstructed* set, require every level tar present. Standalone delta logs and `status:"noop"` logs never reach `verify()`. Legacy logs keep flowing through the existing `verify()`. **Phase 2's `verify_target` (§2.6) is the simpler, disjoint-union sibling of this and is built first.** + +#### 3.5.3 Invariants +- **#1 (dill):** all level-decision/diff/manifest/lock logic is LOCAL in `__main__`. **`--globus` caveat:** under `--globus`, `send_to_fortress` writes logs + sibling catalog on the *endpoint host* while the manifest is read/written on the *launching host*; if they differ, manifest and logs diverge. Fix: (a) the remote worker returns the catalog in its `result()` payload for the LOCAL manifest, or (b) refuse leveling under `--globus`. **Local mode is the only fully-supported mode** (Phase 2 takes option (b) outright). +- **#2 / #4 / #6:** same arcname keying, fresh per-level log pair, `htar_large` retained — as in Phase 2. + +### 3.6 Integration table (indicative, pre-Phase-1 line refs) + +| Piece | File | Local vs Remote | Change | +|---|---|---|---| +| BLONDE badge/blonde encode/decode | `archive.py` | LOCAL | **merged (PR #8)** | +| `enumerate_source_catalog` | `archive.py` | REMOTE-eligible | **merged (PR #8)** | +| `only_arcnames` filter | `make_zip_files` | REMOTE | **introduced in Phase 2 slice 1** | +| `decide_level()` | new, `__main__` | LOCAL | Phase 3 | +| per-badge `flock` | new, `__main__` | LOCAL | `fcntl` (needed by Phase 2 too) | +| level-aware resume guard | resume block | LOCAL | Phase 3 (Phase 2 uses skip-unchanged) | +| manifest read/write (atomic) | new, `__main__` | LOCAL | **shared with Phase 2** | +| `verify_chain()` + badge grouping | `reclaim.py` scan loop | LOCAL | Phase 3 (sibling of Phase-2 `verify_target`) | +| `restore.py` | new file | LOCAL (Negishi) | Phase 3 | +| retain-N-L0 + `hsi ls -j` | `reclaim.py` | LOCAL | §3.8 | + +### 3.7 Code sketches + +**Level decision (LOCAL, `__main__`):** +```python +import hashlib, base64, fcntl # badge_of/make_blonde/floor_s already merged in archive.py + +def decide_level(badge, live_catalog, manifest, force_full): + if force_full or not manifest or not manifest.get("levels"): + return dict(detail="0", parent=None, members=sorted(live_catalog), deleted=[]) + prior, prior_bytes = effective_catalog(manifest) # L0 ∪ deltas, with totals + added = [a for a in live_catalog if a not in prior] + removed = [a for a in prior if a not in live_catalog] + modified = [a for a in live_catalog if a in prior and + (floor_s(live_catalog[a][0]) != floor_s(prior[a][0]) + or live_catalog[a][1] != prior[a][1])] + if not (added or modified or removed): + if sum(v[1] for v in live_catalog.values()) != prior_bytes: + return dict(detail=next_detail(manifest), parent=manifest["current_l0_blonde"], + members=sorted(live_catalog), deleted=[]) # forced re-baseline + return None # NO-OP + return dict(detail=next_detail(manifest), parent=manifest["current_l0_blonde"], + members=sorted(set(added) | set(modified)), deleted=sorted(removed)) +``` + +**Manifest update (LOCAL, atomic, best-effort, after transfer) — shared shape with Phase 2:** +```python +def update_manifest(vault_dir, badge, node, is_l0): + os.makedirs(vault_dir, exist_ok=True) + path = os.path.join(vault_dir, f"{badge}.manifest.json") + m = json.load(open(path)) if os.path.exists(path) else \ + {"schema_version": 1, "badge": badge, "levels": []} + m["levels"].append(node) + if is_l0: m["current_l0_blonde"] = node["blonde"] + tmp = path + ".tmp" + with open(tmp, "w") as fh: json.dump(m, fh, indent=2) + os.replace(tmp, path) # atomic; failure here never fails an archive on tape +``` + +### 3.8 Reclaim fast-follow (design-only) + +- **retain-N-L0 (Bastion default N=3):** group levels by `current_l0_blonde`/`parent_blonde` into chains; keep the newest N; release an older chain only when no retained delta's `parent_blonde` resolves to its L0. Star topology → release/keep at chain granularity. No migration. +- **Release capability is unbuilt.** Releasing an old L0 chain means deleting tars **from tape** (`hsi rm`) + pruning manifest nodes. `reclaim.delete_source` only `rmtree`s a Depot source today; the fast-follow adds a guarded `hsi rm` (re-confirm via `hsi ls -j` immediately before, mirror the snapshot/path guards). +- **`hsi ls -j` hardening (HPSS 10.3 JSON: `Type`/`Size`/`LastWrite`):** `hsi_exists` judges by return code only. Switch to `hsi ls -j` to confirm `Type`, cross-check each node's `tar_bytes` against the tape size (a real audit), and parse `LastWrite`. The reason the manifest records `tar_bytes` separately. **(Phase 2's `verify_target` benefits from the same hardening for its per-object tar audit.)** + +### 3.9 Spike spec — `htar_large` vs (GNU tar `--listed-incremental` + `hsi put`), on Negishi + +**Why:** decide whether Path B ever earns adoption for deltas. Negishi-only (the dev Mac is `bsdtar 3.5.3`, no `--listed-incremental`). + +**Representative target:** `repository_X1D_2_spectral-standoff` (36,186 files, ~2.2 TB). L0 = full; L1 = touch 2 files. + +**Measure:** (1) L0 write throughput `htar_large -cvf` vs single-stream `hsi put`; (2) L0 retrieval round-trip; (3) delta write+retrieve (2-file L1); (4) restore-reassembly L0→L1 **including a deletion** — does GNU incremental restore correctly remove a file deleted before L1 **when L0 is a zip-derived tree** (the suspected hybrid defect)?; (5) native-incremental viability under Depot/NetApp inode rotation (false redump?). + +**Pass/fail:** Path B adoptable for deltas iff delta write+retrieve+restore verify byte-for-byte AND restore-with-deletion is correct. Its hybrid L0 boundary passes only if test 4 shows correct L0→L1 deletion replay over a zip-derived L0. **Default if any fail:** stay on Path A for all levels. + +### 3.10 Phased build order (when Phase 3 activates) + +- **P3-1 — Level decision + delta build + manifest + lock + `--full` + NO-OP + level-aware resume guard.** (Reuses `only_arcnames`, `badge_of`, `make_blonde`, `floor_s`, `enumerate_source_catalog`, and the manifest/lock machinery from Phase 2.) +- **P3-2 — Leveled reclaim (`verify_chain`, REQUIRED with the first delta).** +- **P3-3 — `restore.py`.** +- **P3-4 (fast-follow) — retain-N-L0 + guarded `hsi rm` + `hsi ls -j`.** +- **Spike (§3.9)** runs in parallel on Negishi; decides whether a Path-B-for-deltas PR is ever opened. + +--- + +## Appendix — primitives already merged (PR #8), reused across phases + +These landed behavior-neutral in Phase 1 and are the shared foundation: + +- **`badge_of(project, source_folder, file_pattern)`** — 40-bit `shake_128`, base32, 8 + chars. The stable per-target identity and the **manifest key** for both Phase 2 and + Phase 3. `archive.py:504`. +- **`make_blonde(badge, run_epoch, detail)` / `decode_blonde`** — `badge.quaver.detail.genus`; + `quaver` = lossless base36 run-epoch. Phase 2: `detail∈{s,h}` (solo/shard); + Phase 3: `detail∈{0,1,2,…}` (level). `archive.py:521/537`. +- **`enumerate_source_catalog(source, pattern)`** — `{arcname:(mtime_ns,size)}`, arcname + keyed identically to `make_zip_files`/round-trip verify (invariant #2). The diff/ + fingerprint source for both phases. `archive.py:439`. +- **`floor_s(mtime_ns)`** — ns→whole-seconds floor; the single precision for every + mtime compare. `archive.py:476`. + +Manifests live at `{log_dir}/_vault/{badge}.manifest.json` — the `_vault/` child dodges +reclaim's non-recursive `*.json` glob, so manifests never collide with run-log scanning. +Phase 2 and Phase 3 use **separate manifest schemas** under the same convention and are +never mixed for one badge. From febf616dfdfbb540b6391515d4523bb68fad86d4 Mon Sep 17 00:00:00 2001 From: "Doucette, Jarrod S" Date: Wed, 8 Jul 2026 09:21:42 -0400 Subject: [PATCH 2/5] docs(rfc): make full restore the capstone slice of Phase 2 (not a Phase-3 deferral) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Full restore now §2.9 (object-model reassembly: disjoint union, so simpler than the deferred Phase-3 leveled restore — no overlay/deletion replay/chain fold) and slice 5 of the build order (§2.10). --slot selective restore subsumes single-file restore. Restore is what makes reclaim's aggregate --delete trustworthy (prove round-trip first). Co-Authored-By: Claude Opus 4.8 --- docs/RFC_incremental_v2.md | 73 ++++++++++++++++++++++++++++++++++---- 1 file changed, 67 insertions(+), 6 deletions(-) diff --git a/docs/RFC_incremental_v2.md b/docs/RFC_incremental_v2.md index b4a3e65..fcac23d 100644 --- a/docs/RFC_incremental_v2.md +++ b/docs/RFC_incremental_v2.md @@ -65,7 +65,7 @@ Effort is spent where a live measurement justified it. Three phases, built in or | Phase | Mechanism | Kills | Status | |---|---|---|---| | **1 — Exclusion** | Identity + git-coverage subtractive filter; drop git-versioned regenerable metadata from the backup | Standoff-style metadata churn on a stable bulk target (2.35 TB for 2 files) | **DONE — merged** (`docs/EXCLUSION_SPEC.md`; PRs #8/#10–#17; suite 136 green) | -| **2 — Size-routing / append** | Per target: files ≥ `T_small` ship as their own Fortress object (**solo**, skip-if-unchanged); smaller files bundle into K **content-addressed shards** (`shard = hash(relpath) mod K`, whole-shard re-ship on change). New file → new object; unchanged objects skip. Per-target **aggregate** reclaim. Reship-monitor. | Whole-target re-ship on **growth (append)** and on a single big file changing | **THIS RFC — ready to build** | +| **2 — Size-routing / append** | Per target: files ≥ `T_small` ship as their own Fortress object (**solo**, skip-if-unchanged); smaller files bundle into K **content-addressed shards** (`shard = hash(relpath) mod K`, whole-shard re-ship on change). New file → new object; unchanged objects skip. Per-target **aggregate** reclaim. Reship-monitor. **Full restore** (object-model reassembly) as the capstone slice. | Whole-target re-ship on **growth (append)** and on a single big file changing | **THIS RFC — ready to build** | | **3 — Leveled L0/L1/L2** | Per-target full-anchor + differential/incremental delta tars, chain-aware reclaim, restore | **Uniform-medium-churn**: a large tier of small, non-git-redundant DATA files changing often — the one case exclusion can't drop and routing would re-ship whole shards | **DEFERRED — dormant** (§3; PR #9 branch `feat/incremental-level-engine` preserved). Activated per-asset **only** when the reship-monitor measures the signature. | The phases share one spine (adopted from Nathan Denny's Bastion; see @@ -372,7 +372,61 @@ demands it — matching how Phase 2 itself was gated on Round-2 data. `htar_large -cvf` write + `hsi get`/`tar xvf` retrieve. No new transfer/verify surface (like Phase-3 Path A). -### 2.9 Build order (reviewable slices, same cadence as exclusion) +### 2.9 Full restore — object-model reassembly (the Phase-2 capstone) + +Full restore is the **last slice of Phase 2**, not a Phase-3 deferral. Reclaim's +aggregate `--delete` (§2.6) is only trustworthy once the object set is *proven* to +round-trip off tape — restore is what makes "provably safe to delete" mean what it says. +A new `restore.py` (or `archive.py restore` subcommand), LOCAL on Negishi (needs `hsi` + +filesystem), **always restoring into an empty scratch dir, never over live data.** + +Because Phase-2 objects are a **disjoint partition of the current member set** (each live +file is in exactly one current object), restore is strictly simpler than the deferred +Phase-3 leveled restore (§3.4): **no overlay ordering, no deletion replay, no chain +fold** — just extract every current object and reconcile. + +```text +restore(badge, dest, only_slots=None): # LOCAL, Negishi + m = load_manifest(badge) + objects = m.objects if only_slots is None else [o for o in m.objects if o.slot in only_slots] + + # (a) completeness gate BEFORE any transfer — every referenced tar must exist + for o in objects: + if not hsi_exists(o.fortress_tar): # later: hsi ls -j + tar_bytes audit + ABORT "object {o.slot} tar GONE — target not fully restorable" + + os.makedirs(dest) # MUST be empty + restored = set() + for o in objects: # order irrelevant — disjoint union + local_tar = hsi_get(o.fortress_tar) + verify md5(inner zip) == o.zip_checksum # reuse send_to_fortress verify block + tar xf local_tar -> unzip inner zip -> extract o.arcnames into dest + for member in o.arcnames: + verify md5(dest/member) == object_runlog[o].file_checksums[member] + restored.add(member) + + # (b) final reconciliation (MANDATORY, not advisory) — only on a full restore + if only_slots is None: + assert restored == set().union(*[set(o.arcnames) for o in m.objects]) \ + else ABORT "restored tree != manifest aggregate" +``` + +- **Per-object verify is byte-for-byte today's round-trip** (zip-MD5 anchor + per-file + MD5 against that object's run-log `file_checksums`) — invariant #6 reused verbatim, + no new verify surface. +- **Selective restore falls out for free:** `--slot solo:9f2c1ab7` (one big file) or + `--slot shard:3` restores just that object — the same loop over a filtered set. This + subsumes the "single-file restore" idea as a flag, not a separate tool. The mandatory + final reconciliation runs only on a *full* restore (a selective restore intentionally + yields a subset). +- **Partial-GONE = un-restorable, full stop** (gate (a)): if any current object's tar is + missing, abort — never "restored minus one." Same discipline as §3.4. +- **Not an automatic pre-delete gate:** a full restore pulls every tar off tape (TB-scale, + slow), so it stays operator-invoked, not wired into `reclaim --delete`. It is the tool + an operator runs to *audit* a target before trusting reclaim, and the tool that + actually gets data back. + +### 2.10 Build order (reviewable slices, same cadence as exclusion) Every slice ships its tests in the same change; `python3 -m unittest discover -s tests` before merging (suite 136 → growing). Tests slot beside the existing ones @@ -402,10 +456,17 @@ before merging (suite 136 → growing). Tests slot beside the existing ones - **Slice 4 — reship-monitor (§2.7).** shard/byte reship ratios; `[RESHIP-WATCH]` email; run-log `routing` block. Tests: append run → ~0% (no watch); a wide shard churn → watch fires; ratio math. - -A **single-file restore** utility (extract one solo, or one shard, by slot) is a natural -Phase-2.5 follow-up but is not required to ship the append win; the full multi-level -`restore.py` is Phase 3 (§3.4). +- **Slice 5 — full restore (§2.9), the capstone.** `restore.py` / `archive.py restore`: + completeness gate, per-object `hsi get` + `tar xf` + zip-MD5 + per-file-MD5 verify, + mandatory final reconciliation vs the manifest aggregate, `--slot` selective restore, + empty-dest guard. Tests: full round-trip of a routed target (mock tape) reconstructs + the exact aggregate arcname set; a partial-GONE object aborts before any extraction; + `--slot` restores just one solo / one shard; a per-file MD5 mismatch aborts; restore + refuses a non-empty dest. + +Full `restore.py` handles only the Phase-2 **object** model (disjoint union). The +deferred Phase-3 leveled restore (§3.4) — chain fold, overlay ordering, deletion replay +— is a superset built only if/when Phase 3 activates. --- From 55c14c644b6283eff8e9dd1796d9e51bf4d617d7 Mon Sep 17 00:00:00 2001 From: "Doucette, Jarrod S" Date: Wed, 8 Jul 2026 09:40:22 -0400 Subject: [PATCH 3/5] docs(rfc): sync fingerprint/slot detail with slice-1 impl; mark slice 1 DONE MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - shard fingerprint = {members, byte_sum, digest} (shake_128 over sorted (arcname, floor_s(mtime), size) triples) — closes the same-count/same-bytes reshuffle hole the plain mtime_sum draft had. - solo slot = 64-bit hex(shake_128(arcname)[:8]) (was b32[:4]); route() guards the residual collision. - §2.10 slice 1 marked [DONE] with the real helper names + test coverage. Co-Authored-By: Claude Opus 4.8 --- docs/RFC_incremental_v2.md | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-) diff --git a/docs/RFC_incremental_v2.md b/docs/RFC_incremental_v2.md index fcac23d..cd88a21 100644 --- a/docs/RFC_incremental_v2.md +++ b/docs/RFC_incremental_v2.md @@ -167,8 +167,11 @@ shipping any object, its **live fingerprint** is computed from compared to the fingerprint the manifest recorded for that slot: - **Solo** — `(floor_s(mtime_ns), size)`. Match → skip. -- **Shard** — `(sorted member arcnames, [floor_s(mtime) per member], Σ size)`. Any - change (add / remove / modify / size change) → re-ship the whole shard. +- **Shard** — `{members, byte_sum, digest}` where `digest` = `shake_128` over the + sorted `(arcname, floor_s(mtime), size)` triples. The digest is the authoritative + change signal (add / remove / modify / size change → whole-shard re-ship, and it + also catches a same-count/same-bytes *reshuffle* that a plain sum would miss); + `members`/`byte_sum` are readable cross-checks. A slot's fingerprint is written to the manifest **only after that object's transfer + round-trip verify succeeds** — so a half-shipped object is simply absent/stale in the @@ -221,15 +224,16 @@ JSON (invariant #2/#4 preserved per object). "fortress_tar": "/…/repository_X0E_4_omics-analysis__shard-3of8_20260707_031500.tar", "run_timestamp": "repository_X0E_4_omics-analysis__shard-3of8_20260707_031500", "zip_checksum": "cd34…md5", - "fingerprint": { "members": 214, "mtime_sum": 375112340000, "byte_sum": 88123456 }, + "fingerprint": { "members": 214, "byte_sum": 88123456, "digest": "3f9c…shake128" }, "n_files": 214, "source_bytes": 88123456, "tar_bytes": 61230000 } ] } ``` -- **`slot`** is the stable logical key within the target: `solo:` - (stable per file path) or `shard:` (0..K−1). A solo whose content changes re-ships +- **`slot`** is the stable logical key within the target: `solo:` + (64-bit, stable per file path — 64 bits so hundreds of solos never collide; `route()` + guards the residual case rather than silently merging) or `shard:` (0..K−1). A solo whose content changes re-ships to a **new** `fortress_tar` under the **same** slot; the manifest updates that slot in place. A deleted big file → its solo slot is dropped from `objects` (its tape tar lingers until Phase-3 tape lifecycle reclaims it; source-reclaim ignores it because it @@ -433,14 +437,17 @@ before merging (suite 136 → growing). Tests slot beside the existing ones (`sys.path.insert` + `import archive, reclaim`; mock tape with `reclaim.hsi_exists = lambda tar: True`). -- **Slice 1 — routing primitives (behavior-neutral).** `route(catalog, T_small, K)` → - solos + shards; `slot_of`, `shard_of(arcname, K)`, `derive_K`; solo/shard - `fingerprint`; manifest read/write (atomic, best-effort); budget warnings. `only_arcnames` - param added to `make_zip_files` (no caller yet). **No archive/reclaim behavior change.** - Tests: partition correctness at the T_small boundary, shard stability across runs - (same arcname → same k), K-frozen-from-manifest, fingerprint change detection - (add/remove/modify/size), `only_arcnames` MD5s only the subset with keys matching - `source_files`, manifest round-trip + atomicity. +- **Slice 1 — routing primitives (behavior-neutral). `[DONE]`** Landed in `archive.py`: + `route` → solos + shards; `shard_of`/`solo_slot`/`shard_slot`; `derive_shard_count`; + `object_fingerprint` + `object_unchanged` (the skip/append signal); `clamp_t_small`; + `budget_warnings`; `vault_dir_for`/`manifest_path`/`load_manifest`/`write_manifest` + (atomic, best-effort); the `only_arcnames` param on `make_zip_files` (no caller yet). + **No archive/reclaim behavior change** (full suite green, 136 → 172). Tests + (`tests/test_routing_primitives.py`): partition at the T_small boundary, shard + stability across runs (same arcname → same k), K derived + frozen, fingerprint change + detection (add/remove/modify/size + reshuffle), `object_unchanged` skip vs shard-append, + `only_arcnames` narrows to the subset with keys matching `source_files` (invariant #2), + manifest round-trip + no leftover `.tmp` + `_vault/` dodges the reclaim glob. - **Slice 2 — archive wiring (the append engine).** `size_routing`/`t_small`/ `shard_count`/`shard_target` config keys; `route_and_ship` in `__main__`; skip-unchanged; drop-missing-slots; `--globus` refusal; `--fresh` re-baseline; run-log/manifest/email From dadb1d4686c67a1383a24c9c0328186260a02054 Mon Sep 17 00:00:00 2001 From: "Doucette, Jarrod S" Date: Wed, 8 Jul 2026 09:40:39 -0400 Subject: [PATCH 4/5] feat(routing): Phase-2 size-routing primitives (slice 1, behavior-neutral) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds the pure, additive size-routing helpers to archive.py — nothing wires them into the archive or reclaim flow yet (behavior-neutral; full suite 136 -> 172 green): - route(catalog, t_small, shard_count) -> [{slot, kind, arcnames}] partition: files >= T_small become solo objects; smaller files bucket into content-addressed shards k = shake_128(arcname) mod K (no repack cascade -> cheap APPEND). - shard_of / solo_slot / shard_slot / derive_shard_count (K = ceil(small_bytes / shard_target), frozen per badge at baseline). - object_fingerprint (solo {mtime_s,size}; shard {members,byte_sum,digest}) + object_unchanged (the skip/append signal, also the resume mechanism). - clamp_t_small (100 MB floor), budget_warnings (warn-only max_solo/max_objects/ oversize-shard), vault_dir_for/manifest_path/load_manifest/write_manifest (atomic tmp+os.replace, best-effort; _vault child dodges reclaim's glob). - make_zip_files gains an optional only_arcnames filter (no caller yet) so a later slice can ship exactly one routed object; keys stay arcname-relpath (invariant #2). Tests: tests/test_routing_primitives.py (36 cases) — boundary partition, shard stability, K frozen, fingerprint change detection incl. reshuffle, skip vs shard-append, only_arcnames subset with matching checksum keys, manifest round-trip. See docs/RFC_incremental_v2.md §2 (slice 1). Co-Authored-By: Claude Opus 4.8 --- archive.py | 267 ++++++++++++++++++++++++++++- tests/test_routing_primitives.py | 283 +++++++++++++++++++++++++++++++ 2 files changed, 549 insertions(+), 1 deletion(-) create mode 100644 tests/test_routing_primitives.py diff --git a/archive.py b/archive.py index c12eb9c..c168d6c 100644 --- a/archive.py +++ b/archive.py @@ -548,6 +548,254 @@ def decode_blonde(blonde): } +# --------------------------------------------------------------------------- +# Size-routing primitives (Phase 2 — see docs/RFC_incremental_v2.md §2). +# +# Partition a target's survivors into independent Fortress OBJECTS: +# - solo: one file >= T_small, its own object, skip-if-unchanged. +# - shard: files < T_small bucketed by shard = hash(relpath) mod K +# (K frozen per badge at baseline), whole-shard re-ship on change. +# Content-addressed => a file's bucket is a pure function of its relpath, stable +# every run, so adding/changing one file touches exactly one object (no repack +# cascade) — this is what makes APPEND cheap. +# +# Pure, additive, deterministic helpers. NOTHING wires them into the archive or +# reclaim flow yet — this slice is BEHAVIOR-NEUTRAL. Later slices add +# route_and_ship (archive), verify_target (reclaim aggregate), the reship-monitor, +# and restore.py. Routing is LOCAL-only (a later slice refuses --globus). Imports +# stay inside the bodies to match the file's convention. +# --------------------------------------------------------------------------- + +# Defaults the RFC (§2.2) cites; floor/budgets are enforced/emitted by callers. +T_SMALL_DEFAULT = 256 * 1024 * 1024 # 256 MiB — solo/shard cutoff +T_SMALL_FLOOR = 100 * 1000 * 1000 # 100 MB — COS disk->tape line; hard floor +SHARD_TARGET_DEFAULT = 256 * 1024 ** 3 # 256 GiB — target uncompressed shard size +MAX_SOLO_DEFAULT = 500 +MAX_OBJECTS_DEFAULT = 1000 + + +def clamp_t_small(t_small): + """ + Raise a configured T_small to the 100 MB floor if needed (RFC §2.2): below it a + solo lands in the Fortress COS-10 disk cache — tape-fronted, so not unsafe, just + inefficient. Returns an int >= T_SMALL_FLOOR. + """ + return max(int(t_small), T_SMALL_FLOOR) + + +def shard_of(arcname, shard_count): + """ + Content-addressed bucket for a relpath: k = int(shake_128(arcname)) mod K, in + [0, shard_count). PURE + STABLE — same arcname and same K always give the same k, + every run, so adding/changing one file touches exactly one shard (no repack + cascade). K is frozen per badge at baseline (recorded in the manifest); never + recompute it from a changed shard_count without a --fresh re-baseline. + """ + import hashlib + + if shard_count < 1: + raise ValueError("shard_count must be >= 1") + h = hashlib.shake_128(arcname.encode("utf-8")).digest(8) + return int.from_bytes(h, "big") % shard_count + + +def solo_slot(arcname): + """ + Stable logical slot key for a solo object: 'solo:', hex = 64-bit shake_128 + of the relpath. Stable per file path across runs (a solo whose bytes change + re-ships to a new tar under the SAME slot). 64 bits so hundreds of solos never + collide in practice (birthday ~ n^2 / 2^65). + """ + import hashlib + + return "solo:" + hashlib.shake_128(arcname.encode("utf-8")).hexdigest(8) + + +def shard_slot(k): + """Slot key for shard k: 'shard:'.""" + return "shard:{}".format(int(k)) + + +def derive_shard_count(catalog, t_small=T_SMALL_DEFAULT, + shard_target=SHARD_TARGET_DEFAULT): + """ + Baseline K for a target: ceil(small-tier bytes / shard_target), floored at 1. + Computed ONCE at baseline from the live catalog, then FROZEN in the manifest + (immutable for the badge's life — a content-addressed map must be stable). The + catalog is {arcname: (mtime_ns, size)} from enumerate_source_catalog. + """ + small_bytes = sum(size for (_m, size) in catalog.values() if size < t_small) + if small_bytes <= 0: + return 1 + return max(1, -(-small_bytes // shard_target)) # ceil division, integer-only + + +def route(catalog, t_small=T_SMALL_DEFAULT, shard_count=None, + shard_target=SHARD_TARGET_DEFAULT): + """ + Partition a catalog into objects. Returns a list of dicts, deterministic order + (solos by slot, then shards by k): + + {"slot": str, "kind": "solo"|"shard", "arcnames": [relpath, ...]} + + Files >= t_small each become a solo object; smaller files bucket into shards by + shard_of(arcname, K). Empty buckets are omitted (nothing to ship); K itself is + fixed, so an empty bucket that later gains a file simply reappears. If shard_count + is None it is derived (derive_shard_count) — callers WITH an existing manifest MUST + pass the frozen K from it instead of re-deriving. + """ + if shard_count is None: + shard_count = derive_shard_count(catalog, t_small, shard_target) + + solos, buckets = [], {} + for arcname in sorted(catalog): + size = catalog[arcname][1] + if size >= t_small: + solos.append({"slot": solo_slot(arcname), "kind": "solo", + "arcnames": [arcname]}) + else: + buckets.setdefault(shard_of(arcname, shard_count), []).append(arcname) + + # Guard the (astronomically unlikely) solo-slot collision rather than silently + # merge two distinct files into one slot. + slots = [o["slot"] for o in solos] + if len(set(slots)) != len(slots): + raise RuntimeError("solo slot collision — two relpaths hashed to one solo " + "slot; should be impossible at 64 bits") + + solos.sort(key=lambda o: o["slot"]) + shards = [{"slot": shard_slot(k), "kind": "shard", + "arcnames": sorted(buckets[k])} for k in sorted(buckets)] + return solos + shards + + +def object_fingerprint(kind, arcnames, catalog): + """ + Compact fingerprint of an object's CURRENT content, from the live catalog. + Change detection = recompute this and compare against the manifest's stored + fingerprint for the slot; any difference re-ships the object. + + - solo: {"mtime_s", "size"} of the single file. + - shard: {"members", "byte_sum", "digest"} where digest = shake_128 over the + sorted (arcname, floor_s(mtime), size) triples — the authoritative + change signal (catches add/remove/modify/size, and a same-count/ + same-bytes reshuffle a pure sum would miss). members/byte_sum are + readable cross-checks. + + BLIND SPOT (documented, shared with reclaim): a same-size + same-mtime in-place + rewrite defeats both fingerprints — mitigated by periodic re-hash + --fresh + (RFC §2.3). Floors mtimes to whole seconds (floor_s) so sub-second jitter never + reads as a change. + """ + import hashlib + + if kind == "solo": + mtime_ns, size = catalog[arcnames[0]] + return {"mtime_s": floor_s(mtime_ns), "size": size} + + items = sorted(arcnames) + parts, byte_sum = [], 0 + for a in items: + mtime_ns, size = catalog[a] + parts.append("{}\0{}\0{}".format(a, floor_s(mtime_ns), size)) + byte_sum += size + digest = hashlib.shake_128("\n".join(parts).encode("utf-8")).hexdigest(16) + return {"members": len(items), "byte_sum": byte_sum, "digest": digest} + + +def object_unchanged(current_object, stored_object, catalog): + """ + True iff a routed object can be SKIPPED this run — its live content matches what + the manifest recorded for that slot (the append win + resume, RFC §2.3/§2.8). + + current_object is a fresh route() entry (so its arcnames reflect live membership, + including a small file just appended to a shard); stored_object is the manifest + entry for the same slot (or None at baseline). Compares recomputed vs stored + fingerprint. A brand-new slot (stored_object is None) is never unchanged. + """ + if not stored_object: + return False + fp = object_fingerprint(current_object["kind"], + current_object["arcnames"], catalog) + return fp == stored_object.get("fingerprint") + + +def budget_warnings(objects, catalog, shard_target=SHARD_TARGET_DEFAULT, + max_solo=MAX_SOLO_DEFAULT, max_objects=MAX_OBJECTS_DEFAULT): + """ + Warn-only object-count / shard-size guards (RFC §2.2). Returns a list of + human-readable strings; NEVER raises and never blocks a run — routing proceeds + regardless (the operator raises shard_target / re-baselines in response). + """ + warnings = [] + n_solo = sum(1 for o in objects if o["kind"] == "solo") + n_total = len(objects) + if n_solo > max_solo: + warnings.append("solo objects {} exceed max_solo {}".format(n_solo, max_solo)) + if n_total > max_objects: + warnings.append("total objects {} exceed max_objects_per_target {}" + .format(n_total, max_objects)) + limit = 2 * shard_target + for o in objects: + if o["kind"] != "shard": + continue + b = sum(catalog[a][1] for a in o["arcnames"] if a in catalog) + if b > limit: + warnings.append("shard {} is {} bytes (> 2x shard_target {}) — raise " + "shard_target and re-baseline (--fresh)" + .format(o["slot"], b, shard_target)) + return warnings + + +def vault_dir_for(log_dir): + """ + The per-deployment manifest vault: {log_dir}/_vault. The '_vault' child is + invisible to reclaim.find_logs' non-recursive *.json glob, so manifests never + collide with run-log scanning (RFC §2.4). + """ + import os + + return os.path.join(log_dir, "_vault") + + +def manifest_path(vault_dir, badge): + """Path to a target's manifest file inside the vault.""" + import os + + return os.path.join(vault_dir, "{}.manifest.json".format(badge)) + + +def load_manifest(vault_dir, badge): + """Load a target's manifest dict, or None if it has none yet (baseline).""" + import os + import json + + path = manifest_path(vault_dir, badge) + if not os.path.exists(path): + return None + with open(path) as fh: + return json.load(fh) + + +def write_manifest(vault_dir, badge, manifest): + """ + Atomically write a target's manifest (tmp + os.replace). Callers treat it as + best-effort (a manifest write must never fail an archive already verified on + tape); the atomic replace guarantees a reader never sees a half-written file. + Returns the written path. + """ + import os + import json + + os.makedirs(vault_dir, exist_ok=True) + path = manifest_path(vault_dir, badge) + tmp = path + ".tmp" + with open(tmp, "w") as fh: + json.dump(manifest, fh, indent=2, sort_keys=True) + os.replace(tmp, path) + return path + + # --------------------------------------------------------------------------- # Exclusion primitives (see docs/EXCLUSION_SPEC.md). # @@ -1186,7 +1434,7 @@ def evaluate_exclusions(matched_members, source_folder, spec, def make_zip_files(source_folder, file_pattern, tmp_dir, project_name, compression="deflate", allow_empty_files=False, - exclude_spec=None, exclude_optional=()): + exclude_spec=None, exclude_optional=(), only_arcnames=None): """ On the RCAC compute system: find files in source_folder matching file_pattern (regex), zip them preserving relative paths, verify @@ -1213,6 +1461,12 @@ def make_zip_files(source_folder, file_pattern, tmp_dir, project_name, exclude_optional is the per-asset Tier-2 opt-out list. evaluate_exclusions NEVER raises: any problem yields "back up everything" with a recorded reason. + only_arcnames: optional iterable of relpaths (Phase 2 size-routing, RFC §2.5). + When given, the matched set is narrowed to exactly these members — one routed + object (a solo's single file, or a shard's bucket). The router applies exclusion + LOCALLY before routing, so a routed call passes exclude_spec=None. Default None = + unchanged whole-target behavior; no caller supplies it yet (behavior-neutral). + Returns (zip_path, zip_checksum, file_checksums, members, source_bytes, exclusion) — the 6th element is the exclusion report (status OFF/DISABLED/ON + excluded/demoted/counts) for the run log, banner, and completion email. @@ -1236,11 +1490,22 @@ def make_zip_files(source_folder, file_pattern, tmp_dir, project_name, if pattern.search(arcname): matched_files.append((full_path, arcname)) + # Size-routing (Phase 2, RFC §2.5): when only_arcnames is given, keep just that + # object's members (a solo's one file, or a shard's bucket). The set is already + # exclusion-applied by the LOCAL router, so the caller passes exclude_spec=None + # and the git gate is not re-run per object. No caller passes this yet — routing + # wiring lands in a later slice; here it is inert unless explicitly supplied. + if only_arcnames is not None: + keep = set(only_arcnames) + matched_files = [(fp, arc) for (fp, arc) in matched_files if arc in keep] + if not matched_files: raise RuntimeError( f"No files matched pattern '{file_pattern}' in '{source_folder}' " f"(searched recursively — pattern is tested against relative paths " f"such as 'raw/IMG_001.raw')" + + ("" if only_arcnames is None else + " — after the only_arcnames routing filter (empty object)") ) # Second, subtractive filter (docs/EXCLUSION_SPEC.md): drop git-COVERED diff --git a/tests/test_routing_primitives.py b/tests/test_routing_primitives.py new file mode 100644 index 0000000..6cb5d4e --- /dev/null +++ b/tests/test_routing_primitives.py @@ -0,0 +1,283 @@ +""" +Unit tests for the Phase-2 size-routing primitives (docs/RFC_incremental_v2.md §2, +build-order slice 1). These are pure, additive, deterministic helpers — nothing in +the archive or reclaim flow calls them yet (slice 1 is behavior-neutral), so this +suite is fully self-contained (no tape, no Slurm, no Globus). + +Covers: shard_of stability, slot encoding, derive_shard_count, route() partition at +the T_small boundary, object_fingerprint change detection, object_unchanged (the +skip/append signal), budget warnings, clamp_t_small, manifest round-trip, and the +new (unused) only_arcnames filter on make_zip_files. +""" +import hashlib +import os +import sys +import tempfile +import unittest + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +import archive # noqa: E402 + + +# A small live-style catalog: {arcname: (mtime_ns, size)}. +MiB = 1024 * 1024 +BIG = 300 * MiB # >= 256 MiB default T_small -> solo +SMALL = 4 * MiB # < T_small -> shard + + +def cat(**entries): + """Build a catalog from name=(mtime_s, size) kwargs (mtime given in seconds).""" + return {a: (mt * 1_000_000_000, sz) for a, (mt, sz) in entries.items()} + + +class TestShardOf(unittest.TestCase): + def test_stable_across_calls(self): + self.assertEqual(archive.shard_of("a/b/c.txt", 8), + archive.shard_of("a/b/c.txt", 8)) + + def test_in_range(self): + for i in range(200): + k = archive.shard_of("file_%d.dat" % i, 8) + self.assertTrue(0 <= k < 8) + + def test_distributes(self): + seen = {archive.shard_of("file_%d.dat" % i, 8) for i in range(200)} + self.assertGreater(len(seen), 1) # not all in one bucket + + def test_k1_all_zero(self): + self.assertEqual(archive.shard_of("anything", 1), 0) + + def test_rejects_bad_k(self): + with self.assertRaises(ValueError): + archive.shard_of("x", 0) + + +class TestSlots(unittest.TestCase): + def test_solo_slot_format_and_stable(self): + s = archive.solo_slot("aligned/sample.bam") + self.assertTrue(s.startswith("solo:")) + self.assertEqual(len(s), len("solo:") + 16) # 8-byte hexdigest = 16 chars + self.assertEqual(s, archive.solo_slot("aligned/sample.bam")) + + def test_solo_slot_distinct(self): + self.assertNotEqual(archive.solo_slot("a"), archive.solo_slot("b")) + + def test_shard_slot_format(self): + self.assertEqual(archive.shard_slot(3), "shard:3") + + +class TestDeriveShardCount(unittest.TestCase): + def test_ceil_of_small_bytes(self): + # 3 small files of 100 bytes, shard_target 250 -> ceil(300/250) = 2 + c = cat(a=(1, 100), b=(1, 100), d=(1, 100)) + self.assertEqual(archive.derive_shard_count(c, t_small=BIG, shard_target=250), 2) + + def test_big_files_excluded_from_small_bytes(self): + # one big (solo) + one tiny: only the tiny counts toward K + c = cat(big=(1, BIG), tiny=(1, 10)) + self.assertEqual(archive.derive_shard_count(c, shard_target=archive.SHARD_TARGET_DEFAULT), 1) + + def test_floor_at_one_when_no_small(self): + c = cat(big=(1, BIG)) + self.assertEqual(archive.derive_shard_count(c), 1) + + +class TestRoute(unittest.TestCase): + def test_boundary_solo_vs_shard(self): + # exactly T_small default is a solo (>=); one byte under is a shard + c = cat(at=(1, archive.T_SMALL_DEFAULT), + under=(1, archive.T_SMALL_DEFAULT - 1)) + objs = archive.route(c, shard_count=4) + solos = [o for o in objs if o["kind"] == "solo"] + shards = [o for o in objs if o["kind"] == "shard"] + self.assertEqual([o["arcnames"] for o in solos], [["at"]]) + self.assertEqual(sum(len(o["arcnames"]) for o in shards), 1) + self.assertIn("under", shards[0]["arcnames"]) + + def test_empty_shards_omitted(self): + c = cat(x=(1, SMALL)) # one small file, K=8 -> exactly one shard object + objs = archive.route(c, shard_count=8) + shards = [o for o in objs if o["kind"] == "shard"] + self.assertEqual(len(shards), 1) + + def test_membership_matches_shard_of(self): + c = cat(a=(1, SMALL), b=(1, SMALL), d=(1, SMALL)) + objs = archive.route(c, shard_count=4) + for o in objs: + if o["kind"] == "shard": + k = int(o["slot"].split(":")[1]) + for arc in o["arcnames"]: + self.assertEqual(archive.shard_of(arc, 4), k) + + def test_deterministic_and_frozen_k(self): + c = cat(a=(1, SMALL), b=(1, SMALL), big=(1, BIG)) + self.assertEqual(archive.route(c, shard_count=4), + archive.route(c, shard_count=4)) + + def test_all_members_present_exactly_once(self): + c = cat(a=(1, SMALL), b=(1, BIG), d=(1, SMALL), e=(1, BIG)) + objs = archive.route(c, shard_count=4) + flat = [arc for o in objs for arc in o["arcnames"]] + self.assertEqual(sorted(flat), sorted(c)) + self.assertEqual(len(flat), len(set(flat))) # disjoint + + +class TestFingerprint(unittest.TestCase): + def test_solo_floors_mtime(self): + c = {"f": (1_500_000_000_999_999_999, 10)} # sub-second ns tail + fp = archive.object_fingerprint("solo", ["f"], c) + self.assertEqual(fp, {"mtime_s": 1_500_000_000, "size": 10}) + + def test_shard_shape(self): + c = cat(a=(1, 10), b=(2, 20)) + fp = archive.object_fingerprint("shard", ["a", "b"], c) + self.assertEqual(fp["members"], 2) + self.assertEqual(fp["byte_sum"], 30) + self.assertIn("digest", fp) + + def test_shard_change_signals(self): + base = cat(a=(1, 10), b=(2, 20)) + fp0 = archive.object_fingerprint("shard", ["a", "b"], base) + # size change + self.assertNotEqual(fp0, archive.object_fingerprint( + "shard", ["a", "b"], cat(a=(1, 11), b=(2, 20)))) + # mtime change (whole second) + self.assertNotEqual(fp0, archive.object_fingerprint( + "shard", ["a", "b"], cat(a=(9, 10), b=(2, 20)))) + # membership add + self.assertNotEqual(fp0, archive.object_fingerprint( + "shard", ["a", "b", "c"], cat(a=(1, 10), b=(2, 20), c=(3, 5)))) + # identical -> equal + self.assertEqual(fp0, archive.object_fingerprint("shard", ["a", "b"], base)) + + def test_shard_digest_catches_reshuffle_that_sums_match(self): + # remove b(20) add c(20): same members count + same byte_sum, but the + # digest (over arcnames) must still differ. + fp0 = archive.object_fingerprint("shard", ["a", "b"], cat(a=(1, 10), b=(1, 20))) + fp1 = archive.object_fingerprint("shard", ["a", "c"], cat(a=(1, 10), c=(1, 20))) + self.assertEqual(fp0["members"], fp1["members"]) + self.assertEqual(fp0["byte_sum"], fp1["byte_sum"]) + self.assertNotEqual(fp0["digest"], fp1["digest"]) + + +class TestObjectUnchanged(unittest.TestCase): + def _stored(self, kind, arcnames, catalog): + return {"slot": "s", "kind": kind, "arcnames": arcnames, + "fingerprint": archive.object_fingerprint(kind, arcnames, catalog)} + + def test_baseline_none_is_changed(self): + c = cat(f=(1, BIG)) + cur = {"kind": "solo", "arcnames": ["f"]} + self.assertFalse(archive.object_unchanged(cur, None, c)) + + def test_unchanged_solo_skips(self): + c = cat(f=(1, BIG)) + stored = self._stored("solo", ["f"], c) + cur = {"kind": "solo", "arcnames": ["f"]} + self.assertTrue(archive.object_unchanged(cur, stored, c)) + + def test_solo_mtime_bump_is_changed(self): + stored = self._stored("solo", ["f"], cat(f=(1, BIG))) + cur = {"kind": "solo", "arcnames": ["f"]} + self.assertFalse(archive.object_unchanged(cur, stored, cat(f=(2, BIG)))) + + def test_shard_append_is_changed(self): + # stored shard had {a}; a new small file b was appended to the same shard. + stored = self._stored("shard", ["a"], cat(a=(1, SMALL))) + cur = {"kind": "shard", "arcnames": ["a", "b"]} + self.assertFalse(archive.object_unchanged(cur, stored, cat(a=(1, SMALL), b=(1, SMALL)))) + + def test_shard_unchanged_skips(self): + c = cat(a=(1, SMALL), b=(2, SMALL)) + stored = self._stored("shard", ["a", "b"], c) + cur = {"kind": "shard", "arcnames": ["a", "b"]} + self.assertTrue(archive.object_unchanged(cur, stored, c)) + + +class TestBudgetWarnings(unittest.TestCase): + def test_clean_is_silent(self): + c = cat(a=(1, SMALL), big=(1, BIG)) + objs = archive.route(c, shard_count=2) + self.assertEqual(archive.budget_warnings(objs, c), []) + + def test_max_solo_exceeded(self): + c = cat(**{"f%d" % i: (1, BIG) for i in range(5)}) + objs = archive.route(c, shard_count=1) + w = archive.budget_warnings(objs, c, max_solo=3) + self.assertTrue(any("max_solo" in s for s in w)) + + def test_oversize_shard(self): + c = cat(a=(1, SMALL), b=(1, SMALL)) + objs = archive.route(c, shard_count=1) # both small files into one shard + w = archive.budget_warnings(objs, c, shard_target=1) # 2x = 2 bytes; way over + self.assertTrue(any("shard" in s and "shard_target" in s for s in w)) + + +class TestClampTSmall(unittest.TestCase): + def test_raises_to_floor(self): + self.assertEqual(archive.clamp_t_small(1), archive.T_SMALL_FLOOR) + + def test_keeps_above_floor(self): + self.assertEqual(archive.clamp_t_small(archive.T_SMALL_DEFAULT), + archive.T_SMALL_DEFAULT) + + +class TestManifest(unittest.TestCase): + def test_round_trip_and_missing(self): + with tempfile.TemporaryDirectory() as d: + vault = archive.vault_dir_for(d) + self.assertIsNone(archive.load_manifest(vault, "abc123")) + m = {"schema_version": 1, "badge": "abc123", "objects": []} + path = archive.write_manifest(vault, "abc123", m) + self.assertTrue(path.endswith("abc123.manifest.json")) + self.assertEqual(archive.load_manifest(vault, "abc123"), m) + + def test_write_leaves_no_tmp(self): + with tempfile.TemporaryDirectory() as d: + vault = archive.vault_dir_for(d) + archive.write_manifest(vault, "b", {"x": 1}) + self.assertEqual([f for f in os.listdir(vault) if f.endswith(".tmp")], []) + + def test_vault_dodges_reclaim_glob(self): + # manifests live in a _vault child, not directly in log_dir + with tempfile.TemporaryDirectory() as d: + archive.write_manifest(archive.vault_dir_for(d), "b", {"x": 1}) + self.assertEqual([f for f in os.listdir(d)], ["_vault"]) + + +class TestOnlyArcnamesFilter(unittest.TestCase): + def _make_source(self, d): + for rel, content in [("a.txt", b"aaa"), ("b.txt", b"bbbb"), + ("sub/c.txt", b"ccc-c")]: + p = os.path.join(d, rel) + os.makedirs(os.path.dirname(p), exist_ok=True) + with open(p, "wb") as fh: + fh.write(content) + + def test_none_archives_all(self): + with tempfile.TemporaryDirectory() as src, tempfile.TemporaryDirectory() as tmp: + self._make_source(src) + _, _, checksums, members, _, _ = archive.make_zip_files( + src, ".*", tmp, "proj") + self.assertEqual(set(members), {"a.txt", "b.txt", "sub/c.txt"}) + self.assertEqual(set(checksums), set(members)) + + def test_narrows_to_subset_with_matching_keys(self): + with tempfile.TemporaryDirectory() as src, tempfile.TemporaryDirectory() as tmp: + self._make_source(src) + subset = {"a.txt", "sub/c.txt"} + _, _, checksums, members, _, _ = archive.make_zip_files( + src, ".*", tmp, "proj", only_arcnames=subset) + self.assertEqual(set(members), subset) + self.assertEqual(set(checksums), subset) # invariant #2: keys == arcnames + + def test_empty_object_raises(self): + with tempfile.TemporaryDirectory() as src, tempfile.TemporaryDirectory() as tmp: + self._make_source(src) + with self.assertRaises(RuntimeError): + archive.make_zip_files(src, ".*", tmp, "proj", + only_arcnames={"does/not/exist"}) + + +if __name__ == "__main__": + unittest.main() From 8cb9232a9818b4e49011494f94730ab9526e026d Mon Sep 17 00:00:00 2001 From: "Doucette, Jarrod S" Date: Wed, 8 Jul 2026 10:40:28 -0400 Subject: [PATCH 5/5] docs(rfc): manifest must be shared+durable (Depot log_dir), not home/scratch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Jarrod review calls: - K = derived-and-frozen (confirmed; already in derive_shard_count). - Manifest location: it is durable cross-run state reclaim must find, so it MUST live on shared+durable storage (the deployment Depot logs dir that log_dir already points at), NEVER home (the current default trap) or scratch (purged -> reclaim can't reconstruct the aggregate -> every routed target false-DRIFTs). Add a slice-2 durability guard in load_config; tmp_dir (staging zips) stays in scratch. - --globus: reframed. Real invariant = manifest on a shared log_dir readable by both the run and reclaim, which Negishi (the only launch host) satisfies either way; the split only happens off-box, which we don't do. Globus is off on Negishi now, so size_routing stays local-only as a cheap guard (untested stateful path), not a deep limitation — allowable under --globus later with the durability guard in place. §2.4 manifest-location invariant + guard; §2.8 --globus reframe; §2.10 slice-2 guard. Co-Authored-By: Claude Opus 4.8 --- docs/RFC_incremental_v2.md | 50 ++++++++++++++++++++++++++++++-------- 1 file changed, 40 insertions(+), 10 deletions(-) diff --git a/docs/RFC_incremental_v2.md b/docs/RFC_incremental_v2.md index cd88a21..d3d1850 100644 --- a/docs/RFC_incremental_v2.md +++ b/docs/RFC_incremental_v2.md @@ -195,6 +195,25 @@ so routed run logs and the manifest never collide with legacy scanning. Written The manifest is the **aggregate index**; per-file MD5 stays in each object's own run-log JSON (invariant #2/#4 preserved per object). +**The manifest must live on SHARED, DURABLE storage** — the same deployment logs +directory the run logs already use (FloraSense: `/depot/florasense/etc/logs_fortress/…`), +which `log_dir` points at. It is long-lived cross-run state that a later `reclaim` +(possibly a different session/user/cron) must find and that `verify_target` (§2.6) +reconstructs the aggregate from. Two locations are **forbidden** for a routed target: + +- **Home** — per-user, not shared, quota-bound — and it is the *current default* + (`log_dir` falls back to `tmp_dir` = `~/globus_archive_tmp`), a latent trap. +- **Scratch** — purged on RCAC's retention timer. A purged manifest → `verify_target` + cannot reconstruct the aggregate → **every routed target false-DRIFTs** (or worse, a + wrong verdict). Categorically wrong for durable state. + +**Durability guard (slice 2):** when `size_routing` is on, `load_config` **refuses** at +startup if the resolved `log_dir` is under `$HOME`, matches a scratch path +(`/scratch/…`, `$RCAC_SCRATCH`/`$SCRATCH`), or is the `~/globus_archive_tmp` default — +with a message naming the shared Depot path to use. Only `log_dir`/manifest must be +durable+shared; **`tmp_dir` (the multi-TB staging zips) stays in scratch** — the correct +home for ephemeral bytes. + ```json { "schema_version": 1, @@ -351,11 +370,18 @@ demands it — matching how Phase 2 itself was gated on Round-2 data. ### 2.8 Interactions + invariants -- **`--globus` — refused when `size_routing: true`.** Routing + manifest are LOCAL - (`log_dir`/manifest on the launching host); under `--globus` `log_dir` may be on a - different host and manifest/logs diverge (the Phase-3 §3.5.3 concern). Exit with a - clear error naming the config key. (Phase-1 exclusion still runs under `--globus` - unchanged — only routing is local-only.) +- **`--globus` — refused when `size_routing: true` (for now).** The real invariant is + not "local vs globus" — it is that the manifest must sit on a shared, durable + `log_dir` reachable by **both** the archive run (`__main__`, which reads/writes it) and + a later `reclaim` (§2.4). Launched from **Negishi** — the actual and only production + launch host — that holds under either mode (the endpoint is on the same RCAC + filesystem). The manifest/logs split only occurs when `__main__` runs *off-box* (a Mac + with no `/depot`), which is not a workflow we use. So routing stays **local-only for + now** purely because the stateful-under-Globus path is untested and Globus is disabled + on Negishi today — a cheap guard, not a deep limitation. If Globus is ever enabled + there, `size_routing` can be allowed under `--globus` with a small change (require the + shared `log_dir`, which the durability guard already enforces). `load_config` exits + with a clear message. (Phase-1 exclusion still runs under `--globus` unchanged.) - **Resume = skip-unchanged (no new machinery).** A routed target does **not** use `find_existing_zip`/the stale-zip guard. A failed run re-runs; objects already on tape (fingerprint recorded in the manifest) skip; only the unfinished remainder ships. A @@ -449,11 +475,15 @@ before merging (suite 136 → growing). Tests slot beside the existing ones `only_arcnames` narrows to the subset with keys matching `source_files` (invariant #2), manifest round-trip + no leftover `.tmp` + `_vault/` dodges the reclaim glob. - **Slice 2 — archive wiring (the append engine).** `size_routing`/`t_small`/ - `shard_count`/`shard_target` config keys; `route_and_ship` in `__main__`; skip-unchanged; - drop-missing-slots; `--globus` refusal; `--fresh` re-baseline; run-log/manifest/email - reporting. Tests: baseline ships all objects; append ships only new solos (existing - skip); one big file changed → one solo re-ships; shard member change → whole shard - re-ships; deleted big file → slot dropped; `--globus` refused; `--fresh` re-baselines. + `shard_count`/`shard_target` config keys; the **durability guard** (§2.4: refuse + `size_routing` if `log_dir` is under `$HOME`/scratch/the home default); `route_and_ship` + in `__main__` (exclusion once → route survivors → ship changed objects, K derived at + baseline then read frozen from the manifest); skip-unchanged; drop-missing-slots; + `--globus` refusal; `--fresh` re-baseline; run-log/manifest/email reporting. Tests: + baseline ships all objects; append ships only new solos (existing skip); one big file + changed → one solo re-ships; shard member change → whole shard re-ships; deleted big + file → slot dropped; **durability guard rejects home/scratch log_dir**; `--globus` + refused; `--fresh` re-baselines with a fresh K. - **Slice 3 — aggregate reclaim (REQUIRED with the first routed ship, §2.6).** `verify_target` + badge grouping in the scan loop; object logs excluded from legacy `verify()`; INV-E re-gate on the aggregate; every-tar-present gate. Tests: a routed