diff --git a/CLAUDE.md b/CLAUDE.md index 7941531..9f9bd8e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -324,6 +324,9 @@ Personal auth (endpoint owner) never needs `allowed_principals`. The local | `t_small` | No | 256 MiB | Solo/shard cutoff in bytes (hard floor 100 MB). Files ≥ `t_small` ship as their own object. | | `shard_count` | No | `null` | Shard count K. `null` = derive from `shard_target` at baseline. **Frozen in the manifest** once a baseline exists — changing it needs `--fresh`. | | `shard_target` | No | 256 GiB | Target shard size (bytes) used to derive K at baseline. | +| `incremental` | No | `false` | Phase-3 leveled incrementals (L0 baseline + delta levels — see the Leveled-incremental section and `docs/RFC_incremental_v2.md` §3). **Local mode only** (refuses `--globus`); mutually exclusive with `size_routing`; requires a SHARED, DURABLE `log_dir` (same guard as `size_routing`). Manifest lives at `{log_dir}/_vault/{badge}.leveled.manifest.json`. | + +`--full` (CLI flag, not a config key): force a fresh L0 baseline on an `incremental` target regardless of the (mtime,size) diff — the operational safety valve for the §3.1 same-size/same-mtime blind spot (see the Leveled-incremental section). See `config.example.json` for an annotated template. @@ -447,6 +450,86 @@ archives byte-for-byte as before. Authoritative design: `docs/RFC_incremental_v2 --- +## Leveled incremental (Phase 3) — L0 baseline + delta levels + +Opt-in per config (`"incremental": true`) and **default-off**, mutually exclusive +with `size_routing` (a badge wears exactly one manifest schema). Authoritative +design: `docs/RFC_incremental_v2.md` §3 (all 3 build slices — engine, reclaim, +restore — merged). + +- Ships an **L0 baseline** (all post-exclusion survivors) once, then only the + **(mtime,size)-diffed delta** on every later run: added ∪ modified files ship + as a new **delta level**; removed files are recorded as metadata-only + `deleted` entries (no tar). A run with nothing added/removed/modified is a + **true NO-OP** — nothing ships, no new level node. +- **HARD INVARIANT (§3.1)**: a same-size, same-mtime **in-place content + rewrite is invisible** to the cheap diff and will NOT be captured in any + delta — re-hashing would fix this but defeats the point of the feature. A + true NO-OP always emails loudly with the exact wording *"detected zero + metadata change; same-size in-place content edits are NOT captured. Use + `--full` to force a fresh L0 if you rewrote files preserving mtime+size."* + This email is intentionally **not** paired with the T1 Teams bell + (`notify_bell`) — a NO-OP is a normal, potentially daily outcome, not a + failure; the bell stays reserved for `[FAILED]`/`[STALE]`/ + `[EXCLUSION-ABORT]`. `--full` forces a fresh L0 baseline (a new, + independent chain — the old L0 + its deltas remain in the manifest, + superseded but not deleted) regardless of the diff, as the periodic safety + valve. +- **Per-badge advisory lock** (`badge_lock`, `fcntl.flock`) is held across the + ENTIRE decide+ship+manifest-write critical section — not just the manifest + write — closing the read-modify-write race a narrower lock would leave open. + A failed ship writes no manifest state; a retry re-runs the same decision + from scratch and re-zips fresh (a delta's member set is small, so this is + cheap by construction). +- **Level-aware resume (§3.5.1)**: only an **L0** ship attempts + `find_existing_zip`'s resume (a delta's zip is never a complete source + snapshot, so it always re-zips fresh, never resuming) — the L0 zip is + trusted only if its member count still matches the live post-exclusion + survivor set AND no survivor's mtime is newer than the zip's own timestamp + (`archive.l0_zip_is_stale`), mirroring the whole-target guard's exact + semantics without re-walking the source a second time. +- **Per-target manifest** at `{log_dir}/_vault/{badge}.leveled.manifest.json` + (kept apart from a Phase-2 `{badge}.manifest.json` for the same badge by the + `kind=` filename split — a badge mid-conversion can have both on disk at + once without either clobbering the other). Each level node records + `blonde`/`level`/`parent_blonde`/`fortress_tar`/`tar_bytes`/`n_files`/ + `source_bytes`/`catalog_ref`/`deleted`. `catalog_ref` points at a sibling + JSON file (not inlined) holding that level's own `{arcname: [mtime_ns, + size]}` subset — kept lean even for a 36k-file L0. +- **`archive.effective_catalog(manifest, log_dir)`** folds the current chain + (the L0 node matching `current_l0_blonde`, plus deltas whose + `parent_blonde` points at it) into one `{arcname: (mtime_ns, size)}` view — + the single shared implementation `decide_level`, `reclaim.verify_chain`, and + `restore.select_chain`/`fold_expected` all replay, so all three always agree + on "current state." An OLDER, superseded chain left behind by a `--full` + rebaseline is deliberately excluded from this fold (and from reclaim's + tape-completeness check) — it is not part of what gets restored. +- **Reclaim** verifies a leveled target as an AGGREGATE: + `reclaim.verify_chain()` replays `effective_catalog`, requires every + **current-chain** tar present on Fortress (superseded-chain tars are exempt + — see above), and re-runs the INV-E exclusion re-gate. `--scan` picks up + leveled manifests automatically (leveled per-level/NO-OP run-logs are + skipped by the legacy per-log `verify()`); the single-config invocation + (`python reclaim.py config.json`) routes an `"incremental": true` config the + same way. +- **Restore** (`restore.py`, LOCAL, operator-invoked): full-chain or + point-in-time (`--to-timestamp`) restore into an **empty** dest, replaying + levels in strict run-time order as a file OVERLAY plus per-level deletion + replay. Gates, in order: chain completeness (every level tar on tape BEFORE + any transfer), empty dest, per-level extract + per-member MD5 vs that + level's own run-log, and a mandatory final reconciliation against + `effective_catalog`. No `--slot`/`--member` selective restore for leveled + targets (only full or `--to-timestamp`). + + ```bash + python3 archive.py config.json # normal incremental run (L0 or delta) + python3 archive.py --full config.json # force a fresh L0 rebaseline + python3 restore.py config.json --dest DIR # full chain restore + python3 restore.py config.json --dest DIR --to-timestamp 20260601_000000 # point-in-time + ``` + +--- + ## Known issues / history - **Unbounded subprocess hang (32h+ on Negishi):** `repository_X1D_3_metabolomics_rawspectra` diff --git a/archive.py b/archive.py index 082206a..f2429e9 100644 --- a/archive.py +++ b/archive.py @@ -26,6 +26,17 @@ The default was inverted on 2026-05-29: the bare command now runs LOCAL (no Globus Compute); Globus Compute is opt-in via --globus. The active mode is printed as a loud banner at startup. + +HARD INVARIANT (Phase 3, leveled incrementals, docs/RFC_incremental_v2.md §3.1): +a same-size, same-mtime IN-PLACE content rewrite is INVISIBLE to decide_level's +cheap (mtime,size) diff and will NOT be captured in any delta — this is not a +bug to be fixed by re-hashing (the exact cost this feature exists to avoid), +it is a permanent property of the design. Mitigation: a true NO-OP (nothing +added/removed/modified) always emails loudly with the exact wording "detected +zero metadata change; same-size in-place content edits are NOT captured. Use +`--full` to force a fresh L0 if you rewrote files preserving mtime+size." A +periodic `--full` re-baseline is the only operational safety valve. Never +silence or soften this NO-OP email. """ import json @@ -154,19 +165,38 @@ def load_config(config_path): f"config 'shard_target' must be a positive integer (bytes), " f"got: {config['shard_target']!r}") - # Durability guard (RFC §2.4): a routed target keeps a per-target manifest at - # {log_dir}/_vault/ that MUST outlive runs and be shared, so reclaim can find it. - # Refuse a log_dir under $HOME / scratch / the home default — home is per-user and - # scratch is purged (a lost manifest makes every routed target false-DRIFT). - if config["size_routing"]: + # Phase-3 leveled incremental (docs/RFC_incremental_v2.md §3). Opt-in per config + # and default-off, so an unset config is unaffected. MUTUALLY EXCLUSIVE with + # size_routing: a badge wears exactly one manifest schema (Phase 2's flat + # objects[] vs Phase 3's levels[] star), and the kind= param on + # manifest_path/load_manifest/write_manifest keeps the two files apart on disk + # (routing vs leveled) — but running both against the SAME target would be a + # genuine config error, not just a naming collision, so reject it up front. + config.setdefault("incremental", False) + if not isinstance(config["incremental"], bool): + raise ValueError( + f"config 'incremental' must be true or false, got: {config['incremental']!r}") + if config["size_routing"] and config["incremental"]: + raise ValueError( + "config 'size_routing' and 'incremental' cannot both be true — Phase 2 " + "(size-routing) and Phase 3 (leveled incremental) are mutually exclusive " + "schemes for a target (docs/RFC_incremental_v2.md §3). Pick one.") + + # Durability guard (RFC §2.4 / §3.2): EITHER routing scheme keeps a per-target + # manifest at {log_dir}/_vault/ that MUST outlive runs and be shared, so reclaim + # can find it. Refuse a log_dir under $HOME / scratch / the home default — home + # is per-user and scratch is purged (a lost manifest makes every routed/leveled + # target false-DRIFT). routing_log_dir_ok is reused verbatim for both schemes. + if config["size_routing"] or config["incremental"]: ok, why = routing_log_dir_ok(config["log_dir"]) if not ok: raise ValueError( - f"size_routing needs log_dir on SHARED, DURABLE storage, but it points at " - f"{why} (log_dir={config['log_dir']!r}). The per-target manifest lives at " - f"{{log_dir}}/_vault/ and must outlive runs so reclaim can rebuild the " - f"aggregate. Point log_dir at the deployment's shared Depot logs dir " - f"(e.g. /depot//etc/logs_fortress/...). tmp_dir may stay in scratch.") + f"size_routing/incremental need log_dir on SHARED, DURABLE storage, but " + f"it points at {why} (log_dir={config['log_dir']!r}). The per-target " + f"manifest lives at {{log_dir}}/_vault/ and must outlive runs so reclaim " + f"can rebuild the aggregate. Point log_dir at the deployment's shared " + f"Depot logs dir (e.g. /depot//etc/logs_fortress/...). tmp_dir " + f"may stay in scratch.") return config @@ -443,6 +473,47 @@ def try_reconstruct(existing_zip, runner): return None +def l0_zip_is_stale(resumed_member_count, expected_member_count, + newest_survivor_mtime_s, existing_zip_path): + """ + Staleness check for resuming an existing L0 leveled-incremental zip (RFC + §3.5.1 / §3.5's L0 resume). Same failure-count/newer-than-zip semantics as + the whole-target Step 1 resume guard, but computed from the (mtime,size) + survivor catalog already in hand (the exact set decide_level diffed + against for this L0) rather than re-walking the source a second time: + + - STALE if the resumed zip's actual member count no longer matches the + current post-exclusion survivor count (files added/removed, or an + exclude_spec change shifted the survivor set). + - STALE if any survivor's mtime (floor_s'd, matching decide_level's own + precision) is newer than the zip's own timestamp (parsed from its + "{stem}_{YYYYMMDD_HHMMSS}.zip" filename) — a file was modified after + the zip was built. + + `newest_survivor_mtime_s` is the max floor_s(mtime_ns) over the live + survivor catalog (or None for an empty catalog — never STALE on that + basis, count mismatch would already have caught an empty-vs-nonempty + case). Returns False (safe to resume) if the zip's timestamp can't be + parsed from its filename — the count check alone still applies. + """ + import re + import datetime + + if resumed_member_count != expected_member_count: + return True + + m = re.search(r"_(\d{8})_(\d{6})\.zip$", os.path.basename(existing_zip_path)) + if not m: + return False + try: + zip_ts = datetime.datetime.strptime( + m.group(1) + m.group(2), "%Y%m%d%H%M%S").timestamp() + except ValueError: + return False + + return newest_survivor_mtime_s is not None and newest_survivor_mtime_s > zip_ts + + def enumerate_source(source_folder, file_pattern): """ Enumerate the live source EXACTLY as make_zip_files would, but without @@ -1113,26 +1184,43 @@ def vault_dir_for(log_dir): return os.path.join(log_dir, "_vault") -def manifest_path(vault_dir, badge): - """Path to a target's manifest file inside the vault.""" +def manifest_path(vault_dir, badge, kind="routing"): + """ + Path to a target's manifest file inside the vault. + + kind="routing" (default) preserves the exact pre-Phase-3 filename/behavior: + {badge}.manifest.json — Phase 2's flat objects[] schema. kind="leveled" is + Phase 3's separate file — {badge}.leveled.manifest.json — so converting a + badge from size-routing to leveled incrementals (or running both, though + load_config forbids that per-target) never clobbers the other schema's + manifest: badge_of() keys only on (project, source_folder, file_pattern), so + without this the two phases would otherwise fight over the same path (see + docs/RFC_incremental_v2.md §3, the manifest-collision fix). Both filenames + still match reclaim.find_manifests()'s existing `_vault/*.manifest.json` glob + ('*' matches embedded dots), so that function needs no change. + """ import os - return os.path.join(vault_dir, "{}.manifest.json".format(badge)) + if kind == "routing": + name = "{}.manifest.json".format(badge) + else: + name = "{}.{}.manifest.json".format(badge, kind) + return os.path.join(vault_dir, name) -def load_manifest(vault_dir, badge): +def load_manifest(vault_dir, badge, kind="routing"): """Load a target's manifest dict, or None if it has none yet (baseline).""" import os import json - path = manifest_path(vault_dir, badge) + path = manifest_path(vault_dir, badge, kind) if not os.path.exists(path): return None with open(path) as fh: return json.load(fh) -def write_manifest(vault_dir, badge, manifest): +def write_manifest(vault_dir, badge, manifest, kind="routing"): """ Atomically write a target's manifest (tmp + os.replace). Callers treat it as best-effort (a manifest write must never fail an archive already verified on @@ -1143,7 +1231,7 @@ def write_manifest(vault_dir, badge, manifest): import json os.makedirs(vault_dir, exist_ok=True) - path = manifest_path(vault_dir, badge) + path = manifest_path(vault_dir, badge, kind) tmp = path + ".tmp" with open(tmp, "w") as fh: json.dump(manifest, fh, indent=2, sort_keys=True) @@ -1421,6 +1509,462 @@ def update_reship_watch(prior_watch, stats, run_epoch, } +# --------------------------------------------------------------------------- +# Phase 3 — leveled incremental (L0 full baseline + L1/L2… deltas), RFC §3. +# +# HARD INVARIANT (§3.1, not a footnote): same-size, same-mtime in-place content +# rewrites are INVISIBLE to the cheap (mtime,size) diff decide_level runs. The +# byte-sum check below additionally catches a size-CHANGING in-place rewrite +# that the per-arcname add/removed/modified sets alone would miss, forcing a +# fresh full-content delta rather than a silent no-op — but a rewrite that +# preserves BOTH size and mtime defeats every check simultaneously and is a +# genuine, documented blind spot (the same one reclaim.py already lives with). +# The NO-OP path is the mitigation of last resort: it emails LOUDLY so an +# operator periodically re-baselines with --full rather than trusting a quiet +# run. All level-decision/diff/manifest/lock logic here runs LOCAL only, in +# __main__ — never serialized to a Globus Compute endpoint (invariant #1); +# --globus + incremental is refused up front in __main__ for exactly that +# reason (a leveled target's manifest and catalog live on the launching host). +# --------------------------------------------------------------------------- + + +def effective_catalog(manifest, log_dir, upto_quaver=None): + """ + Fold a leveled manifest's L0 + its deltas into ONE (mtime,size) catalog — + "what does the source look like after replaying every level to date." + Single implementation reused by decide_level, reclaim's verify_chain (P3-2), + and restore's fold_expected (P3-3), so all three agree on "current state." + + Returns (catalog, total_bytes): + catalog — {arcname: (mtime_ns, size)}, L0 ∪ deltas with add/modify + applied and deletions removed, in run-time (quaver) order. + total_bytes — sum(size for size in catalog.values()); NOT independently + tracked — always exactly consistent with `catalog` (see + decide_level's byte-sum guard docstring for why this makes + that specific check unreachable via decide_level's own + per-arcname diff, and is intentionally still performed). + + upto_quaver: optional epoch-seconds cutoff (decoded from each delta's + BLONDE) for a point-in-time fold — only deltas with when_epoch <= + upto_quaver are applied. None (default) = fold the full chain to date. + + Each level node's full/delta catalog is stored in a SIBLING file + (catalog_ref, relative to log_dir) rather than inline in the manifest, so + the manifest itself stays lean even for a 36k-file L0 (§3.2). A missing or + unreadable catalog file degrades to "contributes nothing" rather than + raising — the manifest is best-effort/rebuildable-from-logs, never a hard + dependency for reading it back. + """ + import os + import json + + empty = ({}, 0) + if not manifest or not manifest.get("levels"): + return empty + + levels = manifest["levels"] + l0_blonde = manifest.get("current_l0_blonde") + l0 = next((n for n in levels + if n.get("level") == 0 and n.get("blonde") == l0_blonde), None) + if l0 is None: + # Defensive fallback for a manifest missing/stale current_l0_blonde: + # the most recently appended level==0 node. + zeros = [n for n in levels if n.get("level") == 0] + l0 = zeros[-1] if zeros else None + if l0 is None: + return empty + + def _epoch(node): + return decode_blonde(node["blonde"])["when_epoch"] + + deltas = [n for n in levels + if n.get("level", 0) > 0 and n.get("parent_blonde") == l0["blonde"]] + if upto_quaver is not None: + deltas = [n for n in deltas if _epoch(n) <= upto_quaver] + deltas.sort(key=_epoch) + + def _load(node): + ref = node.get("catalog_ref") + if not ref: + return {} + try: + with open(os.path.join(log_dir, ref)) as fh: + raw = json.load(fh) + except (OSError, ValueError): + return {} + return {a: (v[0], v[1]) for a, v in raw.items()} + + catalog = dict(_load(l0)) + for node in deltas: + for d in node.get("deleted") or []: + catalog.pop(d, None) + catalog.update(_load(node)) + + total_bytes = sum(v[1] for v in catalog.values()) + return catalog, total_bytes + + +def _next_delta_detail(manifest, l0_blonde): + """ + Next delta level number (as the BLONDE `detail` string) for the chain + rooted at l0_blonde: max existing delta level + 1, or "1" if this is the + first delta off this L0. Deltas are a STAR off one L0 (§3.3) — only nodes + with parent_blonde == l0_blonde count, so an older, now-superseded chain's + level numbers never leak into this one's numbering. + """ + deltas = [n for n in manifest.get("levels", []) + if n.get("parent_blonde") == l0_blonde and n.get("level", 0) > 0] + if not deltas: + return "1" + return str(max(n["level"] for n in deltas) + 1) + + +def decide_level(badge, live_catalog, manifest, force_full, log_dir): + """ + Per RFC §3.2/§3.7 — the LOCAL level decision. Returns one of: + + dict(detail, parent, members, deleted[, forced_rebaseline]) + — ship this: detail is the BLONDE level string ("0" for a + baseline/forced rebaseline, "1","2",… for a delta); + parent is the L0 blonde this delta hangs off (None for + L0 itself); members is the sorted arcname list to zip + + MD5 (added ∪ modified for a delta, ALL live arcnames for + a baseline/forced-rebaseline); deleted is the sorted + removed-arcname list (metadata only — never shipped). + None — true NO-OP: nothing added/removed/modified AND the live + byte-sum matches the effective catalog's. Ship nothing. + + force_full (--full) or no live+present L0 in the manifest -> baseline (L0). + Otherwise diffs live_catalog against effective_catalog(manifest, log_dir): + added/removed by arcname set membership; modified by floor_s(mtime) OR size + differing for a shared arcname (floor_s per RFC §3.2 — sub-second jitter + must never manufacture a phantom "modified"). If the arcname-level diff is + completely empty, the byte-sum guard (mirrors reclaim.py's drift check) + still compares live total bytes against the effective catalog's total: by + construction this specific diff already compares per-arcname size, so a + mismatch here can only arise from an inconsistent catalog/manifest state, + not a live content change the diff itself would have already caught — kept + as defense-in-depth (mirroring reclaim.py's own belt-and-suspenders use of + the same check) rather than assumed impossible. A genuine mismatch forces a + full-content delta (ships everything, still a delta node — not a new L0) + rather than silently declaring NO-OP. + """ + if force_full or not manifest or not manifest.get("levels"): + return {"detail": "0", "parent": None, + "members": sorted(live_catalog), "deleted": []} + + l0_blonde = manifest.get("current_l0_blonde") + prior, prior_bytes = effective_catalog(manifest, log_dir) + + added = [a for a in live_catalog if a not in prior] + removed = [a for a in prior if a not in live_catalog] + modified = [a for a in live_catalog if a in prior and + (floor_s(live_catalog[a][0]) != floor_s(prior[a][0]) + or live_catalog[a][1] != prior[a][1])] + + if not (added or removed or modified): + live_bytes = sum(v[1] for v in live_catalog.values()) + if live_bytes != prior_bytes: + return {"detail": _next_delta_detail(manifest, l0_blonde), + "parent": l0_blonde, "members": sorted(live_catalog), + "deleted": [], "forced_rebaseline": True} + return None # true NO-OP — caller must email loudly (§3.1) + + return {"detail": _next_delta_detail(manifest, l0_blonde), "parent": l0_blonde, + "members": sorted(set(added) | set(modified)), "deleted": sorted(removed)} + + +def write_level_catalog(log_dir, run_timestamp, catalog_subset): + """ + Atomically write a level's sibling catalog file (§3.2/§3.3) — + {log_dir}/{run_timestamp}.catalog.json — the full (mtime,size) catalog for + an L0, or just the added∪modified subset for a delta. Keeps the manifest + itself lean (no inline 36k-entry dict) while remaining reconstructable by + effective_catalog. Returns (catalog_ref, catalog_sha256): + + catalog_ref — the bare filename (relative to log_dir; the manifest + stores this, never a host-specific absolute path). + catalog_sha256 — sha256 of the exact bytes written, for the manifest + node's catalog_sha256 field (a tamper/corruption check, + not consulted by effective_catalog itself). + + catalog_subset values may be either (mtime_ns, size) tuples or 2-element + lists (already round-tripped through JSON); both serialize identically. + """ + import os + import json + import hashlib + + os.makedirs(log_dir, exist_ok=True) + ref = "{}.catalog.json".format(run_timestamp) + path = os.path.join(log_dir, ref) + tmp = path + ".tmp" + payload = {a: [v[0], v[1]] for a, v in sorted(catalog_subset.items())} + data = json.dumps(payload, indent=2, sort_keys=True).encode("utf-8") + with open(tmp, "wb") as fh: + fh.write(data) + os.replace(tmp, path) + catalog_sha256 = hashlib.sha256(data).hexdigest() + return ref, catalog_sha256 + + +def record_noop_event(manifest, now_epoch, run_timestamp): + """ + Append a NO-OP event to the manifest's `events` list (mutates + returns + `manifest`) — so a manifest read later can see that a run happened and + shipped nothing, not just silence. No level node is appended (a NO-OP is + not a level); reclaim's verify_chain (P3-2) never sees these as gaps + because there is no new node to reconcile against. + """ + manifest.setdefault("events", []) + manifest["events"].append({ + "type": "noop", "epoch": now_epoch, "run_timestamp": run_timestamp, + }) + return manifest + + +def write_noop_log(log_dir, project_name, run_timestamp, badge, live_catalog, source_bytes): + """ + Write a NO-OP run-log pair ({run_timestamp}.txt/.json), tagged + `"status": "noop"` + `"leveled_object": True` (mirrors ship_object's + `"routed_object"` tag) so reclaim's --scan loop (P3-2) recognizes and skips + it rather than treating a shipless run as a broken/incomplete archive. + """ + import os + import json + import datetime + + os.makedirs(log_dir, exist_ok=True) + record = { + "project": project_name, "run_timestamp": run_timestamp, + "status": "noop", "leveled_object": True, "badge": badge, + "n_files": len(live_catalog), "source_bytes": source_bytes, + "timestamp": datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"), + } + txt_path = os.path.join(log_dir, "{}.txt".format(run_timestamp)) + json_path = os.path.join(log_dir, "{}.json".format(run_timestamp)) + with open(json_path, "w") as fh: + json.dump(record, fh, indent=2) + with open(txt_path, "w") as fh: + fh.write( + "Leveled incremental NO-OP for {}\n" + "Badge: {}\n" + "Live files: {} Live bytes: {}\n" + "Nothing added, removed, or modified since the last checkpoint — " + "no delta shipped.\n".format( + project_name, badge, len(live_catalog), source_bytes)) + return txt_path, json_path + + +class badge_lock: + """ + Advisory per-badge lock (§3.2) — blocking `fcntl.flock` on + `{vault_dir}/{badge}.lock`, meant to be held across the ENTIRE + decide+ship+manifest-write critical section, e.g.: + + with badge_lock(vault_dir, badge): + manifest = load_manifest(vault_dir, badge, kind="leveled") + ... decide_level / ship / write_manifest(..., kind="leveled") ... + + Two concurrent runs of the same target otherwise race the manifest + read-modify-write; last-writer-wins can silently orphan a delta that was + actually shipped to tape but never recorded (unrecoverable by reclaim or + restore, since neither knows it exists). A plain contextmanager-decorated + generator function would need `import fcntl` at module import time to + apply the decorator; this is a class instead so fcntl/os stay imported + inside the methods, consistent with every other function in this file + (Globus Compute / dill, invariant #1) even though this lock itself is + LOCAL-only and never serialized. POSIX-only (fcntl) — matches every other + subprocess/hsi call here, which already assumes a POSIX RCAC host. + """ + + def __init__(self, vault_dir, badge): + self.vault_dir = vault_dir + self.badge = badge + self._fh = None + + def __enter__(self): + import os + import fcntl + + os.makedirs(self.vault_dir, exist_ok=True) + lock_path = os.path.join(self.vault_dir, "{}.lock".format(self.badge)) + self._fh = open(lock_path, "a+") + fcntl.flock(self._fh.fileno(), fcntl.LOCK_EX) # blocks until acquired + return self + + def __exit__(self, exc_type, exc, tb): + import fcntl + + fcntl.flock(self._fh.fileno(), fcntl.LOCK_UN) + self._fh.close() + self._fh = None + return False + + +def level_and_ship(project_name, source_folder, file_pattern, survivors, exclusion, + vault_dir, log_dir, now_epoch, ship_level, force_full=False): + """ + LOCAL orchestration for Phase-3 leveled incrementals (§3) — the leveled + analog of route_and_ship. Holds badge_lock across the whole + decide+ship+manifest-write critical section. + + Args: + survivors: {arcname: (mtime_ns, size)} post-exclusion (from + enumerate_source_catalog_excluded) — the LIVE catalog. + exclusion: the recorded exclusion report (stored in the manifest, same + shape as Phase 2's, for reclaim's INV-E re-gate). + vault_dir: {log_dir}/_vault — manifest lives here (durable+shared). + now_epoch: run epoch seconds (BLONDE quaver). + ship_level(level_obj, run_stem) -> dict(fortress_tar, zip_checksum, + n_files, source_bytes, run_timestamp[, tar_bytes]) — INJECTED + (mirrors route_and_ship's ship_one(obj, stem)) so tests can + mock the tape path. level_obj = {"members", "blonde", + "level", "parent_blonde"} — everything ship_level needs to + zip exactly `members` and tag the run-log. Production wires + it to make_zip_files(..., only_arcnames=level_obj["members"]) + + send_to_fortress(..., extra_record_fields={"leveled_object": + True, "badge", "level", "blonde", "parent_blonde"}). Called + ONLY when there is something to ship — never for a NO-OP or a + deletions-only level (§3.2: "a deletions-only level ships no + tar"). + force_full: --full — force a fresh L0 baseline regardless of drift. + + Returns a summary dict: {badge, status, manifest, ...}; status is one of + "noop" / "deletions_only" / "shipped". On "noop" the caller (__main__) is + responsible for the loud §3.1 email — level_and_ship's signature has no + `emails`/`send_alert_email` dependency by design (this stays pure LOCAL + orchestration, mirroring route_and_ship's shape). + + A failed ship_level call is NEVER caught here — it propagates out (through + badge_lock's `finally`-guaranteed unlock) exactly like route_and_ship's + ship_one. No manifest/catalog state is written until AFTER ship_level + returns successfully, so a retry re-runs decide_level from the same prior + state and re-zips fresh — "a failed delta always re-zips fresh on retry" + (this is also why delta ships never need find_existing_zip's resume + machinery — the level-aware resume guard, §3.5.1, resolved by + construction: a fresh re-zip is cheap by construction for a delta). + """ + import datetime + + badge = badge_of(project_name, source_folder, file_pattern) + + with badge_lock(vault_dir, badge): + manifest = load_manifest(vault_dir, badge, kind="leveled") + if manifest is None: + manifest = { + "schema_version": 1, "schema": "leveled_v1", "badge": badge, + "project": project_name, "source_folder": source_folder, + "file_pattern": file_pattern, "current_l0_blonde": None, + "levels": [], "exclusion": exclusion, "events": [], + } + else: + manifest.setdefault("levels", []) + manifest.setdefault("events", []) + manifest["exclusion"] = exclusion # keep the latest report current + + decision = decide_level(badge, survivors, manifest, force_full, log_dir) + timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S") + + if decision is None: + # True NO-OP (§3.1 hard invariant): no ship, no new level node. + run_timestamp = "{}__noop_{}".format(project_name, timestamp) + record_noop_event(manifest, now_epoch, run_timestamp) + write_manifest(vault_dir, badge, manifest, kind="leveled") + source_bytes = sum(v[1] for v in survivors.values()) + write_noop_log(log_dir, project_name, run_timestamp, badge, + survivors, source_bytes) + return {"badge": badge, "status": "noop", "run_timestamp": run_timestamp, + "manifest": manifest} + + detail = decision["detail"] + level = int(detail) + is_l0 = (level == 0) + members = decision["members"] + deleted = decision.get("deleted") or [] + blonde = make_blonde(badge, now_epoch, detail) + run_stem = "{}__L{}".format(project_name, detail) + + if not members: + # Deletions-only level (§3.2): metadata only, no tar, ship_level is + # never called — the "never call ship on nothing" guard. + run_timestamp = "{}_{}".format(run_stem, timestamp) + node = { + "blonde": blonde, "level": level, "parent_blonde": decision["parent"], + "run_timestamp": run_timestamp, "transfer": None, + "fortress_tar": None, "snar_fortress_path": None, + "zip_checksum": None, "tar_bytes": None, "n_files": 0, + "source_bytes": 0, "catalog_ref": None, "catalog_sha256": None, + "deleted": deleted, + } + manifest["levels"].append(node) + if is_l0: + manifest["current_l0_blonde"] = blonde + write_manifest(vault_dir, badge, manifest, kind="leveled") + return {"badge": badge, "status": "deletions_only", "blonde": blonde, + "level": level, "deleted": deleted, "manifest": manifest} + + level_obj = {"members": members, "blonde": blonde, "level": level, + "parent_blonde": decision["parent"]} + r = ship_level(level_obj, run_stem) + run_timestamp = r["run_timestamp"] + + # Build the manifest catalog from what ACTUALLY shipped (the zip's real + # namelist, reported back as "zip_members"), never from the decision set + # `members` alone. make_zip_files re-walks the source at zip time and + # silently drops any `only_arcnames` entry no longer on disk (no error + # for a keep-entry never encountered) — recording the decision set + # regardless would poison the manifest catalog with a file the zip (and + # its run-log MD5s) don't actually contain, which restore's + # verify_level_members would then fail on FOREVER (every future restore + # of this chain), not just until the next run. If ship_level reports the + # real set and it differs from the decision, fail loudly now (mirrors + # ship_object's fail-fast philosophy for the analogous silent-drop risk) + # rather than let a phantom manifest entry linger silently. + shipped_members = r.get("zip_members") + if shipped_members is None: + # Back-compat: an injected ship_level (tests, or a caller not yet + # updated) that doesn't report the actual set. Falls back to the + # decision set, same as before this fix. + shipped_members = members + else: + vanished = sorted(set(members) - set(shipped_members)) + if vanished: + raise RuntimeError( + "level_and_ship: {} member(s) selected for level {} vanished " + "between enumerate and zip (not in the shipped zip's actual " + "namelist) and were silently dropped by make_zip_files: {}. " + "Aborting before recording a manifest catalog entry for a " + "file that isn't actually on tape.".format( + len(vanished), level, vanished)) + + catalog_subset = {a: survivors[a] for a in shipped_members if a in survivors} + catalog_ref, catalog_sha256 = write_level_catalog( + log_dir, run_timestamp, catalog_subset) + + node = { + "blonde": blonde, "level": level, "parent_blonde": decision["parent"], + "run_timestamp": run_timestamp, "transfer": "zip", + "fortress_tar": r["fortress_tar"], "snar_fortress_path": None, + "zip_checksum": r["zip_checksum"], "tar_bytes": r.get("tar_bytes"), + "n_files": r["n_files"], "source_bytes": r["source_bytes"], + "catalog_ref": catalog_ref, "catalog_sha256": catalog_sha256, + "deleted": deleted, + } + if decision.get("forced_rebaseline"): + node["forced_rebaseline"] = True + manifest["levels"].append(node) + if is_l0: + manifest["current_l0_blonde"] = blonde + write_manifest(vault_dir, badge, manifest, kind="leveled") + + return {"badge": badge, "status": "shipped", "blonde": blonde, + "level": level, "n_files": node["n_files"], + "source_bytes": node["source_bytes"], "deleted": deleted, + "forced_rebaseline": bool(decision.get("forced_rebaseline")), + "manifest": manifest} + + # --------------------------------------------------------------------------- # Exclusion primitives (see docs/EXCLUSION_SPEC.md). # @@ -2230,7 +2774,8 @@ def make_zip_files(source_folder, file_pattern, tmp_dir, project_name, def send_to_fortress(zip_path, zip_checksum, file_checksums, members, source_bytes, source_folder, file_pattern, fortress_base_dir, emails, project_name, log_dir, - cleanup_zip_on_success=False, exclusion=None): + cleanup_zip_on_success=False, exclusion=None, + extra_record_fields=None, quiet_success=False): """ On the RCAC compute system: transfer the zip to Fortress via htar_large, then performs full round-trip retrieval and MD5 verification. @@ -2242,6 +2787,24 @@ def send_to_fortress(zip_path, zip_checksum, file_checksums, members, All steps use full stdout+stderr logging since no binary streaming is involved. Log files are named after the zip (e.g. X0F_20260331_140000.txt/.json). + + extra_record_fields: optional dict merged into run_record at construction + (default None is behavior-neutral for the existing whole-target caller). + Phase 3 (leveled incremental, RFC §3) tags leveled_object/badge/level/ + blonde/parent_blonde here so a level's run-log is self-describing and + reclaim's --scan loop (P3-2) can recognize + route it without depending on + the manifest alone. Applied via dict.update, so any of these keys can + override a same-named default (none currently collide). + + quiet_success: suppress ONLY the "[SUCCESS] Fortress archive complete" + completion email (default False, behavior-neutral for the whole-target and + Phase-2 routing callers). Phase 3's leveled path sets this True: __main__'s + incremental branch already sends its OWN single "[SUCCESS]"/"[NO-OP]" + summary email per run via send_alert_email, so without this a leveled ship + would double up (one from here, one from __main__) on every successful + level — noise that dilutes the notification-triage strategy's signal on a + daily-cron target. Every FAILURE email in this function is untouched by + this flag — only the one success path is affected. """ import os import re @@ -2308,6 +2871,11 @@ def send_to_fortress(zip_path, zip_checksum, file_checksums, members, "status": "running", "error": None, } + # Additive, behavior-neutral for the whole-target caller (default None): + # Phase 3 (RFC §3) tags leveled_object/badge/level/blonde/parent_blonde so + # this level's own run-log is self-describing. + if extra_record_fields: + run_record.update(extra_record_fields) def write_log(msg, level="INFO"): ts = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") @@ -2648,6 +3216,15 @@ def run_watched(cmd, watch_paths, stall_seconds=7200, poll_interval=30): write_log(f"hsi get succeeded — tar retrieved: {retrieved_tar}") write_log(f"hsi get output: {hsi_out.strip()}") + # On-tape tar size (RFC §3.3: "tar_bytes, distinct from uncompressed + # source_bytes" — a day-one reclaim-readiness field, input to the future + # §3.8 `hsi ls -j` size cross-check). Captured here for free: the tar is + # already retrieved to local disk for round-trip verification, so its size + # costs nothing extra to record. + tar_bytes = os.path.getsize(retrieved_tar) + run_record["tar_bytes"] = tar_bytes + write_log(f"Tar size on Fortress: {tar_bytes} bytes") + # Step B: local tar xvf — extract the zip from the retrieved tar tar_extract_out = os.path.join(src_dir, f"tar_extract_{zip_ts}.out") tar_extract_err = os.path.join(src_dir, f"tar_extract_{zip_ts}.err") @@ -2828,21 +3405,22 @@ def run_watched(cmd, watch_paths, stall_seconds=7200, poll_interval=30): else: excl_email = "" - send_email( - f"[SUCCESS] Fortress archive complete: {filename}", - f"Project: {project_name}\n\n" - f"Archive workflow completed successfully.\n\n" - f"Source (Depot): {source_folder}\n" - f"Destination (Fortress): {fortress_tar}\n" - f"Local zip: {zip_note}\n" - f"Zip MD5: {zip_checksum}\n" - f"{excl_email}\n" - f"Log (TXT): {txt_path}\n" - f"Log (JSON): {json_path}\n\n" - f"Full log is available at the path above." - ) + if not quiet_success: + send_email( + f"[SUCCESS] Fortress archive complete: {filename}", + f"Project: {project_name}\n\n" + f"Archive workflow completed successfully.\n\n" + f"Source (Depot): {source_folder}\n" + f"Destination (Fortress): {fortress_tar}\n" + f"Local zip: {zip_note}\n" + f"Zip MD5: {zip_checksum}\n" + f"{excl_email}\n" + f"Log (TXT): {txt_path}\n" + f"Log (JSON): {json_path}\n\n" + f"Full log is available at the path above." + ) - return fortress_tar, txt_path, json_path + return fortress_tar, txt_path, json_path, tar_bytes def ship_object(source_folder, arcnames, object_stem, catalog, fortress_base_dir, @@ -3116,9 +3694,10 @@ def _hsi_ls_ok(path): use_globus = False resume_enabled = True # reuse a valid {project}_*.zip left in tmp_dir if present force_resume = False # resume even if the existing zip no longer matches source + force_full = False # --full (Phase 3, RFC §3.1): force a fresh L0 baseline config_path = None - _USAGE = ("Usage: python archive.py [--globus] [--fresh] [--force-resume] config.json " - "(default: local, no Globus)") + _USAGE = ("Usage: python archive.py [--globus] [--fresh] [--force-resume] [--full] " + "config.json (default: local, no Globus)") for _arg in sys.argv[1:]: if _arg == "--globus": use_globus = True @@ -3128,6 +3707,13 @@ def _hsi_ls_ok(path): resume_enabled = False # ignore any existing zip; always re-zip from source elif _arg == "--force-resume": force_resume = True # escape hatch: skip the stale-zip safety check + elif _arg == "--full": + # Phase 3 (leveled incremental) only: force a fresh L0 baseline even if + # decide_level would otherwise NO-OP or emit a delta — the operational + # safety valve for the §3.1 hard limitation (same-size/same-mtime + # in-place rewrites are invisible to the cheap diff). No-op flag for + # whole-target/size-routing configs (nothing reads force_full there). + force_full = True elif _arg.startswith("-"): print(f"Unknown option: {_arg}") print(_USAGE) @@ -3161,6 +3747,7 @@ def _hsi_ls_ok(path): t_small = config["t_small"] shard_count_cfg = config["shard_count"] shard_target = config["shard_target"] + incremental = config["incremental"] # Size-routing runs LOCAL only (RFC §2.8): the per-target manifest lives on a # shared durable log_dir that __main__ reads/writes, and reclaim later reads. Refuse @@ -3177,6 +3764,23 @@ def _hsi_ls_ok(path): notify_bell(f"[FAILED] Fortress archive: size_routing + --globus ({project_name})", msg) sys.exit(1) + # Leveled incremental (Phase 3, RFC §3.5.3 invariant #1) also runs LOCAL only, for + # the same reason: decide_level/badge_lock/the manifest+catalog all read/write on + # the launching host. Under --globus, send_to_fortress would write the level's log + # + sibling catalog on the ENDPOINT host while the manifest is read/written HERE — + # they'd diverge. Refuse up front rather than let that happen silently. + if incremental and use_globus: + msg = ("incremental is enabled in this config, but --globus was passed.\n" + "Leveled incrementals keep a per-target manifest + sibling catalogs on\n" + "shared durable storage and run LOCAL only (docs/RFC_incremental_v2.md\n" + "§3.5.3). Re-run without --globus (e.g. via archive_local.sbatch), or\n" + "disable incremental for a --globus run.") + print(f"\n ERROR: {msg}") + send_alert_email(emails, + f"[FAILED] Fortress archive: incremental + --globus ({project_name})", msg) + notify_bell(f"[FAILED] Fortress archive: incremental + --globus ({project_name})", msg) + sys.exit(1) + # `run` abstracts how a worker function is executed: in-process by default, or # submitted to the endpoint and waited on with --globus. Step 1/Step 2 call run() # and are identical in both modes. @@ -3387,6 +3991,180 @@ def ship_one(obj, stem): send_alert_email(emails, subject, body) sys.exit(0) + # ------------------------------------------------------------------ # + # Leveled incremental path (Phase 3, RFC §3) — LOCAL only (--globus refused + # above). Ships an L0 baseline once, then only the (mtime,size)-diffed delta + # on every later run — a NO-OP ships nothing at all. Terminal: this branch + # replaces the whole-target Step 1/Step 2 below, same as size_routing above. + # ------------------------------------------------------------------ # + if incremental: + import time + + print("#" * 64) + print("# LEVELED INCREMENTAL (Phase 3) — L0 baseline + delta levels") + print("# whole-target ship disabled") + print("#" * 64) + + vault_dir = vault_dir_for(log_dir) + now_epoch = int(time.time()) + + # One walk: survivor catalog (post-exclusion) + the exclusion report to + # record — same primitive size-routing uses (RFC §2.5/§3.2). + try: + survivors, exclusion = run(enumerate_source_catalog_excluded, + source_folder, file_pattern, + exclude_spec, exclude_optional) + except Exception as e: + msg = (f"Leveled-incremental enumerate failed for {project_name}.\n\n" + f"Source: {source_folder}\nPattern: {file_pattern}\n\nError:\n{e}") + print(f"\n ERROR: {msg}") + send_alert_email(emails, f"[FAILED] Fortress incremental enumerate: {project_name}", msg) + notify_bell(f"[FAILED] Fortress incremental enumerate: {project_name}", msg) + sys.exit(1) + + if not survivors: + msg = (f"Leveled incremental found no files to archive for {project_name} " + f"(pattern {file_pattern!r} under {source_folder}, after exclusion).") + print(f"\n ERROR: {msg}") + send_alert_email(emails, f"[FAILED] Fortress incremental: nothing to archive ({project_name})", msg) + notify_bell(f"[FAILED] Fortress incremental: nothing to archive ({project_name})", msg) + sys.exit(1) + + def ship_level(level_obj, stem): + # Path A (RFC §3.0): zip only this level's members, then the SAME + # make_zip_files -> send_to_fortress chain as the whole-target path + # (verify chain byte-for-byte, invariant #6 for free). + # + # The level-aware resume guard (§3.5.1) draws a hard line between L0 + # and deltas: "zip is a complete source snapshot" is only true for an + # L0 ship, so ONLY L0 attempts find_existing_zip's resume (stem + # "{project_name}__L0" scopes the glob away from any delta's zip — + # they use "{project_name}__L{n}"). A delta NEVER resumes: it always + # re-zips fresh from source (cheap by construction — a delta's member + # set is small), so misfiring the whole-target-style staleness guard + # against a delta's necessarily-partial zip can't happen. + # + # For L0, `level_obj["members"]` IS the full survivor set decide_level + # computed (ALL matched, post-exclusion arcnames) — the exact same + # `survivors` catalog closed over from the enumerate above — so the + # staleness check reuses it directly instead of re-walking the source + # a second time: an existing L0 zip is trusted only if its member + # count matches len(survivors) AND no survivor's mtime is newer than + # the zip's timestamp. Same failure-count/newer-than-zip semantics as + # the whole-target guard, honors --fresh (resume_enabled)/ + # --force-resume, just computed from data already in hand. + resume_result = None + if level_obj["level"] == 0 and resume_enabled: + existing_zip = find_existing_zip(tmp_dir, stem) + if existing_zip: + print(f" [RESUME] Existing L0 zip found: {existing_zip}") + resume_result = try_reconstruct(existing_zip, run) + if resume_result is not None and not force_resume: + _, _, _, resumed_members, _, _ = resume_result + newest_survivor = (max(floor_s(v[0]) for v in survivors.values()) + if survivors else None) + if l0_zip_is_stale(len(resumed_members), len(level_obj["members"]), + newest_survivor, existing_zip): + print(f" [RESUME] Existing L0 zip is STALE (no longer " + f"matches the live survivor set); re-zipping fresh.") + resume_result = None + if resume_result is not None: + print(f" [OK] {len(resume_result[3])} file(s) verified " + f"from existing L0 zip") + if resume_result is not None: + zip_path, zip_checksum, file_checksums, zip_members, lvl_source_bytes, _ = \ + resume_result + else: + zip_path, zip_checksum, file_checksums, zip_members, lvl_source_bytes, _ = run( + make_zip_files, source_folder, file_pattern, tmp_dir, stem, + compression, allow_empty_files, None, (), level_obj["members"]) + fortress_tar, txt_path, json_path, tar_bytes = run( + send_to_fortress, + zip_path, zip_checksum, file_checksums, zip_members, + lvl_source_bytes, source_folder, file_pattern, fortress_base, + emails, project_name, log_dir, cleanup_zip_on_success, exclusion, + {"leveled_object": True, + "badge": badge_of(project_name, source_folder, file_pattern), + "level": level_obj["level"], "blonde": level_obj["blonde"], + "parent_blonde": level_obj["parent_blonde"]}, + True, # quiet_success — __main__'s incremental branch sends its + # own single per-run summary email; without this every + # successful level would double up on completion emails. + ) + run_timestamp = os.path.splitext(os.path.basename(zip_path))[0] + return {"fortress_tar": fortress_tar, "zip_checksum": zip_checksum, + "n_files": len(zip_members), "source_bytes": lvl_source_bytes, + "run_timestamp": run_timestamp, "tar_bytes": tar_bytes, + # The zip's REAL namelist, not the decision set requested — + # lets level_and_ship detect + fail fast on a vanished-member + # race instead of poisoning the manifest catalog. + "zip_members": list(zip_members)} + + try: + summary = level_and_ship( + project_name, source_folder, file_pattern, survivors, exclusion, + vault_dir, log_dir, now_epoch, ship_level, force_full=force_full) + except Exception as e: + msg = (f"Leveled-incremental ship failed for {project_name}.\n\n" + f"If a level already reached tape it is recorded in the manifest;\n" + f"re-running retries the same decision from scratch (a failed delta\n" + f"always re-zips fresh).\n\nError:\n{e}") + print(f"\n ERROR: {msg}") + send_alert_email(emails, f"[FAILED] Fortress incremental ship: {project_name}", msg) + notify_bell(f"[FAILED] Fortress incremental ship: {project_name}", msg) + sys.exit(1) + + print() + print("=" * 60) + print("LEVELED INCREMENTAL COMPLETE") + print("=" * 60) + print(f" Badge: {summary['badge']} status: {summary['status']}") + print(f" Manifest: {manifest_path(vault_dir, summary['badge'], kind='leveled')}") + + if summary["status"] == "noop": + # Hard invariant (§3.1) — must not be silently dropped: a true NO-OP + # means nothing was shippable by the cheap (mtime,size) diff, but a + # same-size/same-mtime in-place content rewrite is INVISIBLE to that + # diff. Email LOUDLY, with the exact wording the plan/RFC specify, so + # an operator knows to periodically --full rather than trust silence. + noop_msg = ( + "detected zero metadata change; same-size in-place content edits " + "are NOT captured. Use `--full` to force a fresh L0 if you " + "rewrote files preserving mtime+size." + ) + print(f" [NO-OP] {noop_msg}") + body = (f"Leveled incremental NO-OP for {project_name}.\n\n" + f"Badge: {summary['badge']}\n\n" + f"{noop_msg}\n\n" + f"Manifest: {manifest_path(vault_dir, summary['badge'], kind='leveled')}\n") + subject = f"[NO-OP] Fortress incremental: {project_name}" + send_alert_email(emails, subject, body) + # NOT notify_bell(): the T1 "ACT NOW" Teams DM bell is reserved for + # [FAILED]/[STALE]/[EXCLUSION-ABORT] — "backup didn't happen / needs + # you NOW" (see CLAUDE.md's Failure notifications section). A NO-OP + # is a normal, potentially daily outcome for an unchanged/low-churn + # target, not a failure; ringing the T1 bell on every routine + # unchanged run would desensitize the operator to the bell that + # signals genuine failures. The durable email above already carries + # the exact §3.1 wording operators need. + else: + n_files = summary.get("n_files", 0) + src_bytes = summary.get("source_bytes", 0) + deleted = summary.get("deleted") or [] + print(f" Level: {summary.get('level')} blonde: {summary.get('blonde')}") + print(f" Files: {n_files} Bytes: {src_bytes} Deleted: {len(deleted)}") + body = (f"Leveled incremental archive complete for {project_name}.\n\n" + f"Badge: {summary['badge']} Level: {summary.get('level')} " + f"BLONDE: {summary.get('blonde')}\n" + f"Status: {summary['status']}\n" + f"Files shipped: {n_files} Bytes shipped: {src_bytes} " + f"Deleted (metadata only): {len(deleted)}\n" + + ("\nForced full re-baseline (byte-sum drift caught by decide_level).\n" + if summary.get("forced_rebaseline") else "") + + f"\nManifest: {manifest_path(vault_dir, summary['badge'], kind='leveled')}\n") + send_alert_email(emails, f"[SUCCESS] Fortress incremental: {project_name}", body) + sys.exit(0) + # Step 1: Find files, zip, and verify on the compute system print() print("=" * 60) @@ -3540,7 +4318,7 @@ def ship_one(obj, stem): print(" (this may take several minutes)") print("=" * 60) try: - fortress_tar, txt_path, json_path = run( + fortress_tar, txt_path, json_path, _tar_bytes = run( send_to_fortress, zip_path, zip_checksum, file_checksums, members, source_bytes, source_folder, file_pattern, fortress_base, diff --git a/config.example.json b/config.example.json index 740815d..60a66cf 100644 --- a/config.example.json +++ b/config.example.json @@ -45,5 +45,8 @@ "shard_count": null, "comment_shard_target": "Optional (default 274877906944 = 256 GiB). Target uncompressed shard size, used to derive K at baseline and to warn (not fail) when a live shard exceeds ~2x this. Only used when size_routing is true.", - "shard_target": 274877906944 + "shard_target": 274877906944, + + "comment_incremental": "Optional (default false). Phase-3 leveled incrementals (docs/RFC_incremental_v2.md §3): ship an L0 baseline once, then only the (mtime,size)-diffed delta on every later run, instead of one whole-target tar OR Phase-2's solo/shard objects. Mutually exclusive with size_routing (pick one scheme per target). LOCAL mode only -- refused under --globus. Same shared+durable log_dir requirement as size_routing: the manifest lives at {log_dir}/_vault/{badge}.leveled.manifest.json. HARD LIMITATION: a same-size, same-mtime in-place content rewrite is invisible to the diff and will NOT be captured -- run with --full periodically (or after any such rewrite) to force a fresh L0 baseline. Omit for the default whole-target behavior.", + "incremental": false } diff --git a/docs/RFC_incremental_v2.md b/docs/RFC_incremental_v2.md index cca862d..b9e4dfa 100644 --- a/docs/RFC_incremental_v2.md +++ b/docs/RFC_incremental_v2.md @@ -1,6 +1,6 @@ # RFC: Staged-Hybrid Incremental Backup for fortress-archive -**Status:** Phase 1 DONE (merged). **Phase 2 (size-routing / append) — this RFC's build target, ready for review.** Phase 3 (leveled) DEFERRED. +**Status:** Phase 1 DONE (merged). Phase 2 (size-routing / append) DONE (merged). **Phase 3 (leveled L0/L1/L2 incremental) BUILT** — engine (P3-1), reclaim (P3-2), and restore (P3-3) all merged; opt-in per config (`"incremental": true`) and default-off, as originally planned (activate per-asset when the reship-monitor measures the uniform-medium-churn signature — see the row below and §3.10). **Base:** `main` @ `f7e505a` (suite 136 passing). Reuses the PR #8 primitives already merged. **Supersedes:** the original leveled-only RFC (this file's prior revision, still the Phase-3 reference — see §3) and the closed design-only RFC (former PR #4). @@ -65,8 +65,8 @@ Effort is spent where a live measurement justified it. Three phases, built in or | Phase | Mechanism | Kills | Status | |---|---|---|---| | **1 — Exclusion** | Identity + git-coverage subtractive filter; drop git-versioned regenerable metadata from the backup | Standoff-style metadata churn on a stable bulk target (2.35 TB for 2 files) | **DONE — merged** (`docs/EXCLUSION_SPEC.md`; PRs #8/#10–#17; suite 136 green) | -| **2 — Size-routing / append** | Per target: files ≥ `T_small` ship as their own Fortress object (**solo**, skip-if-unchanged); smaller files bundle into K **content-addressed shards** (`shard = hash(relpath) mod K`, whole-shard re-ship on change). New file → new object; unchanged objects skip. Per-target **aggregate** reclaim. Reship-monitor. **Full restore** (object-model reassembly) as the capstone slice. | Whole-target re-ship on **growth (append)** and on a single big file changing | **THIS RFC — ready to build** | -| **3 — Leveled L0/L1/L2** | Per-target full-anchor + differential/incremental delta tars, chain-aware reclaim, restore | **Uniform-medium-churn**: a large tier of small, non-git-redundant DATA files changing often — the one case exclusion can't drop and routing would re-ship whole shards | **DEFERRED — dormant** (§3; PR #9 branch `feat/incremental-level-engine` preserved). Activated per-asset **only** when the reship-monitor measures the signature. | +| **2 — Size-routing / append** | Per target: files ≥ `T_small` ship as their own Fortress object (**solo**, skip-if-unchanged); smaller files bundle into K **content-addressed shards** (`shard = hash(relpath) mod K`, whole-shard re-ship on change). New file → new object; unchanged objects skip. Per-target **aggregate** reclaim. Reship-monitor. **Full restore** (object-model reassembly) as the capstone slice. | Whole-target re-ship on **growth (append)** and on a single big file changing | **DONE — merged** | +| **3 — Leveled L0/L1/L2** | Per-target full-anchor + differential/incremental delta tars, chain-aware reclaim, restore | **Uniform-medium-churn**: a large tier of small, non-git-redundant DATA files changing often — the one case exclusion can't drop and routing would re-ship whole shards | **BUILT — merged** (§3; engine/reclaim/restore all landed). Opt-in per config (`"incremental": true`), default-off. Activated per-asset when the reship-monitor measures the signature. | The phases share one spine (adopted from Nathan Denny's Bastion; see `bastion-architecture` memory and §3): a **badge** identity per target, a per-target @@ -592,20 +592,25 @@ deferred Phase-3 leveled restore (§3.4) — chain fold, overlay ordering, delet --- -## 3. Phase 3 — Leveled L0 / L1 / L2 `[DEFERRED — DORMANT]` - -> **DEFERRED.** Everything below is the **Phase-3 reference design**, preserved intact -> from this RFC's prior revision. It is **not being built now.** It is activated -> **per-asset only** when the Phase-2 reship-monitor (§2.7) measures the -> **uniform-medium-churn** signature — a large tier of small, non-git-redundant DATA -> files changing often, the one case exclusion can't drop and size-routing re-ships as -> whole shards. That signature has **not** appeared in any FloraSense asset. The code -> exists and is gated on branch `feat/incremental-level-engine` (parked PR #9). When -> built, it adopts Bastion's drift/cadence/genus/thread model (see `bastion-architecture` -> memory). The chain-aware reclaim (retain-N-L0, prune deltas, re-anchor, -> reconstruct-from-L0+all-deltas) is the expensive/risky part and is built here, not -> before. Line references below are to the pre-Phase-1 tree and are indicative, not -> current. +## 3. Phase 3 — Leveled L0 / L1 / L2 `[BUILT — MERGED]` + +> **BUILT.** Everything below is the **Phase-3 design + build record** — the engine +> (P3-1, level decision + delta ship), reclaim (P3-2, `verify_chain` aggregate +> verification), and restore (P3-3, chain replay + reconciliation) have all landed +> (see CLAUDE.md's "Leveled incremental (Phase 3)" section for the operator-facing +> summary and config key). It remains **opt-in per config** (`"incremental": true`) +> and **default-off**, activated **per-asset** when the Phase-2 reship-monitor +> (§2.7) measures the **uniform-medium-churn** signature — a large tier of small, +> non-git-redundant DATA files changing often, the one case exclusion can't drop +> and size-routing re-ships as whole shards — exactly as originally planned; this +> section describes the mechanism that is now available to turn on, not a +> not-yet-built design. It adopts Bastion's drift/cadence/genus/thread model (see +> `bastion-architecture` memory). Retain-N-L0/prune-deltas/re-anchor tape pruning +> (§3.8-adjacent) is explicitly **not** part of this build (P3-4, future work) — +> today nothing prunes tape, so every level a target has ever shipped stays on +> Fortress. Line/section references below describe the shipped implementation; +> where they still cite pre-Phase-1 line numbers for historical context, treat +> those specific numbers as indicative, not current. ### 3.0 Recommendation — Path A now, Path B only behind a passing spike diff --git a/reclaim.py b/reclaim.py index b6031de..a498a4b 100644 --- a/reclaim.py +++ b/reclaim.py @@ -483,9 +483,18 @@ def find_logs(log_dir, project=None, days=None): """ Return JSON log paths in log_dir, optionally filtered to a project prefix and/or to logs modified within the last `days`. Newest first. + + Excludes Phase-3 leveled-incremental sibling catalog files + (`{run_timestamp}.catalog.json`, written by archive.write_level_catalog next + to each level's run-log pair in the SAME log_dir — RFC §3.2/§3.3, so + effective_catalog can find them by a relative catalog_ref). Without this, + a bare `*.json` glob also matches these (they end in ".json" too), and each + one — having no "project"/"status" fields — surfaces in --scan as a bogus + `[FAILED] None` entry instead of being silently absent. """ pattern = f"{project}_*.json" if project else "*.json" - paths = glob.glob(os.path.join(log_dir, pattern)) + paths = [p for p in glob.glob(os.path.join(log_dir, pattern)) + if not p.endswith(".catalog.json")] if days is not None: cutoff = datetime.datetime.now().timestamp() - days * 86400 paths = [p for p in paths if os.path.getmtime(p) >= cutoff] @@ -721,20 +730,44 @@ def find_manifests(log_dir): return out +def is_leveled_manifest(m): + """ + True if `m` is a Phase-3 leveled-incremental manifest (archive.level_and_ship's + schema: "schema": "leveled_v1", a "levels" list, "current_l0_blonde"), as + opposed to a Phase-2 size-routing manifest (flat "objects": [...] schema, no + "schema" discriminator field at all). The "levels" in m fallback keeps this + working even for a manifest written before the "schema" discriminator existed, + or one some future minor-version bump renames — "levels" is the one structural + fact that is true of every leveled manifest and false of every routing one. + """ + return m.get("schema") == "leveled_v1" or "levels" in m + + def manifest_for_project(log_dir, project): """ - The size-routing manifest for `project`, or None if it has never routed (no + The manifest for `project`, or None if it has never routed/leveled (no manifest yet). Mirrors latest_log_for_project()'s role for the whole-target - log path — a deployment wrapper driving BOTH routed and non-routed targets - (e.g. a per-target size_routing config flag) needs a single lookup that - works either way without re-deriving the badge (badge_of lives in archive.py, - not reclaim.py; matching on the manifest's own recorded `project` field avoids - that cross-module dependency). + log path — a deployment wrapper driving routed, leveled, AND non-routed + targets (e.g. a per-target size_routing/incremental config flag) needs a + single lookup that works across all three without re-deriving the badge + (badge_of lives in archive.py, not reclaim.py; matching on the manifest's own + recorded `project` field avoids that cross-module dependency). + + A badge mid-conversion from Phase-2 (size-routing) to Phase-3 (leveled) can + have BOTH a `{badge}.manifest.json` (routing) and a + `{badge}.leveled.manifest.json` (leveled) for the SAME project on disk at + once — exactly the point of manifest_path's kind= split: neither schema ever + clobbers the other's file. find_manifests() returns both, and glob-order + (the prior behavior here) doesn't reliably reflect which one is current. + Fix: deterministically PREFER the leveled manifest when both exist — a badge + converted to Phase 3 is on Phase 3 going forward; its old Phase-2 manifest is + a frozen historical record, not the active source of truth. """ - for m in find_manifests(log_dir): - if m.get("project") == project: - return m - return None + matches = [m for m in find_manifests(log_dir) if m.get("project") == project] + if not matches: + return None + leveled = [m for m in matches if is_leveled_manifest(m)] + return leveled[0] if leveled else matches[0] def verify_target(manifest, holds, canonical_roots=(), allowlist=None, opt_in=False): @@ -883,6 +916,173 @@ def verify_target(manifest, holds, canonical_roots=(), allowlist=None, opt_in=Fa } +def verify_chain(manifest, holds, log_dir, canonical_roots=(), allowlist=None, opt_in=False): + """ + Per-target AGGREGATE structural double-check for a Phase-3 leveled-incremental + target (RFC §3.5.2), the leveled sibling of verify_target (§2.6). A leveled + target ships as an L0 baseline PLUS delta levels; reclaim's per-log verify() + would see only the newest level's tiny member set and perpetually false-DRIFT + (a delta's archived count is a handful of files against the WHOLE live tree). + + verify_chain instead reconstructs the expected state by REPLAYING the level + tree via archive.effective_catalog (L0 ∪ deltas, add/modify applied, deletions + removed) — the exact same fold decide_level itself diffs against, so reclaim's + notion of "what's on tape" agrees with the engine's own notion of "what's + already shipped": + + expected catalog/bytes = archive.effective_catalog(manifest, log_dir) + archived count = len(expected catalog) + archived bytes = expected catalog's total + every non-null `fortress_tar` across the CURRENT chain (the L0 node + matching current_l0_blonde plus deltas whose parent_blonde points at + it — the same node set effective_catalog/select_chain replay, NOT + every node manifest["levels"] has ever accumulated) must be present on + tape (hsi_exists per level) — ANY missing one means the chain cannot be + replayed end-to-end, so the verdict is FAILED ("chain is broken — not + restorable"), never a partial SAFE. A deletions-only or NO-OP level + legitimately carries no tar and is not part of this check. An OLDER, + superseded chain left behind by a `--full` rebaseline is deliberately + excluded — effective_catalog doesn't restore from it either, so its + tars going missing (once P3-4 tape pruning exists) must not FAIL a + target whose actual restorable chain is intact. + per-arcname newer-than-archived is judged directly against the expected + catalog's own recorded (mtime,size) for that arcname (floor_s on both + sides, matching decide_level's own comparison) rather than a single + per-object run timestamp — the leveled manifest carries full per-file + mtime/size via effective_catalog, a strictly more precise signal than + verify_target's "newer than the owning object's run" proxy. + + Verdict vocabulary + INV-E exclusion re-gate match verify()/verify_target + exactly. Returns the same result-dict shape (plus `all_tars`), so the report + and guarded --delete paths work unchanged. The manifest carries + `source_folder`/`file_pattern`/`exclusion`, so no config or original log is + needed. log_dir is required (unlike verify_target) because effective_catalog + reads each level's sibling catalog_ref file relative to it. + """ + project = manifest.get("project") + source_folder = manifest.get("source_folder") + pattern = manifest.get("file_pattern") + levels = manifest.get("levels") or [] + reasons = [] + + expected, expected_bytes = archive.effective_catalog(manifest, log_dir) + archived = len(expected) + archived_bytes = expected_bytes + + # Completeness is scoped to the CURRENT chain only — the same node set + # effective_catalog (above) and restore.select_chain replay: the L0 node + # matching current_l0_blonde, plus deltas whose parent_blonde points at + # that L0. `levels` can also hold an OLDER, now-superseded L0 + its deltas + # left behind by a `--full` rebaseline (a second, independent L0 gets + # appended and current_l0_blonde repointed — the old chain is never + # removed). Requiring those superseded tars to still be on tape would be + # wrong: they are not part of what effective_catalog/select_chain restore + # from, and nothing prunes tape today (P3-4, out of scope), but a false + # FAILED the moment a superseded chain's tar is released would otherwise + # conflate historical and current-chain levels under one "chain is broken" + # verdict. + current_l0_blonde = manifest.get("current_l0_blonde") + current_l0 = next((n for n in levels + if n.get("level") == 0 and n.get("blonde") == current_l0_blonde), + None) + if current_l0 is None: + zeros = [n for n in levels if n.get("level") == 0] + current_l0 = zeros[-1] if zeros else None + current_chain = [] + if current_l0 is not None: + current_chain = [current_l0] + [ + n for n in levels + if n.get("level", 0) > 0 and n.get("parent_blonde") == current_l0["blonde"] + ] + + all_tars = [n["fortress_tar"] for n in current_chain if n.get("fortress_tar")] + missing_tars = [t for t in all_tars if not hsi_exists(t)] + tars_ok = bool(all_tars) and not missing_tars + + hold = hold_entry_for(holds, project, source_folder) + + # INV-E: re-prove the archive's recorded exclusions are still git-COVERED. The + # leveled manifest carries the same `exclusion` block a log/routing-manifest + # does (level_and_ship keeps it current on every run), so recheck_exclusions + # works on it directly. + excl_check = recheck_exclusions(manifest, source_folder) + still_covered = excl_check["still_covered"] + + live = None + live_bytes = None + newer_count = 0 + newer_samples = [] + if source_folder and os.path.isdir(source_folder): + raw_live = archive.enumerate_source_catalog(source_folder, pattern or ".*") + live_catalog = {a: v for a, v in raw_live.items() if a not in still_covered} + live = len(live_catalog) + live_bytes = sum(v[1] for v in live_catalog.values()) + for a, (mtime_ns, _size) in live_catalog.items(): + exp = expected.get(a) + if exp is not None and archive.floor_s(mtime_ns) > archive.floor_s(exp[0]): + newer_count += 1 + if len(newer_samples) < 3: + newer_samples.append(a) + + # Verdict — most-blocking first, mirroring verify()/verify_target(). + if is_canonical(source_folder, canonical_roots): + verdict = CANONICAL + reasons.append("under canonical root — never deletable (hard invariant)") + elif not levels or source_folder is None: + verdict = FAILED + reasons.append("manifest has no levels" if not levels else "source_folder unknown") + elif not tars_ok: + verdict = FAILED + if not all_tars: + reasons.append("no level tars recorded in manifest") + else: + reasons.append(f"chain is broken — {len(missing_tars)} of {len(all_tars)} " + f"level tar(s) NOT found on Fortress (e.g. {missing_tars[0]}) " + f"— not restorable") + elif live is None: + verdict = GONE + reasons.append("source already removed from Depot") + elif excl_check.get("applies") and excl_check.get("no_longer_covered"): + verdict = DRIFTED + gone = excl_check["no_longer_covered"] + sample = ", ".join(f"{a} ({v})" for a, v in gone[:3]) + reasons.append(f"{len(gone)} previously-excluded file(s) no longer git-COVERED " + f"(e.g. {sample}) — edited/regenerated since archive; current " + f"bytes are not on tape") + elif newer_count > 0: + verdict = DRIFTED + reasons.append(f"{newer_count} file(s) modified since being archived (e.g. " + f"{', '.join(newer_samples)})") + elif live != archived: + verdict = DRIFTED + reasons.append(f"live files ({live}) != archived files ({archived}) across " + f"{len(levels)} level(s)") + elif live_bytes != archived_bytes: + verdict = DRIFTED + reasons.append(f"live bytes ({live_bytes}) != archived bytes ({archived_bytes}) " + f"— same count, content changed") + elif hold is not None: + # Typed hold tiering (docs/HOLDS_POLICY.md), identical to verify()/verify_target(). + verdict, hreason = classify_hold(hold, source_folder, live) + reasons.append(hreason) + elif opt_in and not on_allowlist(project, source_folder, allowlist): + verdict = KEEP + reasons.append("not on reclaim allowlist (opt-in mode) — kept") + else: + verdict = SAFE + reasons.append(f"{live} files across {len(levels)} level(s) match; nothing " + f"modified since archived; chain complete") + + return { + "project": project, "source_folder": source_folder, + "fortress_tar": (all_tars[-1] if all_tars else None), "all_tars": all_tars, + "archived": archived, "live": live, "newer": newer_count, + "archived_bytes": archived_bytes, "live_bytes": live_bytes, + "pattern": pattern, "run": manifest.get("badge"), + "verdict": verdict, "reasons": reasons, + } + + def safe_to_rm(folder, canonical_roots=()): """Refuse to rm anything that isn't a deep, existing Depot data path — or that lives under a declared canonical root (the hard invariant, re-checked @@ -1169,6 +1369,7 @@ def main(): # Build the list of (log_dict, config_dict) to verify. items = [] configs_by_project = {} + leveled_log_dir = args.log_dir # overridden below for single-config leveled targets if args.configs: for cpath in glob.glob(os.path.join(args.configs, "*.json")): try: @@ -1178,7 +1379,8 @@ def main(): except (ValueError, OSError): pass - manifests = [] # Phase-2 size-routed targets, verified as aggregates (verify_target) + manifests = [] # Phase-2 size-routed targets, verified as aggregates (verify_target) + leveled_manifests = [] # Phase-3 leveled-incremental targets (verify_chain) if args.scan: seen_projects = set() for lpath in find_logs(args.log_dir, days=args.days): @@ -1186,9 +1388,11 @@ def main(): log = load_json(lpath) except (ValueError, OSError): continue - # Phase-2 routed OBJECT logs are covered by their target's manifest below; - # never feed them to the per-log verify() (they'd each false-DRIFT). - if log.get("routed_object"): + # Phase-2 routed OBJECT logs and Phase-3 leveled level/NO-OP logs are + # covered by their target's manifest below; never feed either to the + # per-log verify() (they'd each false-DRIFT — a delta or single object + # log's tiny member set compared against the WHOLE live tree). + if log.get("routed_object") or log.get("leveled_object"): continue proj = log.get("project") # newest-first: only verify the most recent successful-or-not log per project @@ -1196,7 +1400,9 @@ def main(): continue seen_projects.add(proj) items.append((log, configs_by_project.get(proj))) - manifests = find_manifests(args.log_dir) + all_manifests = find_manifests(args.log_dir) + leveled_manifests = [m for m in all_manifests if is_leveled_manifest(m)] + manifests = [m for m in all_manifests if not is_leveled_manifest(m)] elif args.config: config = load_json(args.config) proj = config["project_name"] @@ -1210,6 +1416,19 @@ def main(): print(f"No size-routing manifest for {proj!r} at {mpath}") sys.exit(1) manifests = [load_json(mpath)] + elif config.get("incremental"): + # Leveled target (RFC §3.5.2): verify the level-chain aggregate via + # verify_chain, never the latest per-level/NO-OP log through legacy + # verify() (that log's tiny member set false-DRIFTs/FAILEDs against + # the whole live tree — see manifest_for_project's docstring). + badge = archive.badge_of(config["project_name"], config["source_folder"], + config["file_pattern"]) + leveled_log_dir = config.get("log_dir") or args.log_dir + mpath = archive.manifest_path(os.path.join(leveled_log_dir, "_vault"), badge, kind="leveled") + if not os.path.exists(mpath): + print(f"No leveled manifest for {proj!r} at {mpath}") + sys.exit(1) + leveled_manifests = [load_json(mpath)] else: lpath = latest_log_for_project(args.log_dir, proj) if not lpath: @@ -1231,10 +1450,12 @@ def matches(log, cfg): items = [(log, cfg) for (log, cfg) in items if matches(log, cfg)] def manifest_matches(m): - hay = " ".join([m.get("project") or "", m.get("source_folder") or ""] - + [o.get("fortress_tar") or "" for o in (m.get("objects") or [])]) + tars = ([o.get("fortress_tar") or "" for o in (m.get("objects") or [])] + + [n.get("fortress_tar") or "" for n in (m.get("levels") or [])]) + hay = " ".join([m.get("project") or "", m.get("source_folder") or ""] + tars) return args.match in hay manifests = [m for m in manifests if manifest_matches(m)] + leveled_manifests = [m for m in leveled_manifests if manifest_matches(m)] # Verify and report. Attach scratch-zip locators (for post-delete cleanup): # zip_name from the log + tmp_dir from the config. @@ -1254,6 +1475,15 @@ def manifest_matches(m): r["zip_name"] = None r["tmp_dir"] = None results.append(r) + # Phase-3 leveled targets: one aggregate verdict per manifest (verify_chain), + # replaying the level tree via archive.effective_catalog. Same "no staging zip" + # shape as the routed targets above. + for m in leveled_manifests: + r = verify_chain(m, holds, leveled_log_dir, canonical_roots=canonical_roots, + allowlist=allowlist, opt_in=opt_in) + r["zip_name"] = None + r["tmp_dir"] = None + results.append(r) results.sort(key=lambda r: (r["verdict"] != SAFE, r["project"] or "")) display = [r for r in results if not (args.skip_gone and r["verdict"] == GONE)] diff --git a/restore.py b/restore.py index 24eb7b9..68872a8 100644 --- a/restore.py +++ b/restore.py @@ -1,14 +1,14 @@ #!/usr/bin/env python3 """ restore.py — full / selective restore of a Phase-2 size-routed target -(docs/RFC_incremental_v2.md §2.9, the Phase-2 capstone). +(docs/RFC_incremental_v2.md §2.9, the Phase-2 capstone), PLUS full / point-in-time +restore of a Phase-3 leveled-incremental target (docs/RFC_incremental_v2.md §3.4). A size-routed target lives on Fortress as MANY objects (solos + shards), indexed by the per-target manifest at {log_dir}/_vault/{badge}.manifest.json. Because the objects are a DISJOINT partition of the current member set, restore is a plain union: extract every current object into an empty destination and reconcile — -no overlay ordering, no deletion replay (that is the deferred Phase-3 leveled -restore, RFC §3.4). +no overlay ordering, no deletion replay. Order of operations (each gate ABORTS the whole restore — never partial): (a) completeness gate BEFORE any transfer — every selected object's tar must @@ -24,12 +24,28 @@ ({log_dir}/{run_timestamp}.json, key `source_files`, arcname-keyed — invariant #2); (e) on a FULL restore, a MANDATORY final reconciliation: - set(walk(dest)) == ⋃ objects' arcnames, else ABORT. + set(walk(dest)) == union of objects' arcnames, else ABORT. Selective restore (`--slot solo:` / `--slot shard:` / `--member relpath`) filters the object set — for `htar` objects this reads only the requested members' bytes off tape. Reconciliation (e) runs only on a full restore. +A Phase-3 leveled target (`reclaim.is_leveled_manifest()` true) instead ships an +L0 baseline PLUS delta levels off it (a STAR, RFC §3.3) — restore there is +file-OVERLAY, not a disjoint union: `restore_leveled_target` walks the chain +L0 -> L1 -> L2 -> ... in strict run-time (quaver) order, extracting each level's +zip on top of the accumulating tree (Path A: the inner artifact is a zip, so +`zipfile.ZipFile(...).extractall(dest)`'s overwrite-same-name behavior IS the +overlay) and then applying that level's recorded deletions — the "never call +hsi_get(None)" guard for a deletions-only level lives in that loop, not by +hoping the extractor no-ops safely. Same gates (a)/(b)/(d)/(e) apply, with (a) += `check_chain_complete` (every non-null `fortress_tar` in the selected chain +must be on tape — a partial chain is un-restorable, full stop) and (e) reusing +`archive.effective_catalog`'s own L0+delta fold (`fold_expected`) so restore's +notion of "correct" always agrees with the engine's (decide_level) and +reclaim's (verify_chain). `--to-timestamp` truncates the chain for a +point-in-time restore (mirrors `effective_catalog`'s `upto_quaver`). + LOCAL only (needs `hsi`/`htar` at /opt/hsi/bin + the filesystem) — run on the storage host (Negishi), like reclaim.py. Operator-invoked; NOT wired into `reclaim --delete` (a full restore pulls every tar off tape, TB-scale). @@ -40,16 +56,20 @@ --dest DIR [--log-dir DIR] [--tmp-dir DIR] python3 restore.py config.json --dest DIR --slot shard:3 --slot solo:1a2b... python3 restore.py config.json --dest DIR --member sub/path/file.bin + python3 restore.py leveled_config.json --dest DIR --to-timestamp 20260701_030000 """ import argparse +import datetime import hashlib import json import os +import shutil import subprocess import sys +import zipfile sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -import archive # noqa: E402 (badge_of, vault_dir_for, manifest_path, load_config) +import archive # noqa: E402 (badge_of, vault_dir_for, manifest_path, load_config, decode_blonde) import reclaim # noqa: E402 (hsi_exists — the same tape-presence check reclaim trusts) @@ -282,27 +302,328 @@ def restore_target(manifest, dest, log_dir, tmp_dir, slots=None, members=None, "files": len(restored), "restored": restored, "reconciled": reconciled} +# --------------------------------------------------------------------------- +# Phase-3 leveled-incremental restore (docs/RFC_incremental_v2.md §3.4, P3-3). +# +# A leveled target's manifest ("schema": "leveled_v1", reclaim.is_leveled_manifest) +# ships an L0 baseline plus delta levels off it, in a STAR topology (every delta's +# parent_blonde points at current_l0_blonde, never at the prior delta — RFC §3.3). +# Restore is therefore file-OVERLAY in strict run-time (quaver) order, not the +# disjoint-union restore above: extract each level's zip on top of the +# accumulating tree, then apply that level's recorded deletions. `select_chain`, +# `check_chain_complete`, `extract_level`, `apply_deletions`, and `fold_expected` +# are the level-aware SIBLINGS of `select_objects`/(the completeness gate in +# restore_target)/`extract_object`/(nothing, Phase 2 has no deletions)/(the +# manifest-aggregate expected set in restore_target) respectively — Phase 2's own +# CLI path and restore_target are untouched by any of this. +# --------------------------------------------------------------------------- + +def select_chain(manifest, to_quaver=None): + """ + The restorable chain for a leveled manifest: L0 + its deltas + (`parent_blonde == current_l0_blonde`), sorted by quaver (run-time order — + the ONLY sequencer for replay, RFC §3.3/§3.4). Returns `[]` if the manifest + has no usable L0 (mirrors `archive.effective_catalog`'s own defensive + fallback: prefer the node matching `current_l0_blonde`, else the most + recently appended level==0 node, for a manifest with a stale/missing + pointer). + + `to_quaver`: optional epoch-seconds cutoff — only deltas with + `when_epoch <= to_quaver` are included, for a point-in-time restore. This + mirrors `effective_catalog`'s `upto_quaver` exactly (same nodes, same + ordering), so `restore_leveled_target`'s replay and `fold_expected`'s + reconciliation target always agree on "what state does this cutoff mean." + """ + levels = manifest.get("levels") or [] + l0_blonde = manifest.get("current_l0_blonde") + l0 = next((n for n in levels + if n.get("level") == 0 and n.get("blonde") == l0_blonde), None) + if l0 is None: + zeros = [n for n in levels if n.get("level") == 0] + l0 = zeros[-1] if zeros else None + if l0 is None: + return [] + + def _epoch(node): + return archive.decode_blonde(node["blonde"])["when_epoch"] + + deltas = [n for n in levels + if n.get("level", 0) > 0 and n.get("parent_blonde") == l0["blonde"]] + if to_quaver is not None: + deltas = [n for n in deltas if _epoch(n) <= to_quaver] + deltas.sort(key=_epoch) + return [l0] + deltas + + +def check_chain_complete(chain): + """ + Gate (a) for a leveled restore, BEFORE any transfer: every node in the + chain whose `fortress_tar` is not None (a deletions-only level legitimately + carries no tar) must still exist on Fortress. ANY missing tar means the + chain cannot be replayed end-to-end from L0 forward — this raises + immediately, naming every broken level, rather than letting a later node's + extraction discover the gap after earlier levels are already on disk + ("chain is broken — not restorable", never a partial/silent restore). + """ + missing = [n for n in chain + if n.get("fortress_tar") is not None + and not reclaim.hsi_exists(n["fortress_tar"])] + if missing: + sample = ", ".join( + f"level {n.get('level')} ({n.get('blonde')})" for n in missing[:3]) + raise RestoreError( + f"chain is broken — {len(missing)} of {len(chain)} level tar(s) NOT " + f"found on Fortress (e.g. {sample}); not restorable. Aborting before " + f"any extraction.") + + +def extract_level(node, dest, tmp_dir): + """ + Gate (c) for ONE level (Path A, RFC §3.4): `hsi get` the level's tar to + scratch, `tar xf` it (extracting the inner zip — the level's tar contains + exactly one member, the zip `make_zip_files` built for that level, invariant + #6: never htar_large -xvf through a pipe), then + `zipfile.ZipFile(...).extractall(dest)`. `extractall`'s overwrite-same-name + behavior on a pre-existing dest tree IS the overlay — a later level's member + simply replaces an earlier level's file in place, with no special-casing + needed here. Must never be called on a deletions-only node (`fortress_tar` + is None) — `restore_leveled_target`'s loop guards that; this defends in + depth rather than silently no-op'ing on a bad call. + """ + prepend_hpss_path() + tar = node.get("fortress_tar") + if tar is None: + raise RestoreError( + "extract_level called on a deletions-only node (fortress_tar is " + "None) — this must never happen; the caller's loop should have " + "skipped extraction entirely for this node") + + os.makedirs(tmp_dir, exist_ok=True) + stem = node.get("run_timestamp") or node["blonde"] + local_tar = os.path.join(tmp_dir, f"{stem}.restore.tar") + extract_dir = os.path.join(tmp_dir, f"{stem}.restore.extract") + try: + g = subprocess.run(f'hsi -q "get {local_tar} : {tar}"', shell=True, + stdout=subprocess.PIPE, stderr=subprocess.STDOUT) + if g.returncode != 0: + raise RestoreError( + f"hsi get failed (rc={g.returncode}) for {tar}:\n" + + g.stdout.decode("utf-8", "replace")[-4000:]) + + os.makedirs(extract_dir, exist_ok=True) + x = subprocess.run(["tar", "xf", local_tar], cwd=extract_dir, + stdout=subprocess.PIPE, stderr=subprocess.STDOUT) + if x.returncode != 0: + raise RestoreError( + f"tar extract failed (rc={x.returncode}) for {local_tar}:\n" + + x.stdout.decode("utf-8", "replace")[-4000:]) + + # send_to_fortress names the tar's sole member after the run_timestamp + # (the zip make_zip_files built for this level): {run_timestamp}.zip. + zip_name = f"{node.get('run_timestamp')}.zip" + zip_path = os.path.join(extract_dir, zip_name) + if not os.path.isfile(zip_path): + cands = [f for f in os.listdir(extract_dir) if f.endswith(".zip")] + if len(cands) == 1: + zip_path = os.path.join(extract_dir, cands[0]) + else: + raise RestoreError( + f"expected inner zip {zip_name!r} not found after " + f"extracting {tar} (extract dir has: {os.listdir(extract_dir)})") + + with zipfile.ZipFile(zip_path) as zf: + zf.extractall(dest) + finally: + for p in (local_tar,): + try: + os.remove(p) + except OSError: + pass + shutil.rmtree(extract_dir, ignore_errors=True) + + +def apply_deletions(node, dest): + """ + Apply one level's recorded `deleted` list to `dest` — idempotent (a path + already gone, e.g. from an earlier partial run, is fine; missing is not an + error). Deletions come from the MANIFEST, never inferred from the extracted + tree, which is exactly why Path A's overlay restore needs no GNU + `--listed-incremental` dumpdir baseline to know what to remove (RFC §3.0/ + §3.4). Called for EVERY node in the chain, including deletions-only nodes + (which never call `extract_level` at all). + """ + for rel in node.get("deleted") or (): + full = os.path.join(dest, rel) + try: + os.remove(full) + except FileNotFoundError: + pass + except IsADirectoryError: + shutil.rmtree(full, ignore_errors=True) + + +def fold_expected(manifest, log_dir, to_quaver=None): + """ + Gate (e)'s mandatory final-reconciliation target: the exact arcname set the + restored tree must equal after replaying the chain, reconstructed via + `archive.effective_catalog` — the SAME L0+delta fold `decide_level` diffs + against and `reclaim.verify_chain` reconstructs from, so restore's notion of + "correct" always agrees with the engine's and reclaim's rather than being a + third, independently-computed notion of "current state." + """ + catalog, _total_bytes = archive.effective_catalog( + manifest, log_dir, upto_quaver=to_quaver) + return set(catalog) + + +def verify_level_members(node, dest, log_dir): + """ + Gate (d) for ONE level: every member THIS LEVEL shipped (its own + catalog_ref sibling file — the delta-only add/modify subset for L1+, the + full catalog for L0) must be present in `dest` with the MD5 recorded in + this level's own run-log (`object_runlog_checksums`, reused verbatim — + invariant #2: checksum keys match arcnames). A deletions-only node + (`fortress_tar`/`catalog_ref` both None) has nothing to verify. + """ + ref = node.get("catalog_ref") + if node.get("fortress_tar") is None or not ref: + return + try: + with open(os.path.join(log_dir, ref)) as fh: + catalog = json.load(fh) + except (OSError, ValueError) as e: + raise RestoreError( + f"cannot read catalog_ref {ref!r} for level {node.get('blonde')} " + f"under {log_dir!r} ({e}) — rebuild it from the level's run-log " + f"({node.get('run_timestamp')}.json) if it was lost") + if not catalog: + return + checks = object_runlog_checksums(log_dir, node["run_timestamp"]) + for arc in sorted(catalog): + expected = checks.get(arc) + if expected is None: + raise RestoreError( + f"no recorded MD5 for {arc!r} in run-log " + f"{node['run_timestamp']}.json — checksum keys must match " + f"arcnames (invariant #2)") + full = os.path.join(dest, arc) + if not os.path.isfile(full): + raise RestoreError( + f"member {arc!r} missing from dest after extracting level " + f"{node.get('blonde')} ({node.get('fortress_tar')})") + got = md5_file(full) + if got != expected: + raise RestoreError( + f"MD5 MISMATCH for {arc!r} from level {node.get('blonde')} " + f"({node.get('fortress_tar')}) (expected {expected}, got {got})") + + +def _parse_to_timestamp(to_timestamp): + """ + `--to-timestamp` (a `YYYYMMDD_HHMMSS` string, same shape as a run_timestamp's + trailing stamp) -> epoch seconds, for `select_chain`/`fold_expected`'s + `to_quaver` cutoff. `None` in, `None` out (no cutoff = the full chain to + date). Naive local-time parsing, consistent with how `run_timestamp`s are + generated elsewhere in this codebase (`datetime.datetime.now().strftime(...)`). + """ + if to_timestamp is None: + return None + return datetime.datetime.strptime(to_timestamp, "%Y%m%d_%H%M%S").timestamp() + + +def restore_leveled_target(manifest, dest, log_dir, tmp_dir, to_timestamp=None, + extract=extract_level): + """ + Restore a Phase-3 leveled-incremental target (or a point-in-time prefix of + it) into an empty dest. Implements the RFC §3.4 gate order exactly: + + select_chain -> check_chain_complete -> check_dest -> + for each node in chain (strict L0 -> L1 -> L2 -> ... order): + if fortress_tar is not None: extract_level + verify_level_members + (a deletions-only node skips extraction entirely) + apply_deletions (every node, including deletions-only ones) + -> final walk_restored(dest) == fold_expected(...) or ABORT + + `extract` is injected (mirrors restore_target's `extract`/route_and_ship's + `ship_one`) so tests can mock the tape path. Raises RestoreError on the + first violated gate — nothing partially restored is ever trustworthy. + + Returns a summary dict: badge, levels (chain length), files, restored (set + of arcnames), reconciled (always True on success — a leveled restore has no + selective/--slot mode yet, so reconciliation always runs). + """ + to_quaver = _parse_to_timestamp(to_timestamp) + + chain = select_chain(manifest, to_quaver=to_quaver) + if not chain: + raise RestoreError( + "manifest has no levels (or no usable L0) — nothing to restore") + + check_chain_complete(chain) # (a) — before any transfer + check_dest(dest) # (b) — cheap, no side effects + + os.makedirs(dest, exist_ok=True) + for node in chain: + if node.get("fortress_tar") is not None: + print(f" [RESTORE] level {node.get('level')} ({node.get('blonde')}) " + f"<- {node.get('fortress_tar')}") + extract(node, dest, tmp_dir) # (c) + verify_level_members(node, dest, log_dir) # (d) + print(f" [OK] level {node.get('level')}: {node.get('n_files', 0)} " + f"member MD5(s) verified") + deleted = node.get("deleted") or () + apply_deletions(node, dest) + if deleted: + print(f" [OK] level {node.get('level')}: applied {len(deleted)} " + f"deletion(s)") + + expected = fold_expected(manifest, log_dir, to_quaver=to_quaver) # (e) + actual = walk_restored(dest) + if actual != expected: + extra = sorted(actual - expected) + miss = sorted(expected - actual) + raise RestoreError( + "final reconciliation FAILED — restored tree != manifest chain " + f"({len(extra)} unexpected, e.g. {extra[:3]}; " + f"{len(miss)} missing, e.g. {miss[:3]}). " + "The restored tree must NOT be trusted.") + print(f" [OK] final reconciliation: {len(actual)} restored file(s) == " + f"manifest chain") + + return {"badge": manifest.get("badge"), "levels": len(chain), + "files": len(actual), "restored": actual, "reconciled": True} + + def main(argv=None): ap = argparse.ArgumentParser( - description="Restore a Phase-2 size-routed target from Fortress " - "(docs/RFC_incremental_v2.md §2.9). LOCAL only; empty dest only.") + description="Restore a Phase-2 size-routed OR Phase-3 leveled-incremental " + "target from Fortress (docs/RFC_incremental_v2.md §2.9/§3.4). " + "LOCAL only; empty dest only.") ap.add_argument("config", nargs="?", help="The target's archive config JSON (resolves badge -> " - "{log_dir}/_vault manifest). Alternative: --manifest.") + "{log_dir}/_vault manifest — routing or leveled, whichever " + "the target was actually archived with). Alternative: " + "--manifest.") ap.add_argument("--manifest", - help="Path to a {badge}.manifest.json (bypasses the config)") + help="Path to a {badge}[.leveled].manifest.json (bypasses the config)") ap.add_argument("--dest", required=True, help="Destination directory — MUST be empty (or absent)") ap.add_argument("--log-dir", dest="log_dir", - help="Where the object run-logs (*.json) live. Default: the " + help="Where the object/level run-logs (*.json) live. Default: the " "config's log_dir, or the manifest's _vault parent.") ap.add_argument("--tmp-dir", dest="tmp_dir", help="Scratch for member lists + htar_large staging tars. " "Default: the config's tmp_dir, else .restore-tmp") ap.add_argument("--slot", action="append", dest="slots", metavar="SLOT", - help="Restore only this object (solo: or shard:); repeatable") + help="(Phase-2 routed targets only) restore only this object " + "(solo: or shard:); repeatable") ap.add_argument("--member", action="append", dest="members", metavar="RELPATH", - help="Restore only this file (relpath under source_folder); repeatable") + help="(Phase-2 routed targets only) restore only this file " + "(relpath under source_folder); repeatable") + ap.add_argument("--to-timestamp", dest="to_timestamp", metavar="YYYYMMDD_HHMMSS", + help="(Phase-3 leveled targets only) point-in-time restore: " + "replay only levels up to and including this run " + "timestamp, instead of the full chain to date.") args = ap.parse_args(argv) if bool(args.config) == bool(args.manifest): @@ -332,13 +653,27 @@ def main(argv=None): badge = archive.badge_of(config["project_name"], config["source_folder"], config["file_pattern"]) vault = archive.vault_dir_for(config["log_dir"]) - mpath = archive.manifest_path(vault, badge) - manifest = archive.load_manifest(vault, badge) + # Resolve by EXACT badge (project_name + source_folder + file_pattern), + # never by project-name substring match alone: manifest_for_project + # matches on the manifest's recorded "project" field across every + # manifest in the vault, so two configs sharing a project_name but + # differing in source_folder/file_pattern (distinct badges — the exact + # scenario badge_of's 3-field hash exists to disambiguate) could + # otherwise silently resolve to the WRONG target's manifest here, and + # every downstream gate (per-member MD5, reconciliation) would then + # "pass" against data restore never asked for. Prefer the leveled + # manifest when both schemas exist for this badge (the mid-Phase-2-> + # Phase-3-conversion collision state) — same tie-break as + # manifest_for_project, just scoped to the one badge this config maps to. + manifest = (archive.load_manifest(vault, badge, kind="leveled") + or archive.load_manifest(vault, badge, kind="routing")) if manifest is None: - print(f" [ABORT] no size-routing manifest for " - f"{config['project_name']!r} at {mpath} — was this target ever " - f"archived with size_routing on?") + print(f" [ABORT] no size-routing or leveled-incremental manifest for " + f"{config['project_name']!r} under {vault} (badge {badge}) — " + f"was this target ever archived with size_routing or incremental on?") sys.exit(1) + kind = "leveled" if reclaim.is_leveled_manifest(manifest) else "routing" + mpath = archive.manifest_path(vault, badge, kind=kind) log_dir = args.log_dir or config["log_dir"] tmp_dir = args.tmp_dir or config["tmp_dir"] @@ -346,6 +681,38 @@ def main(argv=None): if not tmp_dir: tmp_dir = dest.rstrip(os.sep) + ".restore-tmp" + leveled = reclaim.is_leveled_manifest(manifest) + + if leveled: + if args.slots or args.members: + ap.error("--slot/--member selection is not supported for Phase-3 " + "leveled targets (only full or --to-timestamp restore); " + "omit them") + + print("#" * 64) + print("# FORTRESS RESTORE — Phase-3 leveled-incremental target (RFC §3.4), LOCAL") + print(f"# badge: {manifest.get('badge')} project: {manifest.get('project')}") + print(f"# manifest: {mpath}") + print(f"# levels: {len(manifest.get('levels') or [])}" + + (f" (up to {args.to_timestamp})" if args.to_timestamp else " (FULL chain)")) + print(f"# dest: {dest}") + print("#" * 64) + + try: + summary = restore_leveled_target(manifest, dest, log_dir, tmp_dir, + to_timestamp=args.to_timestamp) + except RestoreError as e: + print(f"\n [ABORT] {e}") + print(" Nothing restored is trustworthy until every gate passes — if dest " + "was partially written, remove it before re-running.") + sys.exit(1) + + scope = (f"point-in-time chain restore (up to {args.to_timestamp}), reconciled" + if args.to_timestamp else "FULL chain restore, reconciled") + print(f"\n [SUCCESS] {scope}: {summary['files']} " + f"file(s) from {summary['levels']} level(s) -> {dest}") + return + print("#" * 64) print("# FORTRESS RESTORE — Phase-2 routed target (RFC §2.9), LOCAL") print(f"# badge: {manifest.get('badge')} project: {manifest.get('project')}") diff --git a/tests/test_incremental_engine.py b/tests/test_incremental_engine.py new file mode 100644 index 0000000..c7a846d --- /dev/null +++ b/tests/test_incremental_engine.py @@ -0,0 +1,648 @@ +""" +Unit tests for the Phase-3 leveled-incremental ENGINE (P3-1, +docs/RFC_incremental_v2.md §3): the level decision (decide_level), the +catalog fold (effective_catalog), the sibling catalog writer +(write_level_catalog), the manifest kind= extension (Phase 2/Phase 3 +coexistence), the per-badge advisory lock (badge_lock), and the LOCAL +orchestration (level_and_ship). + +tests/test_incremental_primitives.py already covers the merged PR#8 +primitives (badge_of/make_blonde/decode_blonde/floor_s/enumerate_source_catalog) +and is NOT extended here — this file is fully separate, per the plan. + +No tape/htar/hsi/Slurm/Globus is touched: ship_level is always injected. +""" +import hashlib +import json +import os +import sys +import tempfile +import threading +import time +import unittest + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +import archive # noqa: E402 + + +def cat(**entries): + """catalog from name=(mtime_s, size); mtime given in whole seconds.""" + return {a: (mt * 1_000_000_000, sz) for a, (mt, sz) in entries.items()} + + +# --------------------------------------------------------------------------- +# effective_catalog +# --------------------------------------------------------------------------- + +class TestEffectiveCatalog(unittest.TestCase): + def _node(self, log_dir, blonde, level, parent_blonde, run_timestamp, + catalog_subset, deleted=()): + ref, sha = archive.write_level_catalog(log_dir, run_timestamp, catalog_subset) + return { + "blonde": blonde, "level": level, "parent_blonde": parent_blonde, + "run_timestamp": run_timestamp, "catalog_ref": ref, + "catalog_sha256": sha, "deleted": sorted(deleted), + } + + def test_empty_manifest_returns_empty(self): + got, total = archive.effective_catalog(None, "/nonexistent") + self.assertEqual(got, {}) + self.assertEqual(total, 0) + got2, total2 = archive.effective_catalog({"levels": []}, "/nonexistent") + self.assertEqual(got2, {}) + self.assertEqual(total2, 0) + + def test_l0_only(self): + with tempfile.TemporaryDirectory() as d: + badge = "b1" + l0_blonde = archive.make_blonde(badge, 1000, "0") + c = cat(a=(1, 10), b=(1, 20)) + l0 = self._node(d, l0_blonde, 0, None, "run_l0", c) + manifest = {"current_l0_blonde": l0_blonde, "levels": [l0]} + got, total = archive.effective_catalog(manifest, d) + self.assertEqual(set(got), {"a", "b"}) + self.assertEqual(got["a"], (1_000_000_000, 10)) + self.assertEqual(total, 30) + + def test_plus_one_delta_add_and_modify(self): + with tempfile.TemporaryDirectory() as d: + badge = "b1" + l0_blonde = archive.make_blonde(badge, 1000, "0") + l0 = self._node(d, l0_blonde, 0, None, "run_l0", cat(a=(1, 10), b=(1, 20))) + d1_blonde = archive.make_blonde(badge, 1100, "1") + d1 = self._node(d, d1_blonde, 1, l0_blonde, "run_l1", + cat(a=(5, 15), c=(5, 5))) + manifest = {"current_l0_blonde": l0_blonde, "levels": [l0, d1]} + got, total = archive.effective_catalog(manifest, d) + self.assertEqual(set(got), {"a", "b", "c"}) + self.assertEqual(got["a"], (5_000_000_000, 15)) # modified value wins + self.assertEqual(got["b"], (1_000_000_000, 20)) # untouched by delta + self.assertEqual(total, 15 + 20 + 5) + + def test_plus_n_overlapping_modify_and_delete(self): + with tempfile.TemporaryDirectory() as d: + badge = "b1" + l0_blonde = archive.make_blonde(badge, 1000, "0") + l0 = self._node(d, l0_blonde, 0, None, "run_l0", + cat(a=(1, 10), b=(1, 20), x=(1, 1))) + d1_blonde = archive.make_blonde(badge, 1100, "1") + d1 = self._node(d, d1_blonde, 1, l0_blonde, "run_l1", + cat(a=(5, 15)), deleted=["x"]) + d2_blonde = archive.make_blonde(badge, 1200, "2") + d2 = self._node(d, d2_blonde, 2, l0_blonde, "run_l2", + cat(a=(9, 9), c=(9, 3))) + manifest = {"current_l0_blonde": l0_blonde, "levels": [l0, d1, d2]} + got, total = archive.effective_catalog(manifest, d) + # x deleted at d1; a re-modified at d2 (d2's value wins); c added at d2; + # b never touched. + self.assertEqual(set(got), {"a", "b", "c"}) + self.assertEqual(got["a"], (9_000_000_000, 9)) + self.assertNotIn("x", got) + self.assertEqual(total, 9 + 20 + 3) + + def test_upto_quaver_cutoff(self): + with tempfile.TemporaryDirectory() as d: + badge = "b1" + l0_blonde = archive.make_blonde(badge, 1000, "0") + l0 = self._node(d, l0_blonde, 0, None, "run_l0", cat(a=(1, 10))) + d1_blonde = archive.make_blonde(badge, 1100, "1") + d1 = self._node(d, d1_blonde, 1, l0_blonde, "run_l1", cat(b=(1, 1))) + d2_blonde = archive.make_blonde(badge, 1200, "2") + d2 = self._node(d, d2_blonde, 2, l0_blonde, "run_l2", cat(c=(1, 1))) + manifest = {"current_l0_blonde": l0_blonde, "levels": [l0, d1, d2]} + got, _ = archive.effective_catalog(manifest, d, upto_quaver=1150) + self.assertEqual(set(got), {"a", "b"}) # d2 (epoch 1200) excluded + + def test_missing_catalog_file_degrades_to_empty_contribution(self): + with tempfile.TemporaryDirectory() as d: + badge = "b1" + l0_blonde = archive.make_blonde(badge, 1000, "0") + l0 = {"blonde": l0_blonde, "level": 0, "parent_blonde": None, + "run_timestamp": "run_l0", "catalog_ref": "does_not_exist.json", + "catalog_sha256": "x", "deleted": []} + manifest = {"current_l0_blonde": l0_blonde, "levels": [l0]} + got, total = archive.effective_catalog(manifest, d) + self.assertEqual(got, {}) + self.assertEqual(total, 0) + + +# --------------------------------------------------------------------------- +# decide_level +# --------------------------------------------------------------------------- + +class TestDecideLevel(unittest.TestCase): + BADGE = "badge1" + + def _manifest_with_l0(self, log_dir, catalog, epoch=1000): + blonde = archive.make_blonde(self.BADGE, epoch, "0") + ref, sha = archive.write_level_catalog(log_dir, "l0_run", catalog) + node = {"blonde": blonde, "level": 0, "parent_blonde": None, + "run_timestamp": "l0_run", "catalog_ref": ref, + "catalog_sha256": sha, "deleted": []} + return {"current_l0_blonde": blonde, "levels": [node]} + + def test_baseline_no_manifest(self): + live = cat(a=(1, 10), b=(1, 5)) + decision = archive.decide_level(self.BADGE, live, None, False, "/x") + self.assertEqual(decision["detail"], "0") + self.assertIsNone(decision["parent"]) + self.assertEqual(decision["members"], ["a", "b"]) + self.assertEqual(decision["deleted"], []) + + def test_baseline_manifest_with_no_levels(self): + live = cat(a=(1, 10)) + decision = archive.decide_level(self.BADGE, live, {"levels": []}, False, "/x") + self.assertEqual(decision["detail"], "0") + + def test_force_full_ignores_existing_manifest(self): + with tempfile.TemporaryDirectory() as d: + c = cat(a=(1, 10)) + manifest = self._manifest_with_l0(d, c) + decision = archive.decide_level(self.BADGE, c, manifest, True, d) + self.assertEqual(decision["detail"], "0") + self.assertIsNone(decision["parent"]) + self.assertEqual(decision["members"], ["a"]) + + def test_append_only(self): + with tempfile.TemporaryDirectory() as d: + c1 = cat(a=(1, 10)) + manifest = self._manifest_with_l0(d, c1) + live = cat(a=(1, 10), b=(2, 5)) + decision = archive.decide_level(self.BADGE, live, manifest, False, d) + self.assertEqual(decision["detail"], "1") + self.assertEqual(decision["parent"], manifest["current_l0_blonde"]) + self.assertEqual(decision["members"], ["b"]) + self.assertEqual(decision["deleted"], []) + + def test_modify_only(self): + with tempfile.TemporaryDirectory() as d: + c1 = cat(a=(1, 10), b=(1, 5)) + manifest = self._manifest_with_l0(d, c1) + live = cat(a=(9, 99), b=(1, 5)) # a's mtime+size both changed + decision = archive.decide_level(self.BADGE, live, manifest, False, d) + self.assertEqual(decision["members"], ["a"]) + self.assertEqual(decision["deleted"], []) + + def test_modify_detected_by_size_alone(self): + with tempfile.TemporaryDirectory() as d: + c1 = cat(a=(1, 10)) + manifest = self._manifest_with_l0(d, c1) + live = cat(a=(1, 999)) # same mtime bucket, size differs + decision = archive.decide_level(self.BADGE, live, manifest, False, d) + self.assertEqual(decision["members"], ["a"]) + + def test_sub_second_jitter_is_not_modified(self): + with tempfile.TemporaryDirectory() as d: + c1 = {"a": (1_500_000_000_100_000_000, 10)} + manifest = self._manifest_with_l0(d, c1) + live = {"a": (1_500_000_000_900_000_000, 10)} # same floored second + decision = archive.decide_level(self.BADGE, live, manifest, False, d) + self.assertIsNone(decision) # true NOOP, not a phantom modify + + def test_delete_only_is_deletions_only_decision(self): + with tempfile.TemporaryDirectory() as d: + c1 = cat(a=(1, 10), b=(1, 5)) + manifest = self._manifest_with_l0(d, c1) + live = cat(a=(1, 10)) + decision = archive.decide_level(self.BADGE, live, manifest, False, d) + self.assertEqual(decision["members"], []) + self.assertEqual(decision["deleted"], ["b"]) + self.assertEqual(decision["detail"], "1") + + def test_true_noop(self): + with tempfile.TemporaryDirectory() as d: + c1 = cat(a=(1, 10), b=(1, 5)) + manifest = self._manifest_with_l0(d, c1) + live = cat(a=(1, 10), b=(1, 5)) + decision = archive.decide_level(self.BADGE, live, manifest, False, d) + self.assertIsNone(decision) + + def test_forced_rebaseline_via_byte_sum(self): + # decide_level's own per-arcname diff already compares size, so a + # byte-sum mismatch while added/removed/modified are all empty cannot + # arise from decide_level's own effective_catalog call in practice + # (see decide_level's docstring) -- it is still a required + # defense-in-depth branch (mirrors reclaim.py's byte-sum guard). Test + # the branch directly by mocking effective_catalog's return so the + # per-arcname view is consistent (no add/remove/modify) while the + # reported total deliberately disagrees. + live = cat(a=(1, 10), b=(1, 5)) + prior = {"a": (1_000_000_000, 10), "b": (1_000_000_000, 5)} + blonde = archive.make_blonde(self.BADGE, 1000, "0") + manifest = {"current_l0_blonde": blonde, "levels": [{"blonde": blonde, "level": 0}]} + orig = archive.effective_catalog + archive.effective_catalog = lambda m, ld, upto_quaver=None: (prior, 999999) + try: + decision = archive.decide_level(self.BADGE, live, manifest, False, "/x") + finally: + archive.effective_catalog = orig + self.assertIsNotNone(decision) + self.assertTrue(decision.get("forced_rebaseline")) + self.assertEqual(decision["parent"], blonde) + self.assertEqual(decision["detail"], "1") + self.assertEqual(decision["members"], ["a", "b"]) + self.assertEqual(decision["deleted"], []) + + def test_second_delta_numbers_incrementally(self): + with tempfile.TemporaryDirectory() as d: + c1 = cat(a=(1, 10)) + manifest = self._manifest_with_l0(d, c1) + l0_blonde = manifest["current_l0_blonde"] + d1_ref, d1_sha = archive.write_level_catalog(d, "d1_run", cat(b=(2, 5))) + manifest["levels"].append({ + "blonde": archive.make_blonde(self.BADGE, 1100, "1"), "level": 1, + "parent_blonde": l0_blonde, "run_timestamp": "d1_run", + "catalog_ref": d1_ref, "catalog_sha256": d1_sha, "deleted": [], + }) + live = cat(a=(1, 10), b=(2, 5), c=(3, 1)) + decision = archive.decide_level(self.BADGE, live, manifest, False, d) + self.assertEqual(decision["detail"], "2") + self.assertEqual(decision["members"], ["c"]) + + +# --------------------------------------------------------------------------- +# write_level_catalog +# --------------------------------------------------------------------------- + +class TestWriteLevelCatalog(unittest.TestCase): + def test_round_trip_and_hash(self): + with tempfile.TemporaryDirectory() as d: + catalog = cat(a=(1, 10), b=(2, 20)) + ref, sha = archive.write_level_catalog(d, "run_1", catalog) + path = os.path.join(d, ref) + self.assertTrue(os.path.exists(path)) + with open(path, "rb") as fh: + data = fh.read() + self.assertEqual(hashlib.sha256(data).hexdigest(), sha) + loaded = json.loads(data) + self.assertEqual(set(loaded), {"a", "b"}) + self.assertEqual(loaded["a"], [1_000_000_000, 10]) + self.assertEqual(loaded["b"], [2_000_000_000, 20]) + + def test_ref_is_bare_filename(self): + with tempfile.TemporaryDirectory() as d: + ref, _ = archive.write_level_catalog(d, "run_2", cat(a=(1, 1))) + self.assertNotIn(os.sep, ref) + self.assertEqual(ref, "run_2.catalog.json") + + def test_atomic_no_tmp_left_behind(self): + with tempfile.TemporaryDirectory() as d: + archive.write_level_catalog(d, "run_3", cat(a=(1, 1))) + self.assertFalse(os.path.exists(os.path.join(d, "run_3.catalog.json.tmp"))) + + def test_empty_subset_round_trips(self): + with tempfile.TemporaryDirectory() as d: + ref, sha = archive.write_level_catalog(d, "run_4", {}) + with open(os.path.join(d, ref)) as fh: + self.assertEqual(json.load(fh), {}) + + +# --------------------------------------------------------------------------- +# manifest_path / load_manifest / write_manifest -- kind= extension +# --------------------------------------------------------------------------- + +class TestManifestKind(unittest.TestCase): + def test_default_kind_is_unchanged_routing_filename(self): + with tempfile.TemporaryDirectory() as d: + p = archive.manifest_path(d, "b1") + self.assertEqual(os.path.basename(p), "b1.manifest.json") + + def test_leveled_kind_filename(self): + with tempfile.TemporaryDirectory() as d: + p = archive.manifest_path(d, "b1", kind="leveled") + self.assertEqual(os.path.basename(p), "b1.leveled.manifest.json") + + def test_leveled_glob_still_matches_manifest_json_star(self): + import glob + with tempfile.TemporaryDirectory() as d: + archive.write_manifest(d, "b1", {"schema": "leveled_v1"}, kind="leveled") + matches = glob.glob(os.path.join(d, "*.manifest.json")) + self.assertEqual(len(matches), 1) + self.assertTrue(matches[0].endswith("b1.leveled.manifest.json")) + + def test_collision_routing_and_leveled_coexist(self): + with tempfile.TemporaryDirectory() as d: + badge = "b1" + routing_manifest = {"schema_version": 1, "badge": badge, "objects": ["r"]} + leveled_manifest = {"schema_version": 1, "schema": "leveled_v1", + "badge": badge, "levels": ["l"]} + archive.write_manifest(d, badge, routing_manifest) # kind="routing" + archive.write_manifest(d, badge, leveled_manifest, kind="leveled") + + got_routing = archive.load_manifest(d, badge) + got_leveled = archive.load_manifest(d, badge, kind="leveled") + self.assertEqual(got_routing["objects"], ["r"]) + self.assertEqual(got_leveled["levels"], ["l"]) + self.assertNotEqual( + archive.manifest_path(d, badge), + archive.manifest_path(d, badge, kind="leveled")) + self.assertEqual(len(os.listdir(d)), 2) # both files present, independently + + # Re-writing one must never touch the other. + routing_manifest["objects"].append("r2") + archive.write_manifest(d, badge, routing_manifest) + self.assertEqual(archive.load_manifest(d, badge)["objects"], ["r", "r2"]) + self.assertEqual(archive.load_manifest(d, badge, kind="leveled")["levels"], ["l"]) + + +# --------------------------------------------------------------------------- +# badge_lock +# --------------------------------------------------------------------------- + +class TestBadgeLock(unittest.TestCase): + def test_context_manager_usage(self): + with tempfile.TemporaryDirectory() as d: + with archive.badge_lock(d, "b2"): + pass + self.assertTrue(os.path.exists(os.path.join(d, "b2.lock"))) + + def test_blocks_a_second_acquirer_until_released(self): + with tempfile.TemporaryDirectory() as d: + badge = "b1" + lock1 = archive.badge_lock(d, badge) + lock1.__enter__() + acquired = threading.Event() + + def try_acquire(): + lock2 = archive.badge_lock(d, badge) + lock2.__enter__() + acquired.set() + lock2.__exit__(None, None, None) + + t = threading.Thread(target=try_acquire) + t.start() + try: + time.sleep(0.3) + self.assertFalse(acquired.is_set(), "second acquirer should still be blocked") + finally: + lock1.__exit__(None, None, None) + t.join(timeout=2) + self.assertTrue(acquired.is_set(), "second acquirer should proceed after release") + + def test_lock_released_on_exception(self): + with tempfile.TemporaryDirectory() as d: + badge = "b3" + with self.assertRaises(ValueError): + with archive.badge_lock(d, badge): + raise ValueError("boom") + # A fresh acquire must succeed immediately (not deadlocked). + acquired = threading.Event() + + def try_acquire(): + with archive.badge_lock(d, badge): + acquired.set() + + t = threading.Thread(target=try_acquire) + t.start() + t.join(timeout=2) + self.assertTrue(acquired.is_set()) + + +# --------------------------------------------------------------------------- +# level_and_ship orchestration +# --------------------------------------------------------------------------- + +class LevelRecorder: + """Injectable ship_level: records the level_obj passed, returns plausible metadata.""" + def __init__(self, catalog, raise_on_call=None): + self.catalog = catalog + self.calls = [] + self.raise_on_call = raise_on_call + + def __call__(self, level_obj, stem): + self.calls.append(level_obj) + if self.raise_on_call is not None and len(self.calls) == self.raise_on_call: + raise RuntimeError("simulated tape failure") + sb = sum(self.catalog[a][1] for a in level_obj["members"]) + return {"fortress_tar": "/tape/%s.tar" % stem, "zip_checksum": "md5-%s" % stem, + "n_files": len(level_obj["members"]), "source_bytes": sb, + "run_timestamp": "%s_20260101_000000" % stem} + + +def lship(catalog, vault, log_dir, recorder, force_full=False, exclusion=None, + now_epoch=1_700_000_000): + return archive.level_and_ship( + "proj", "/src", ".*", catalog, exclusion or {"status": "OFF"}, + vault, log_dir, now_epoch, recorder, force_full=force_full) + + +class TestLevelAndShip(unittest.TestCase): + def test_baseline_ships_all_and_writes_leveled_manifest(self): + with tempfile.TemporaryDirectory() as d: + vault = archive.vault_dir_for(d) + c = cat(a=(1, 10), b=(1, 20)) + rec = LevelRecorder(c) + summ = lship(c, vault, d, rec) + self.assertEqual(summ["status"], "shipped") + self.assertEqual(summ["level"], 0) + self.assertEqual(len(rec.calls), 1) + self.assertEqual(set(rec.calls[0]["members"]), {"a", "b"}) + self.assertIsNone(rec.calls[0]["parent_blonde"]) + + m = archive.load_manifest(vault, summ["badge"], kind="leveled") + self.assertEqual(m["schema"], "leveled_v1") + self.assertEqual(len(m["levels"]), 1) + self.assertEqual(m["current_l0_blonde"], summ["blonde"]) + self.assertEqual(m["levels"][0]["fortress_tar"], "/tape/proj__L0.tar") + # Phase-2 manifest for the same badge must not exist / be confused. + self.assertIsNone(archive.load_manifest(vault, summ["badge"])) + + def test_append_ships_delta_only(self): + with tempfile.TemporaryDirectory() as d: + vault = archive.vault_dir_for(d) + c1 = cat(a=(1, 10)) + lship(c1, vault, d, LevelRecorder(c1)) + c2 = cat(a=(1, 10), b=(2, 5)) + rec2 = LevelRecorder(c2) + summ = lship(c2, vault, d, rec2, now_epoch=1_700_000_100) + self.assertEqual(summ["status"], "shipped") + self.assertEqual(summ["level"], 1) + self.assertEqual(rec2.calls[0]["members"], ["b"]) # only the new file + + m = archive.load_manifest(vault, summ["badge"], kind="leveled") + self.assertEqual(len(m["levels"]), 2) + self.assertEqual(m["current_l0_blonde"], m["levels"][0]["blonde"]) # L0 unchanged + self.assertEqual(m["levels"][1]["parent_blonde"], m["levels"][0]["blonde"]) + + def test_delete_only_never_calls_ship_level(self): + with tempfile.TemporaryDirectory() as d: + vault = archive.vault_dir_for(d) + c1 = cat(a=(1, 10), b=(1, 5)) + lship(c1, vault, d, LevelRecorder(c1)) + c2 = cat(a=(1, 10)) # b deleted + rec2 = LevelRecorder(c2) + summ = lship(c2, vault, d, rec2, now_epoch=1_700_000_100) + self.assertEqual(summ["status"], "deletions_only") + self.assertEqual(rec2.calls, []) # ship_level never invoked + self.assertEqual(summ["deleted"], ["b"]) + + m = archive.load_manifest(vault, summ["badge"], kind="leveled") + self.assertEqual(len(m["levels"]), 2) + self.assertIsNone(m["levels"][1]["fortress_tar"]) + self.assertEqual(m["levels"][1]["deleted"], ["b"]) + + def test_noop_no_ship_and_no_new_level_but_events_recorded(self): + with tempfile.TemporaryDirectory() as d: + vault = archive.vault_dir_for(d) + c = cat(a=(1, 10), b=(1, 5)) + lship(c, vault, d, LevelRecorder(c)) + rec2 = LevelRecorder(c) + summ = lship(c, vault, d, rec2, now_epoch=1_700_000_100) + self.assertEqual(summ["status"], "noop") + self.assertEqual(rec2.calls, []) + + m = archive.load_manifest(vault, summ["badge"], kind="leveled") + self.assertEqual(len(m["levels"]), 1) # no new level node from a NOOP + self.assertEqual(len(m["events"]), 1) + self.assertEqual(m["events"][0]["type"], "noop") + + def test_resume_after_failure_retries_cleanly(self): + with tempfile.TemporaryDirectory() as d: + vault = archive.vault_dir_for(d) + c1 = cat(a=(1, 10)) + lship(c1, vault, d, LevelRecorder(c1)) + + c2 = cat(a=(1, 10), b=(2, 5)) + failing = LevelRecorder(c2, raise_on_call=1) + with self.assertRaises(RuntimeError): + lship(c2, vault, d, failing, now_epoch=1_700_000_100) + + badge = archive.badge_of("proj", "/src", ".*") + m = archive.load_manifest(vault, badge, kind="leveled") + self.assertEqual(len(m["levels"]), 1) # the failed delta was never recorded + + rec2 = LevelRecorder(c2) + summ = lship(c2, vault, d, rec2, now_epoch=1_700_000_200) + self.assertEqual(summ["status"], "shipped") + self.assertEqual(rec2.calls[0]["members"], ["b"]) + m2 = archive.load_manifest(vault, badge, kind="leveled") + self.assertEqual(len(m2["levels"]), 2) + + def test_force_full_reships_baseline_even_with_existing_l0(self): + with tempfile.TemporaryDirectory() as d: + vault = archive.vault_dir_for(d) + c = cat(a=(1, 10), b=(1, 5)) + lship(c, vault, d, LevelRecorder(c)) + rec2 = LevelRecorder(c) + summ = lship(c, vault, d, rec2, force_full=True, now_epoch=1_700_000_100) + self.assertEqual(summ["status"], "shipped") + self.assertEqual(summ["level"], 0) + self.assertIsNone(rec2.calls[0]["parent_blonde"]) + m = archive.load_manifest(vault, summ["badge"], kind="leveled") + self.assertEqual(len(m["levels"]), 2) # a second, independent L0 node + self.assertEqual(m["current_l0_blonde"], m["levels"][1]["blonde"]) + + def test_catalog_built_from_reported_zip_members_not_decision_set(self): + # ship_level honestly reports a `zip_members` narrower than the decision + # `members` (simulating a file present at enumerate/decide time but gone + # by zip time) -- level_and_ship must raise rather than record a manifest + # catalog entry for a file that isn't actually on tape. + with tempfile.TemporaryDirectory() as d: + vault = archive.vault_dir_for(d) + c = cat(a=(1, 10), b=(1, 5)) + + def vanishing_ship_level(level_obj, stem): + shipped = [m for m in level_obj["members"] if m != "b"] + sb = sum(c[a][1] for a in shipped) + return {"fortress_tar": "/tape/%s.tar" % stem, + "zip_checksum": "md5-%s" % stem, "n_files": len(shipped), + "source_bytes": sb, "run_timestamp": "%s_20260101_000000" % stem, + "zip_members": shipped} + + with self.assertRaises(RuntimeError) as ctx: + lship(c, vault, d, vanishing_ship_level) + self.assertIn("b", str(ctx.exception)) + + # No manifest state is left behind by the aborted ship. + badge = archive.badge_of("proj", "/src", ".*") + self.assertIsNone(archive.load_manifest(vault, badge, kind="leveled")) + + def test_catalog_matches_reported_zip_members_when_all_present(self): + # Normal path: ship_level reports zip_members == the decision set (the + # production wiring's common case) -- catalog_ref must reflect exactly + # what shipped, and no error is raised. + with tempfile.TemporaryDirectory() as d: + vault = archive.vault_dir_for(d) + c = cat(a=(1, 10), b=(1, 5)) + + def full_ship_level(level_obj, stem): + shipped = list(level_obj["members"]) + sb = sum(c[a][1] for a in shipped) + return {"fortress_tar": "/tape/%s.tar" % stem, + "zip_checksum": "md5-%s" % stem, "n_files": len(shipped), + "source_bytes": sb, "run_timestamp": "%s_20260101_000000" % stem, + "zip_members": shipped} + + summ = lship(c, vault, d, full_ship_level) + self.assertEqual(summ["status"], "shipped") + m = archive.load_manifest(vault, summ["badge"], kind="leveled") + ref = m["levels"][0]["catalog_ref"] + with open(os.path.join(d, ref)) as fh: + catalog = json.load(fh) + self.assertEqual(set(catalog.keys()), {"a", "b"}) + + def test_tar_bytes_propagated_from_ship_level_into_manifest_node(self): + # RFC §3.3: tar_bytes is a day-one reclaim-readiness field, distinct + # from uncompressed source_bytes. When ship_level reports it (as the + # real send_to_fortress-backed production wiring now does), the + # manifest node must carry it through, not hardcode None. + with tempfile.TemporaryDirectory() as d: + vault = archive.vault_dir_for(d) + c = cat(a=(1, 10), b=(1, 5)) + + def ship_level_with_tar_bytes(level_obj, stem): + shipped = list(level_obj["members"]) + sb = sum(c[a][1] for a in shipped) + return {"fortress_tar": "/tape/%s.tar" % stem, + "zip_checksum": "md5-%s" % stem, "n_files": len(shipped), + "source_bytes": sb, "run_timestamp": "%s_20260101_000000" % stem, + "zip_members": shipped, "tar_bytes": 424242} + + summ = lship(c, vault, d, ship_level_with_tar_bytes) + m = archive.load_manifest(vault, summ["badge"], kind="leveled") + self.assertEqual(m["levels"][0]["tar_bytes"], 424242) + + +# --------------------------------------------------------------------------- +# load_config -- incremental key / mutual exclusion / durability guard +# --------------------------------------------------------------------------- + +class TestConfigIncremental(unittest.TestCase): + def _write_cfg(self, d, **over): + cfg = {"project_name": "p", "source_folder": "/s", "file_pattern": ".*", + "fortress_base_dir": "/f", "emails": ["a@b.c"]} + cfg.update(over) + path = os.path.join(d, "c.json") + with open(path, "w") as fh: + json.dump(cfg, fh) + return path + + def test_defaults_off(self): + with tempfile.TemporaryDirectory() as d: + c = archive.load_config(self._write_cfg(d)) + self.assertFalse(c["incremental"]) + + def test_mutually_exclusive_with_size_routing(self): + with tempfile.TemporaryDirectory() as d: + p = self._write_cfg(d, size_routing=True, incremental=True, + log_dir="/depot/x/logs") + with self.assertRaises(ValueError): + archive.load_config(p) + + def test_durability_guard_applies_to_incremental(self): + with tempfile.TemporaryDirectory() as d: + p = self._write_cfg(d, incremental=True, + log_dir=os.path.expanduser("~/globus_archive_tmp")) + with self.assertRaises(ValueError): + archive.load_config(p) + + def test_durability_guard_accepts_depot_logdir_for_incremental(self): + with tempfile.TemporaryDirectory() as d: + p = self._write_cfg(d, incremental=True, + log_dir="/depot/florasense/etc/logs_fortress") + c = archive.load_config(p) + self.assertTrue(c["incremental"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_leveled_reclaim.py b/tests/test_leveled_reclaim.py new file mode 100644 index 0000000..787e3c1 --- /dev/null +++ b/tests/test_leveled_reclaim.py @@ -0,0 +1,498 @@ +""" +Unit tests for Phase-3 leveled-incremental reclaim (docs/RFC_incremental_v2.md §3.5.2, +P3-2): verify_chain() reconstructs a leveled target from its manifest by replaying the +level tree via archive.effective_catalog (L0 UNION deltas, add/modify applied, deletions +removed) and returns the same verdict vocabulary as verify()/verify_target(). Also covers +is_leveled_manifest, the manifest_for_project Phase-2/Phase-3 disambiguation fix, and the +--scan loop's leveled_object skip + manifest routing split. Tape is mocked +(reclaim.hsi_exists); no live Slurm/Globus/hsi calls. + +Mirrors tests/test_routing_reclaim.py's mock/fixture conventions. +""" +import contextlib +import io +import json +import os +import sys +import tempfile +import time +import unittest + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +import archive # noqa: E402 +import reclaim # noqa: E402 + + +L0_EPOCH = 1577836800 # 2020-01-01T00:00:00Z — arbitrary, well before any live mtime +DELTA_EPOCH = 1580515200 # 2020-02-01T00:00:00Z + + +def _write(path, body, mtime=None): + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "wb") as fh: + fh.write(body) + if mtime is not None: + os.utime(path, (mtime, mtime)) + return len(body) + + +def _catalog_from(arcnames, src): + """{arcname: (mtime_ns, size)} straight off disk for the given arcnames.""" + cat = {} + for arc in arcnames: + st = os.stat(os.path.join(src, arc)) + cat[arc] = (st.st_mtime_ns, st.st_size) + return cat + + +def build_leveled_target(d, badge="tstbadge", project="proj", with_delta=False): + """ + Create a source tree + a matching Phase-3 leveled manifest (L0, optionally + one + genuine delta) with real sibling catalog files under log_dir, built via archive.py's + own write_level_catalog/make_blonde so the fixture matches production shape exactly. + Returns (src, log_dir, manifest). By default (with_delta=False) the manifest exactly + matches the live source tree -> SAFE. + """ + src = os.path.join(d, "src") + log_dir = os.path.join(d, "logs") + + _write(os.path.join(src, "a.bin"), b"A" * 100) + _write(os.path.join(src, "b.bin"), b"B" * 200) + _write(os.path.join(src, "sub", "c.bin"), b"C" * 50) + + l0_catalog = archive.enumerate_source_catalog(src, ".*") + l0_ts = "%s__L0_20200101_000000" % project + catalog_ref, catalog_sha256 = archive.write_level_catalog(log_dir, l0_ts, l0_catalog) + l0_blonde = archive.make_blonde(badge, L0_EPOCH, "0") + l0_node = { + "blonde": l0_blonde, "level": 0, "parent_blonde": None, + "run_timestamp": l0_ts, "transfer": "zip", + "fortress_tar": "/tape/%s_L0.tar" % project, + "snar_fortress_path": None, "zip_checksum": "l0md5", "tar_bytes": 1234, + "n_files": len(l0_catalog), + "source_bytes": sum(v[1] for v in l0_catalog.values()), + "catalog_ref": catalog_ref, "catalog_sha256": catalog_sha256, "deleted": [], + } + levels = [l0_node] + + if with_delta: + # A genuine delta: rewrite b.bin with new content + a later mtime, exactly + # what decide_level would ship as an L1 (added ∪ modified = {"b.bin"}). + _write(os.path.join(src, "b.bin"), b"BB" * 150, mtime=time.time() + 5) + delta_catalog = _catalog_from(["b.bin"], src) + delta_ts = "%s__L1_20200201_000000" % project + d_ref, d_sha = archive.write_level_catalog(log_dir, delta_ts, delta_catalog) + delta_node = { + "blonde": archive.make_blonde(badge, DELTA_EPOCH, "1"), + "level": 1, "parent_blonde": l0_blonde, + "run_timestamp": delta_ts, "transfer": "zip", + "fortress_tar": "/tape/%s_L1.tar" % project, + "snar_fortress_path": None, "zip_checksum": "l1md5", "tar_bytes": 56, + "n_files": 1, "source_bytes": sum(v[1] for v in delta_catalog.values()), + "catalog_ref": d_ref, "catalog_sha256": d_sha, "deleted": [], + } + levels.append(delta_node) + + manifest = { + "schema_version": 1, "schema": "leveled_v1", "badge": badge, "project": project, + "source_folder": src, "file_pattern": ".*", + "exclusion": {"status": "OFF"}, + "current_l0_blonde": l0_blonde, + "levels": levels, "events": [], + } + return src, log_dir, manifest + + +class VerifyChainBase(unittest.TestCase): + def setUp(self): + self._orig_hsi = reclaim.hsi_exists + reclaim.hsi_exists = lambda tar: True # all tars present by default + + def tearDown(self): + reclaim.hsi_exists = self._orig_hsi + + def vc(self, manifest, log_dir, holds=()): + return reclaim.verify_chain(manifest, holds, log_dir) + + +class TestIsLeveledManifest(unittest.TestCase): + def test_true_for_schema_discriminator(self): + self.assertTrue(reclaim.is_leveled_manifest( + {"schema": "leveled_v1", "levels": []})) + + def test_true_for_bare_levels_key(self): + # No "schema" field (e.g. an older/edge-case manifest) — "levels" alone + # is still a leveled manifest. + self.assertTrue(reclaim.is_leveled_manifest({"levels": []})) + + def test_false_for_routing_manifest(self): + self.assertFalse(reclaim.is_leveled_manifest({"objects": []})) + + def test_false_for_empty_dict(self): + self.assertFalse(reclaim.is_leveled_manifest({})) + + +class TestVerifyChainVerdicts(VerifyChainBase): + def test_safe_baseline_only(self): + with tempfile.TemporaryDirectory() as d: + _, log_dir, m = build_leveled_target(d) + r = self.vc(m, log_dir) + self.assertEqual(r["verdict"], reclaim.SAFE) + self.assertEqual(r["live"], 3) + self.assertEqual(r["archived"], 3) + self.assertEqual(len(r["all_tars"]), 1) + + def test_safe_with_delta_chain(self): + # Baseline + a genuine delta that changed b.bin: effective_catalog must fold + # L0 UNION the delta so the reconstructed set matches the now-modified live tree. + with tempfile.TemporaryDirectory() as d: + _, log_dir, m = build_leveled_target(d, with_delta=True) + r = self.vc(m, log_dir) + self.assertEqual(r["verdict"], reclaim.SAFE) + self.assertEqual(r["live"], 3) + self.assertEqual(r["archived"], 3) + self.assertEqual(len(r["all_tars"]), 2) + + def test_count_drift_extra_file(self): + with tempfile.TemporaryDirectory() as d: + src, log_dir, m = build_leveled_target(d) + _write(os.path.join(src, "sub", "new.bin"), b"new") # not in any level + r = self.vc(m, log_dir) + self.assertEqual(r["verdict"], reclaim.DRIFTED) + self.assertTrue(any("live files" in x for x in r["reasons"])) + + def test_bytes_drift_same_count_and_mtime(self): + # Simulate an in-place rewrite the cheap diff can't see directly: the recorded + # L0 catalog's size for one arcname is wrong relative to the (unchanged) live + # file — same trick test_routing_reclaim.py uses (mutate the recorded number, + # not the live file), isolating the byte-sum comparison from the mtime one. + with tempfile.TemporaryDirectory() as d: + _, log_dir, m = build_leveled_target(d) + ref = m["levels"][0]["catalog_ref"] + path = os.path.join(log_dir, ref) + with open(path) as fh: + raw = json.load(fh) + raw["a.bin"][1] += 100 # size off; mtime untouched + with open(path, "w") as fh: + json.dump(raw, fh) + r = self.vc(m, log_dir) + self.assertEqual(r["verdict"], reclaim.DRIFTED) + self.assertTrue(any("live bytes" in x for x in r["reasons"])) + + def test_newer_drift(self): + # The recorded L0 catalog's mtime for one arcname predates the live file's + # actual mtime -> that file was modified after being archived. + with tempfile.TemporaryDirectory() as d: + _, log_dir, m = build_leveled_target(d) + ref = m["levels"][0]["catalog_ref"] + path = os.path.join(log_dir, ref) + with open(path) as fh: + raw = json.load(fh) + raw["a.bin"][0] = int((time.time() - 3600) * 1e9) # 1h before actual mtime + with open(path, "w") as fh: + json.dump(raw, fh) + r = self.vc(m, log_dir) + self.assertEqual(r["verdict"], reclaim.DRIFTED) + self.assertTrue(any("modified since being archived" in x for x in r["reasons"])) + + def test_missing_intermediate_tar_is_broken_chain(self): + with tempfile.TemporaryDirectory() as d: + _, log_dir, m = build_leveled_target(d, with_delta=True) + l0_tar = m["levels"][0]["fortress_tar"] + reclaim.hsi_exists = lambda tar: tar != l0_tar # L0 tar missing on tape + r = self.vc(m, log_dir) + self.assertEqual(r["verdict"], reclaim.FAILED) + self.assertTrue(any("chain is broken" in x for x in r["reasons"])) + + def test_no_levels_failed(self): + with tempfile.TemporaryDirectory() as d: + _, log_dir, m = build_leveled_target(d) + m["levels"] = [] + r = self.vc(m, log_dir) + self.assertEqual(r["verdict"], reclaim.FAILED) + + def test_gone(self): + with tempfile.TemporaryDirectory() as d: + _, log_dir, m = build_leveled_target(d) + m["source_folder"] = os.path.join(d, "does_not_exist") + r = self.vc(m, log_dir) + self.assertEqual(r["verdict"], reclaim.GONE) + + def test_hold(self): + with tempfile.TemporaryDirectory() as d: + _, log_dir, m = build_leveled_target(d) + r = self.vc(m, log_dir, holds=["proj"]) + self.assertEqual(r["verdict"], reclaim.HOLD) + + def test_canonical(self): + with tempfile.TemporaryDirectory() as d: + src, log_dir, m = build_leveled_target(d) + r = reclaim.verify_chain(m, [], log_dir, canonical_roots=[src]) + self.assertEqual(r["verdict"], reclaim.CANONICAL) + + def test_superseded_chain_tar_missing_does_not_fail_current_chain(self): + # After a `--full` rebaseline, a SECOND, independent L0 is appended and + # current_l0_blonde repointed to it (archive.test_force_full_reships_ + # baseline_even_with_existing_l0 proves the engine does exactly this); + # the OLD L0 (now superseded) is never removed from manifest["levels"]. + # verify_chain's completeness gate must only require the CURRENT + # chain's tar(s) to be on tape -- the superseded L0's tar going missing + # (e.g. once P3-4 tape pruning exists) must never FAIL a target whose + # actual restorable (current) chain is intact. + with tempfile.TemporaryDirectory() as d: + src, log_dir, m = build_leveled_target(d, badge="rebase", project="rebproj") + old_l0 = m["levels"][0] + old_l0_tar = old_l0["fortress_tar"] + + # A new, independent L0 (a force_full rebaseline) with the SAME + # live-matching catalog (so the aggregate diff still reads SAFE), + # a later blonde, and current_l0_blonde repointed at it. + new_catalog = archive.enumerate_source_catalog(src, ".*") + new_l0_ts = "rebproj__L0_20200601_000000" + new_ref, new_sha = archive.write_level_catalog(log_dir, new_l0_ts, new_catalog) + new_blonde = archive.make_blonde("rebase", DELTA_EPOCH, "0") + new_l0 = { + "blonde": new_blonde, "level": 0, "parent_blonde": None, + "run_timestamp": new_l0_ts, "transfer": "zip", + "fortress_tar": "/tape/rebproj_newL0.tar", + "snar_fortress_path": None, "zip_checksum": "newl0md5", "tar_bytes": 1, + "n_files": len(new_catalog), + "source_bytes": sum(v[1] for v in new_catalog.values()), + "catalog_ref": new_ref, "catalog_sha256": new_sha, "deleted": [], + } + m["levels"].append(new_l0) + m["current_l0_blonde"] = new_blonde + + # The OLD chain's tar is gone (simulating a future tape-pruning + # release); the NEW (current) chain's tar is present. + reclaim.hsi_exists = lambda tar: tar != old_l0_tar + + r = self.vc(m, log_dir) + self.assertEqual(r["verdict"], reclaim.SAFE) + self.assertEqual(r["all_tars"], ["/tape/rebproj_newL0.tar"]) + + +class TestVerifyChainExclusion(VerifyChainBase): + def test_inv_e_no_longer_covered_drifts(self): + with tempfile.TemporaryDirectory() as d: + _, log_dir, m = build_leveled_target(d) + orig = reclaim.recheck_exclusions + reclaim.recheck_exclusions = lambda mani, sf: { + "applies": True, "still_covered": set(), + "no_longer_covered": [("meta/x.json", "DIRTY")], "reason": None} + try: + r = self.vc(m, log_dir) + finally: + reclaim.recheck_exclusions = orig + self.assertEqual(r["verdict"], reclaim.DRIFTED) + self.assertTrue(any("no longer git-COVERED" in x for x in r["reasons"])) + + def test_inv_e_still_covered_subtracted(self): + with tempfile.TemporaryDirectory() as d: + src, log_dir, m = build_leveled_target(d) + # An excluded-but-still-covered file lives on disk but is NOT in any level; + # it must be subtracted from the live walk so live still matches archived. + _write(os.path.join(src, "meta", "cov.json"), b"{}") + orig = reclaim.recheck_exclusions + reclaim.recheck_exclusions = lambda mani, sf: { + "applies": True, "still_covered": {"meta/cov.json"}, + "no_longer_covered": [], "reason": None} + try: + r = self.vc(m, log_dir) + finally: + reclaim.recheck_exclusions = orig + self.assertEqual(r["verdict"], reclaim.SAFE) # cov.json subtracted -> 3==3 + + +class TestManifestForProject(unittest.TestCase): + def test_prefers_leveled_when_both_schemas_present(self): + with tempfile.TemporaryDirectory() as d: + vault = os.path.join(d, "_vault") + os.makedirs(vault) + # Phase-2 routing manifest for this badge/project (kind="routing" filename). + with open(os.path.join(vault, "aaa.manifest.json"), "w") as fh: + json.dump({"badge": "aaa", "project": "proj", "objects": []}, fh) + # Phase-3 leveled manifest for the SAME project, same badge (kind="leveled" + # filename) — the mid-conversion collision state manifest_path's kind= param + # exists to make possible. + with open(os.path.join(vault, "aaa.leveled.manifest.json"), "w") as fh: + json.dump({"badge": "aaa", "project": "proj", "schema": "leveled_v1", + "levels": []}, fh) + got = reclaim.manifest_for_project(d, "proj") + self.assertTrue(reclaim.is_leveled_manifest(got)) + self.assertEqual(got["badge"], "aaa") + + def test_routing_only_still_returned(self): + with tempfile.TemporaryDirectory() as d: + vault = os.path.join(d, "_vault") + os.makedirs(vault) + with open(os.path.join(vault, "bbb.manifest.json"), "w") as fh: + json.dump({"badge": "bbb", "project": "routed_proj", "objects": []}, fh) + got = reclaim.manifest_for_project(d, "routed_proj") + self.assertFalse(reclaim.is_leveled_manifest(got)) + + def test_leveled_only_still_returned(self): + with tempfile.TemporaryDirectory() as d: + vault = os.path.join(d, "_vault") + os.makedirs(vault) + with open(os.path.join(vault, "ccc.leveled.manifest.json"), "w") as fh: + json.dump({"badge": "ccc", "project": "leveled_proj", + "schema": "leveled_v1", "levels": []}, fh) + got = reclaim.manifest_for_project(d, "leveled_proj") + self.assertTrue(reclaim.is_leveled_manifest(got)) + + def test_none_when_no_match(self): + with tempfile.TemporaryDirectory() as d: + self.assertIsNone(reclaim.manifest_for_project(d, "nope")) + + +class TestScanLoopLeveledWiring(unittest.TestCase): + """ + Integration test for the --scan loop wiring itself (main()): a leveled target's + per-level run-log (tagged leveled_object=True, mirroring ship_object's + routed_object=True) must be SKIPPED by the per-log verify() path — feeding it there + would false-DRIFT (its member set is a single tiny delta against the WHOLE live + tree) — and the target must instead get exactly one aggregate verdict from + verify_chain() via its manifest. A legacy (non-routed, non-leveled) project's normal + log must still flow through verify() unaffected. + """ + def setUp(self): + self._orig_hsi = reclaim.hsi_exists + reclaim.hsi_exists = lambda tar: True + self._orig_argv = sys.argv + + def tearDown(self): + reclaim.hsi_exists = self._orig_hsi + sys.argv = self._orig_argv + + def test_leveled_object_log_skipped_and_routed_to_verify_chain(self): + with tempfile.TemporaryDirectory() as d: + log_dir = os.path.join(d, "logs") + os.makedirs(log_dir) + leveled_src, leveled_log_dir, m = build_leveled_target(d, badge="lvl1", + project="leveled_proj") + # build_leveled_target put its own log_dir under d/logs already; reuse it as + # THE scan log_dir so the manifest's catalog_ref paths resolve correctly. + self.assertEqual(leveled_log_dir, log_dir) + vault = os.path.join(log_dir, "_vault") + os.makedirs(vault, exist_ok=True) + with open(os.path.join(vault, "lvl1.leveled.manifest.json"), "w") as fh: + json.dump(m, fh) + + # A per-level run-log for the L0 ship — status success but only 3 files + # (the WHOLE tiny fixture here, but in production this would be a delta's + # handful of files against a much larger live tree). Tagged leveled_object. + level_log = { + "project": "leveled_proj", "status": "success", + "fortress_tar": m["levels"][0]["fortress_tar"], + "source_folder": leveled_src, "file_pattern": ".*", + "source_files": {"a.bin": "x", "b.bin": "y", "sub/c.bin": "z"}, + "source_bytes": 350, "run_timestamp": m["levels"][0]["run_timestamp"], + "leveled_object": True, "badge": "lvl1", "level": 0, + } + with open(os.path.join(log_dir, "leveled_proj__L0_20200101_000000.json"), + "w") as fh: + json.dump(level_log, fh) + + # A normal legacy whole-target project, untouched by Phase 2/3. + legacy_src = os.path.join(d, "legacy_src") + _write(os.path.join(legacy_src, "only.bin"), b"X" * 10) + legacy_log = { + "project": "legacy_proj", "status": "success", + "fortress_tar": "/tape/legacy.tar", "source_folder": legacy_src, + "file_pattern": ".*", "source_files": {"only.bin": "x"}, + "source_bytes": 10, "run_timestamp": "legacy_proj_20990101_000000", + } + with open(os.path.join(log_dir, "legacy_proj_20990101_000000.json"), + "w") as fh: + json.dump(legacy_log, fh) + + sys.argv = ["reclaim.py", "--scan", "--log-dir", log_dir] + out = io.StringIO() + with contextlib.redirect_stdout(out): + reclaim.main() + report = out.getvalue() + + # The leveled target appears exactly once (via verify_chain -> SAFE), + # never as a second, separately-false-drifted per-log verify() entry. + self.assertEqual(report.count("leveled_proj"), 1) + self.assertIn("[SAFE ] leveled_proj", report) + # The legacy per-log path still works unaffected. + self.assertIn("[SAFE ] legacy_proj", report) + + +class TestConfigModeLeveledWiring(unittest.TestCase): + """ + Integration test for `python reclaim.py config.json` (the non-`--scan`, + single-config invocation) against a leveled (`incremental: true`) target. + Before this fix, this path had no `incremental` branch and fell through to + `latest_log_for_project()` + legacy per-log `verify()`, which always picks up + the newest per-level/NO-OP log (a tiny delta member set) and perpetually + false-DRIFTs/FAILs. It must instead route to verify_chain() via the target's + leveled manifest, exactly like --scan does. + """ + def setUp(self): + self._orig_hsi = reclaim.hsi_exists + reclaim.hsi_exists = lambda tar: True + self._orig_argv = sys.argv + + def tearDown(self): + reclaim.hsi_exists = self._orig_hsi + sys.argv = self._orig_argv + + def test_config_mode_routes_leveled_target_to_verify_chain_safe(self): + with tempfile.TemporaryDirectory() as d: + log_dir = os.path.join(d, "logs") + os.makedirs(log_dir) + project = "leveled_proj_cfg" + source_folder = os.path.join(d, "src") + file_pattern = ".*" + badge = archive.badge_of(project, source_folder, file_pattern) + + src, leveled_log_dir, m = build_leveled_target( + d, badge=badge, project=project) + self.assertEqual(leveled_log_dir, log_dir) + self.assertEqual(src, source_folder) + vault = os.path.join(log_dir, "_vault") + os.makedirs(vault, exist_ok=True) + with open(os.path.join(vault, "%s.leveled.manifest.json" % badge), + "w") as fh: + json.dump(m, fh) + + # A stale per-level run-log also sits in log_dir (as production would + # leave one behind) -- if the config-mode path fell through to + # latest_log_for_project + verify(), THIS is what it would wrongly + # grade instead of the manifest aggregate. + level_log = { + "project": project, "status": "success", + "fortress_tar": m["levels"][0]["fortress_tar"], + "source_folder": source_folder, "file_pattern": file_pattern, + "source_files": {"a.bin": "x"}, "source_bytes": 100, + "run_timestamp": m["levels"][0]["run_timestamp"], + "leveled_object": True, "badge": badge, "level": 0, + } + with open(os.path.join(log_dir, "%s__L0_20200101_000000.json" % project), + "w") as fh: + json.dump(level_log, fh) + + config_path = os.path.join(d, "config.json") + with open(config_path, "w") as fh: + json.dump({ + "project_name": project, "source_folder": source_folder, + "file_pattern": file_pattern, "fortress_base_dir": "/tape", + "emails": ["x@example.com"], "log_dir": log_dir, + "incremental": True, + }, fh) + + sys.argv = ["reclaim.py", config_path, "--log-dir", log_dir] + out = io.StringIO() + with contextlib.redirect_stdout(out): + reclaim.main() + report = out.getvalue() + + self.assertEqual(report.count(project), 1) + self.assertIn("[SAFE ] %s" % project, report) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_leveled_restore.py b/tests/test_leveled_restore.py new file mode 100644 index 0000000..7bb8ff1 --- /dev/null +++ b/tests/test_leveled_restore.py @@ -0,0 +1,422 @@ +""" +Unit tests for Phase-3 leveled-incremental restore (docs/RFC_incremental_v2.md §3.4, +P3-3): restore.restore_leveled_target() replays a leveled manifest's L0+delta chain as +a strict run-time-ordered file OVERLAY (Path A — each level's zip extracted on top of +the accumulating tree) plus per-level deletion replay, gated by chain completeness +(BEFORE any transfer), an empty-dest guard, per-member MD5 verification against each +level's own run-log, and a mandatory final reconciliation against +archive.effective_catalog's fold. Tape is mocked (reclaim.hsi_exists + an injected +extract fn); no hsi/htar/Slurm needed. + +Reuses tests/test_restore.py's fake_extract-closure and reclaim.hsi_exists monkeypatch +conventions, and tests/test_leveled_reclaim.py's leveled-manifest fixture-building style +(archive.write_level_catalog / archive.make_blonde) so fixtures match production shape +exactly. +""" +import calendar +import contextlib +import hashlib +import io +import json +import os +import sys +import tempfile +import unittest + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +import archive # noqa: E402 +import reclaim # noqa: E402 +import restore # noqa: E402 + + +def _epoch(y, m, d, hh=0, mm=0, ss=0): + return calendar.timegm((y, m, d, hh, mm, ss, 0, 0, 0)) + + +# Deliberately far apart (months, not days) so a --to-timestamp cutoff chosen between +# two of them is robust to any local timezone offset (+/-14h) the test runs under — +# _parse_to_timestamp interprets the cutoff string as NAIVE LOCAL TIME. +L0_EPOCH = _epoch(2020, 1, 1) +L1_EPOCH = _epoch(2020, 2, 1) +L2_EPOCH = _epoch(2020, 8, 1) + +L0_CONTENT = {"a.bin": b"A0" * 4, "b.bin": b"B0" * 4, "c.bin": b"C0" * 4} +L1_CONTENT = {"b.bin": b"B1" * 6, "d.bin": b"D1" * 3} # b modified, d added +L1_DELETED = [] +L2_CONTENT = {"c.bin": b"C2" * 5} # c modified +L2_DELETED = ["a.bin"] # ... and a deleted + +FULL_CHAIN_EXPECTED = { + "b.bin": L1_CONTENT["b.bin"], + "c.bin": L2_CONTENT["c.bin"], + "d.bin": L1_CONTENT["d.bin"], +} # a.bin removed by L2's deletion + +TO_TIMESTAMP_CUTOFF = "20200401_000000" # April 1 — between L1 (Feb) and L2 (Aug) +TO_TIMESTAMP_EXPECTED = { + "a.bin": L0_CONTENT["a.bin"], # L2's deletion not yet applied + "b.bin": L1_CONTENT["b.bin"], + "c.bin": L0_CONTENT["c.bin"], # L2's modification not yet applied + "d.bin": L1_CONTENT["d.bin"], +} + + +def _mk_node(log_dir, run_ts, badge, level, parent_blonde, blonde, content, deleted, + project): + """ + One manifest level node + its sibling catalog_ref + run-log, mirroring + archive.level_and_ship's shape exactly. `content` = {} (falsy) builds a + DELETIONS-ONLY node: no tar, no catalog_ref, no run-log — the case restore must + never call extract() for. + """ + if content: + catalog_subset = {a: (1_600_000_000_000_000_000 + level, len(b)) + for a, b in content.items()} + catalog_ref, catalog_sha256 = archive.write_level_catalog( + log_dir, run_ts, catalog_subset) + fortress_tar = "/tape/%s.tar" % run_ts + checks = {a: hashlib.md5(b).hexdigest() for a, b in content.items()} + log = { + "project": project, "run_timestamp": run_ts, "status": "success", + "leveled_object": True, "badge": badge, "level": level, + "fortress_tar": fortress_tar, "source_files": checks, + "n_files": len(content), + "source_bytes": sum(len(b) for b in content.values()), + } + with open(os.path.join(log_dir, run_ts + ".json"), "w") as fh: + json.dump(log, fh) + n_files = len(content) + source_bytes = sum(len(b) for b in content.values()) + zip_checksum = "fakemd5-%s" % run_ts + tar_bytes = 1000 + level + transfer = "zip" + else: + catalog_ref = catalog_sha256 = fortress_tar = zip_checksum = tar_bytes = None + n_files = 0 + source_bytes = 0 + transfer = None + + return { + "blonde": blonde, "level": level, "parent_blonde": parent_blonde, + "run_timestamp": run_ts, "transfer": transfer, "fortress_tar": fortress_tar, + "snar_fortress_path": None, "zip_checksum": zip_checksum, "tar_bytes": tar_bytes, + "n_files": n_files, "source_bytes": source_bytes, "catalog_ref": catalog_ref, + "catalog_sha256": catalog_sha256, "deleted": list(deleted or []), + } + + +def build_chain(d, badge="lvlbadge", project="lvlproj"): + """ + L0 (a,b,c) + L1 (adds d, modifies b) + L2 (modifies c, DELETES a) — a genuine + 3-level chain with a real deletion mixed into a delta. Returns (manifest, log_dir, + content_by_run_timestamp); the latter feeds fake_extract_level (the in-memory + 'tape'). + """ + log_dir = os.path.join(d, "logs") + os.makedirs(log_dir, exist_ok=True) + + l0_blonde = archive.make_blonde(badge, L0_EPOCH, "0") + l1_blonde = archive.make_blonde(badge, L1_EPOCH, "1") + l2_blonde = archive.make_blonde(badge, L2_EPOCH, "2") + + l0_ts = "%s__L0_20200101_000000" % project + l1_ts = "%s__L1_20200201_000000" % project + l2_ts = "%s__L2_20200801_000000" % project + + node0 = _mk_node(log_dir, l0_ts, badge, 0, None, l0_blonde, L0_CONTENT, [], project) + node1 = _mk_node(log_dir, l1_ts, badge, 1, l0_blonde, l1_blonde, L1_CONTENT, + L1_DELETED, project) + node2 = _mk_node(log_dir, l2_ts, badge, 2, l0_blonde, l2_blonde, L2_CONTENT, + L2_DELETED, project) + + manifest = { + "schema_version": 1, "schema": "leveled_v1", "badge": badge, "project": project, + "source_folder": "/depot/wherever/src", "file_pattern": ".*", + "exclusion": {"status": "OFF"}, "current_l0_blonde": l0_blonde, + "levels": [node0, node1, node2], "events": [], + } + content_by_run_ts = {l0_ts: L0_CONTENT, l1_ts: L1_CONTENT, l2_ts: L2_CONTENT} + return manifest, log_dir, content_by_run_ts + + +def fake_extract_level(content_by_run_ts=None, calls=None, override=None): + """ + An extract fn that materializes ONE level's members from an in-memory 'tape' keyed + by run_timestamp — mirrors tests/test_restore.py's fake_extract closure. `calls` + collects every run_timestamp actually passed to this function (used to assert a + deletions-only node's fortress_tar=None NEVER reaches extract — the "never call + hsi_get(None)" guard is real, not hopeful). `override` = {run_timestamp: + {arcname: bytes}} injects corrupted/extra/missing content for one level (bitrot / + reconciliation-failure fixtures) without mutating the shared content map. + """ + content_by_run_ts = content_by_run_ts or {} + override = override or {} + + def _extract(node, dest, tmp_dir): + rts = node["run_timestamp"] + if calls is not None: + calls.append(rts) + payload = override.get(rts, content_by_run_ts.get(rts, {})) + for arc, body in payload.items(): + p = os.path.join(dest, arc) + os.makedirs(os.path.dirname(p), exist_ok=True) + with open(p, "wb") as fh: + fh.write(body) + return _extract + + +class RestoreLeveledBase(unittest.TestCase): + def setUp(self): + self._orig_hsi = reclaim.hsi_exists + reclaim.hsi_exists = lambda tar: True # all tars present by default + + def tearDown(self): + reclaim.hsi_exists = self._orig_hsi + + def run_restore(self, d, manifest, log_dir, content_by_run_ts, calls=None, + override=None, **kw): + kw.setdefault("extract", + fake_extract_level(content_by_run_ts, calls=calls, override=override)) + return restore.restore_leveled_target( + manifest, os.path.join(d, "dest"), log_dir, os.path.join(d, "tmp"), **kw) + + +class TestFullChainRestore(RestoreLeveledBase): + def test_full_chain_reconstructs_exact_tree(self): + with tempfile.TemporaryDirectory() as d: + manifest, log_dir, content = build_chain(d) + r = self.run_restore(d, manifest, log_dir, content) + dest = os.path.join(d, "dest") + self.assertEqual(restore.walk_restored(dest), set(FULL_CHAIN_EXPECTED)) + for arc, body in FULL_CHAIN_EXPECTED.items(): # bytes, not just names + with open(os.path.join(dest, arc), "rb") as fh: + self.assertEqual(fh.read(), body) + self.assertFalse(os.path.exists(os.path.join(dest, "a.bin"))) # deleted by L2 + self.assertTrue(r["reconciled"]) + self.assertEqual(r["levels"], 3) + self.assertEqual(r["files"], len(FULL_CHAIN_EXPECTED)) + + +class TestCompletenessGate(RestoreLeveledBase): + def test_missing_intermediate_tar_aborts_before_any_extraction(self): + with tempfile.TemporaryDirectory() as d: + manifest, log_dir, content = build_chain(d) + l1_tar = manifest["levels"][1]["fortress_tar"] + reclaim.hsi_exists = lambda tar: tar != l1_tar # L1's tar is GONE + calls = [] + with self.assertRaisesRegex(restore.RestoreError, "chain is broken"): + self.run_restore(d, manifest, log_dir, content, calls=calls) + self.assertEqual(calls, []) # nothing extracted + self.assertFalse(os.path.exists(os.path.join(d, "dest"))) # dest never created + + def test_no_levels_aborts(self): + with tempfile.TemporaryDirectory() as d: + manifest, log_dir, content = build_chain(d) + manifest["levels"] = [] + with self.assertRaisesRegex(restore.RestoreError, "no levels"): + self.run_restore(d, manifest, log_dir, content) + + +class TestDeletionsOnlyNode(RestoreLeveledBase): + def test_deletions_only_node_zero_extractor_calls(self): + with tempfile.TemporaryDirectory() as d: + badge, project = "delbadge", "delproj" + log_dir = os.path.join(d, "logs") + os.makedirs(log_dir) + l0_blonde = archive.make_blonde(badge, L0_EPOCH, "0") + l1_blonde = archive.make_blonde(badge, L1_EPOCH, "1") + l0_ts = "%s__L0_20200101_000000" % project + l1_ts = "%s__L1_20200201_000000" % project + + node0 = _mk_node(log_dir, l0_ts, badge, 0, None, l0_blonde, L0_CONTENT, + [], project) + # A pure deletions-only delta: no members, so content={} -> fortress_tar + # is None. restore_leveled_target's loop must apply "deleted" WITHOUT + # ever calling extract() for this node. + node1 = _mk_node(log_dir, l1_ts, badge, 1, l0_blonde, l1_blonde, {}, + ["b.bin"], project) + self.assertIsNone(node1["fortress_tar"]) + + manifest = { + "schema_version": 1, "schema": "leveled_v1", "badge": badge, + "project": project, "source_folder": "/depot/x", "file_pattern": ".*", + "exclusion": {"status": "OFF"}, "current_l0_blonde": l0_blonde, + "levels": [node0, node1], "events": [], + } + content = {l0_ts: L0_CONTENT} # deliberately no entry for l1_ts + + calls = [] + r = self.run_restore(d, manifest, log_dir, content, calls=calls) + dest = os.path.join(d, "dest") + self.assertEqual(calls, [l0_ts]) # L1 (deletions-only) never reached extract() + self.assertEqual(restore.walk_restored(dest), {"a.bin", "c.bin"}) # b.bin removed + self.assertTrue(r["reconciled"]) + self.assertEqual(r["levels"], 2) + + +class TestMD5Mismatch(RestoreLeveledBase): + def test_md5_mismatch_aborts(self): + with tempfile.TemporaryDirectory() as d: + manifest, log_dir, content = build_chain(d) + l1_ts = manifest["levels"][1]["run_timestamp"] + bitrot = {l1_ts: {"b.bin": b"BITROT", "d.bin": L1_CONTENT["d.bin"]}} + with self.assertRaisesRegex(restore.RestoreError, "MD5 MISMATCH"): + self.run_restore(d, manifest, log_dir, content, override=bitrot) + + def test_missing_catalog_ref_raises_restore_error_not_raw_exception(self): + # A level's sibling catalog_ref file lost from log_dir must surface as + # a named, actionable RestoreError (matching every other gate's abort + # convention), not a raw FileNotFoundError traceback. + with tempfile.TemporaryDirectory() as d: + manifest, log_dir, content = build_chain(d) + ref = manifest["levels"][0]["catalog_ref"] + os.remove(os.path.join(log_dir, ref)) + with self.assertRaisesRegex(restore.RestoreError, "cannot read catalog_ref"): + self.run_restore(d, manifest, log_dir, content) + + +class TestFinalReconciliation(RestoreLeveledBase): + def test_reconciliation_mismatch_aborts(self): + with tempfile.TemporaryDirectory() as d: + manifest, log_dir, content = build_chain(d) + l2_ts = manifest["levels"][2]["run_timestamp"] + # A stray file the manifest never recorded for ANY level — per-level + # verify_level_members only checks the members ITS OWN catalog_ref + # names, so this slips past level-by-level verification and must + # instead be caught by the MANDATORY final reconciliation. + stray = dict(L2_CONTENT) + stray["stray/ghost.bin"] = b"not in any manifest" + with self.assertRaisesRegex(restore.RestoreError, "reconciliation FAILED"): + self.run_restore(d, manifest, log_dir, content, override={l2_ts: stray}) + + +class TestToTimestamp(RestoreLeveledBase): + def test_to_timestamp_stops_chain_at_right_point(self): + with tempfile.TemporaryDirectory() as d: + manifest, log_dir, content = build_chain(d) + l0_ts = manifest["levels"][0]["run_timestamp"] + l1_ts = manifest["levels"][1]["run_timestamp"] + l2_ts = manifest["levels"][2]["run_timestamp"] + + calls = [] + r = self.run_restore(d, manifest, log_dir, content, calls=calls, + to_timestamp=TO_TIMESTAMP_CUTOFF) + dest = os.path.join(d, "dest") + + self.assertEqual(set(calls), {l0_ts, l1_ts}) + self.assertNotIn(l2_ts, calls) # L2 excluded by the cutoff + self.assertEqual(restore.walk_restored(dest), set(TO_TIMESTAMP_EXPECTED)) + for arc, body in TO_TIMESTAMP_EXPECTED.items(): + with open(os.path.join(dest, arc), "rb") as fh: + self.assertEqual(fh.read(), body) + self.assertEqual(r["levels"], 2) + self.assertTrue(r["reconciled"]) + + +class TestDestGuard(RestoreLeveledBase): + def test_nonempty_dest_refused(self): + with tempfile.TemporaryDirectory() as d: + manifest, log_dir, content = build_chain(d) + dest = os.path.join(d, "dest") + os.makedirs(dest) + with open(os.path.join(dest, "live.txt"), "w") as fh: + fh.write("precious") + calls = [] + with self.assertRaisesRegex(restore.RestoreError, "NOT empty"): + self.run_restore(d, manifest, log_dir, content, calls=calls) + self.assertEqual(calls, []) # nothing extracted + with open(os.path.join(dest, "live.txt")) as fh: # untouched + self.assertEqual(fh.read(), "precious") + + +class TestCLIConfigModeExactBadgeResolution(unittest.TestCase): + """ + restore.main()'s config path must resolve the manifest by the config's OWN + exact badge (project_name + source_folder + file_pattern), never by a + project-name substring match over every manifest in the vault + (reclaim.manifest_for_project's role, which two DIFFERENT badges can + legitimately share a project name against). Two leveled targets sharing a + project_name but differing in source_folder (distinct badges) live in the + same vault; a config for the SECOND one must resolve to the SECOND one's + manifest, never silently fall back to the first (which would restore -- or + here, report chain-completeness against -- the WRONG target's data). + """ + def setUp(self): + self._orig_hsi = reclaim.hsi_exists + # Both targets' tars report "missing" so check_chain_complete raises + # RestoreError immediately (before any real extraction/tape access) -- + # exactly the signal needed to prove which manifest was picked, without + # needing to fake htar/hsi. + reclaim.hsi_exists = lambda tar: False + + def tearDown(self): + reclaim.hsi_exists = self._orig_hsi + + def test_config_resolves_own_badge_not_other_project_namesake(self): + with tempfile.TemporaryDirectory() as d: + log_dir = os.path.join(d, "logs") + vault = os.path.join(log_dir, "_vault") + os.makedirs(vault) + + project = "shared_name" + # Fixed (not tempdir-derived) so badge_of's hash -- and therefore + # find_manifests' sorted-glob order -- is deterministic across runs: + # badge_of("shared_name", src_a, ".*") sorts alphabetically BEFORE + # badge_of("shared_name", src_b, ".*"), so the old project-name-only + # lookup (manifest_for_project, sorted glob, first leveled match) + # would deterministically pick manifest_a even when the config asks + # for src_b -- exactly the bug this test guards against. + src_a = "/depot/proj/src_a" + src_b = "/depot/proj/src_b" + pattern = ".*" + badge_a = archive.badge_of(project, src_a, pattern) + badge_b = archive.badge_of(project, src_b, pattern) + self.assertNotEqual(badge_a, badge_b) + + def _one_level_manifest(badge, tar): + blonde = archive.make_blonde(badge, L0_EPOCH, "0") + node = {"blonde": blonde, "level": 0, "parent_blonde": None, + "run_timestamp": "%s__L0_20200101_000000" % project, + "transfer": "zip", "fortress_tar": tar, + "snar_fortress_path": None, "zip_checksum": "md5", + "tar_bytes": 1, "n_files": 1, "source_bytes": 1, + "catalog_ref": None, "catalog_sha256": None, "deleted": []} + return {"schema_version": 1, "schema": "leveled_v1", "badge": badge, + "project": project, "source_folder": src_a if badge == badge_a else src_b, + "file_pattern": pattern, "exclusion": {"status": "OFF"}, + "current_l0_blonde": blonde, "levels": [node], "events": []} + + manifest_a = _one_level_manifest(badge_a, "/tape/a_L0.tar") + manifest_b = _one_level_manifest(badge_b, "/tape/b_L0.tar") + with open(os.path.join(vault, "%s.leveled.manifest.json" % badge_a), "w") as fh: + json.dump(manifest_a, fh) + with open(os.path.join(vault, "%s.leveled.manifest.json" % badge_b), "w") as fh: + json.dump(manifest_b, fh) + + config_path = os.path.join(d, "config_b.json") + with open(config_path, "w") as fh: + json.dump({ + "project_name": project, "source_folder": src_b, + "file_pattern": pattern, "fortress_base_dir": "/tape", + "emails": ["x@example.com"], "log_dir": log_dir, + "incremental": True, + }, fh) + + dest = os.path.join(d, "dest") + out = io.StringIO() + with contextlib.redirect_stdout(out): + with self.assertRaises(SystemExit): + restore.main([config_path, "--dest", dest]) + report = out.getvalue() + + # Must print/resolve badge_b's manifest (the config's own badge), + # never badge_a's -- the printed badge line and the manifest path + # itself must name badge_b throughout, and badge_a must not appear + # at all (proving manifest_b, not manifest_a, was loaded). + self.assertIn(f"badge: {badge_b}", report) + self.assertIn(f"{badge_b}.leveled.manifest.json", report) + self.assertNotIn(badge_a, report) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_resume_fallback.py b/tests/test_resume_fallback.py index 1eb28b3..db1cd4c 100644 --- a/tests/test_resume_fallback.py +++ b/tests/test_resume_fallback.py @@ -84,5 +84,36 @@ def test_corrupt_zip_returns_none_and_is_removed(self): self.assertFalse(os.path.exists(bad)) +class TestL0ZipIsStale(unittest.TestCase): + """ + l0_zip_is_stale() is the pure decision core of the Phase-3 leveled- + incremental L0 resume guard (RFC §3.5.1): the L0 zip is resumed (find_ + existing_zip -> try_reconstruct, wired only for level==0 in ship_level, a + delta never attempts this) only when it still matches the live survivor + set, using the same failure-count/newer-than-zip-timestamp semantics as + the whole-target guard. + """ + ZIP_20260101 = "/tmp/T__L0_20260101_000000.zip" + + def test_matching_count_and_older_mtime_is_not_stale(self): + older = archive.floor_s(int(1_700_000_000 * 1e9)) + self.assertFalse(archive.l0_zip_is_stale(3, 3, older, self.ZIP_20260101)) + + def test_count_mismatch_is_stale(self): + self.assertTrue(archive.l0_zip_is_stale(3, 4, None, self.ZIP_20260101)) + + def test_newer_survivor_mtime_is_stale(self): + # "20260101_000000" -> well before this "newer" survivor mtime. + newer = archive.floor_s(int(1_800_000_000 * 1e9)) + self.assertTrue(archive.l0_zip_is_stale(3, 3, newer, self.ZIP_20260101)) + + def test_no_newest_survivor_mtime_and_matching_count_is_not_stale(self): + self.assertFalse(archive.l0_zip_is_stale(0, 0, None, self.ZIP_20260101)) + + def test_unparseable_zip_filename_falls_back_to_count_only(self): + self.assertFalse(archive.l0_zip_is_stale(2, 2, 9_999_999_999, "/tmp/weird.zip")) + self.assertTrue(archive.l0_zip_is_stale(2, 3, 9_999_999_999, "/tmp/weird.zip")) + + if __name__ == "__main__": unittest.main()