diff --git a/CLAUDE.md b/CLAUDE.md index ce0491a..cba54d8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -90,6 +90,7 @@ python archive.py --globus /path/to/configs/config_.json archive.py # Single-file script — all archive logic lives here archive_local.sbatch # Default path: runs archive.py (local mode) as a Slurm job reclaim.py # Inverse of archive.py: verify archives, reclaim source space +restore.py # Full/selective restore of a size-routed target (RFC §2.9) verify_exclusions.py # Post-hoc re-verifier: prove archive-time exclusions are still git-covered reclaim_holds.example.txt# Template holds file (pass a real one via --holds) config.example.json # Annotated config template @@ -272,6 +273,10 @@ Personal auth (endpoint owner) never needs `allowed_principals`. The local | `cleanup_zip_on_success` | No | `false` | Remove the staging zip after the round-trip verification passes (recurring/automated archives; see invariant #5) | | `exclude_spec` | No | `null` | PATH to a shared exclusion spec JSON (see `docs/EXCLUSION_SPEC.md`) that leaves git-versioned regenerable metadata off the backup. Classified by role-bearing path (never extension); dropped only on a positive git `COVERED` proof — any uncertainty backs the file up. Default-safe; read where the zip is built so it works under `--globus`. Omit to disable. | | `exclude_optional` | No | `[]` | Per-asset opt-out list of Tier-2 (OPTIONAL) rule ids from the spec (regenerable provenance/QC). Honored only if the asset also ships the raw inputs. Omit to back up all OPTIONAL artifacts. | +| `size_routing` | No | `false` | Phase-2 per-object routing (solos + shards, append-only ships — see the Size-routing section and `docs/RFC_incremental_v2.md` §2). **Local mode only** (refuses `--globus`); requires a SHARED, DURABLE `log_dir` (load_config refuses `$HOME`/scratch/the home default). | +| `t_small` | No | 256 MiB | Solo/shard cutoff in bytes (hard floor 100 MB). Files ≥ `t_small` ship as their own object. | +| `shard_count` | No | `null` | Shard count K. `null` = derive from `shard_target` at baseline. **Frozen in the manifest** once a baseline exists — changing it needs `--fresh`. | +| `shard_target` | No | 256 GiB | Target shard size (bytes) used to derive K at baseline. | See `config.example.json` for an annotated template. @@ -312,6 +317,46 @@ changed since (`CHANGED`), and fails on any that are no longer covered (`LOST`) so "prove nothing real was dropped" is answerable months later. Exits non-zero on any `LOST`. +## Size-routing (Phase 2) — solos, shards, append-only ships + +Opt-in per config (`"size_routing": true`) and **default-off** — an unset config +archives byte-for-byte as before. Authoritative design: `docs/RFC_incremental_v2.md` +§2 (all 5 slices merged; Negishi-validated). + +- Post-exclusion survivors are partitioned into Fortress **objects**: each file + ≥ `t_small` ships alone (a **solo**, skip-if-unchanged); smaller files bundle + into **content-addressed shards** (`hash(relpath) mod K`) so changing one file + re-ships one shard, and growth (append) ships only new objects. K is derived at + baseline and **frozen in the manifest**. +- **Skip-unchanged doubles as resume**: an object's fingerprint is recorded only + after transfer + verify, so a half-shipped object simply re-ships. The + whole-target `find_existing_zip` resume machinery does not apply to routed runs. +- **Transport per object** (`ship_object`): tars the SOURCE directly (no zip); + real `htar` (indexed `.tar.idx`, random-access restore) unless a member ≥ 64 GiB + → `htar_large`. Invariant #6 still applies to retrieval. +- **Per-target manifest** at `{log_dir}/_vault/{badge}.manifest.json` — the + durable cross-run state reclaim and restore rebuild the aggregate from. This is + why routing requires a shared, durable `log_dir` (guard in `load_config`). +- **Reclaim** verifies routed targets as AGGREGATES: `verify_target()` checks + ⋃ objects' arcnames / Σ counts / Σ bytes, every object tar present, per-object + newer-than-run, and the INV-E exclusion re-gate. `--scan` picks manifests up + automatically; routed per-object run-logs (`routed_object: true`) are skipped + by the legacy per-log `verify()`. +- **Reship-monitor**: a run re-shipping > 25% of shards emails `[RESHIP-WATCH]` + — the Phase-3 (leveled engine) trigger. Pure append trips ~0%. +- **Restore** (`restore.py`, LOCAL, operator-invoked): full or selective restore + into an **empty** dest. Gates, in order, each aborting the whole restore: + completeness (every selected object tar on tape BEFORE any transfer), empty + dest, per-object extract by recorded transport (`htar -xvf -L` random-access / + `hsi get` + `tar xf`), per-file MD5 vs each object's run-log, and a mandatory + final reconciliation (`walk(dest) == ⋃ arcnames`) on a full restore. + + ```bash + python3 restore.py config.json --dest /scratch/.../restore_here # full + python3 restore.py config.json --dest DIR --slot shard:3 # one shard + python3 restore.py config.json --dest DIR --member sub/path/file.bin # one file + ``` + --- ## Known issues / history @@ -334,4 +379,4 @@ any `LOST`. - **Pre-run confirmation prompt** — print matched file count + size, then prompt `Proceed? [y/N]` - **Overlap detection** — warn if files were already archived in a previous run -- **Vault-side manifest** — a `manifest.json` in `fortress_base_dir` listing all archives for the project +- **Vault-side manifest** — a `manifest.json` in `fortress_base_dir` listing all archives for the project. (Size-routed targets already keep a per-target manifest at `{log_dir}/_vault/`; this gap is about a Fortress-side listing for whole-target archives.)