Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 46 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ python archive.py --globus /path/to/configs/config_<name>.json
archive.py # Single-file script — all archive logic lives here
archive_local.sbatch # Default path: runs archive.py (local mode) as a Slurm job
reclaim.py # Inverse of archive.py: verify archives, reclaim source space
restore.py # Full/selective restore of a size-routed target (RFC §2.9)
verify_exclusions.py # Post-hoc re-verifier: prove archive-time exclusions are still git-covered
reclaim_holds.example.txt# Template holds file (pass a real one via --holds)
config.example.json # Annotated config template
Expand Down Expand Up @@ -272,6 +273,10 @@ Personal auth (endpoint owner) never needs `allowed_principals`. The local
| `cleanup_zip_on_success` | No | `false` | Remove the staging zip after the round-trip verification passes (recurring/automated archives; see invariant #5) |
| `exclude_spec` | No | `null` | PATH to a shared exclusion spec JSON (see `docs/EXCLUSION_SPEC.md`) that leaves git-versioned regenerable metadata off the backup. Classified by role-bearing path (never extension); dropped only on a positive git `COVERED` proof — any uncertainty backs the file up. Default-safe; read where the zip is built so it works under `--globus`. Omit to disable. |
| `exclude_optional` | No | `[]` | Per-asset opt-out list of Tier-2 (OPTIONAL) rule ids from the spec (regenerable provenance/QC). Honored only if the asset also ships the raw inputs. Omit to back up all OPTIONAL artifacts. |
| `size_routing` | No | `false` | Phase-2 per-object routing (solos + shards, append-only ships — see the Size-routing section and `docs/RFC_incremental_v2.md` §2). **Local mode only** (refuses `--globus`); requires a SHARED, DURABLE `log_dir` (load_config refuses `$HOME`/scratch/the home default). |
| `t_small` | No | 256 MiB | Solo/shard cutoff in bytes (hard floor 100 MB). Files ≥ `t_small` ship as their own object. |
| `shard_count` | No | `null` | Shard count K. `null` = derive from `shard_target` at baseline. **Frozen in the manifest** once a baseline exists — changing it needs `--fresh`. |
| `shard_target` | No | 256 GiB | Target shard size (bytes) used to derive K at baseline. |

See `config.example.json` for an annotated template.

Expand Down Expand Up @@ -312,6 +317,46 @@ changed since (`CHANGED`), and fails on any that are no longer covered (`LOST`)
so "prove nothing real was dropped" is answerable months later. Exits non-zero on
any `LOST`.

## Size-routing (Phase 2) — solos, shards, append-only ships

Opt-in per config (`"size_routing": true`) and **default-off** — an unset config
archives byte-for-byte as before. Authoritative design: `docs/RFC_incremental_v2.md`
§2 (all 5 slices merged; Negishi-validated).

- Post-exclusion survivors are partitioned into Fortress **objects**: each file
≥ `t_small` ships alone (a **solo**, skip-if-unchanged); smaller files bundle
into **content-addressed shards** (`hash(relpath) mod K`) so changing one file
re-ships one shard, and growth (append) ships only new objects. K is derived at
baseline and **frozen in the manifest**.
- **Skip-unchanged doubles as resume**: an object's fingerprint is recorded only
after transfer + verify, so a half-shipped object simply re-ships. The
whole-target `find_existing_zip` resume machinery does not apply to routed runs.
- **Transport per object** (`ship_object`): tars the SOURCE directly (no zip);
real `htar` (indexed `.tar.idx`, random-access restore) unless a member ≥ 64 GiB
→ `htar_large`. Invariant #6 still applies to retrieval.
- **Per-target manifest** at `{log_dir}/_vault/{badge}.manifest.json` — the
durable cross-run state reclaim and restore rebuild the aggregate from. This is
why routing requires a shared, durable `log_dir` (guard in `load_config`).
- **Reclaim** verifies routed targets as AGGREGATES: `verify_target()` checks
⋃ objects' arcnames / Σ counts / Σ bytes, every object tar present, per-object
newer-than-run, and the INV-E exclusion re-gate. `--scan` picks manifests up
automatically; routed per-object run-logs (`routed_object: true`) are skipped
by the legacy per-log `verify()`.
- **Reship-monitor**: a run re-shipping > 25% of shards emails `[RESHIP-WATCH]`
— the Phase-3 (leveled engine) trigger. Pure append trips ~0%.
- **Restore** (`restore.py`, LOCAL, operator-invoked): full or selective restore
into an **empty** dest. Gates, in order, each aborting the whole restore:
completeness (every selected object tar on tape BEFORE any transfer), empty
dest, per-object extract by recorded transport (`htar -xvf -L` random-access /
`hsi get` + `tar xf`), per-file MD5 vs each object's run-log, and a mandatory
final reconciliation (`walk(dest) == ⋃ arcnames`) on a full restore.

```bash
python3 restore.py config.json --dest /scratch/.../restore_here # full
python3 restore.py config.json --dest DIR --slot shard:3 # one shard
python3 restore.py config.json --dest DIR --member sub/path/file.bin # one file
```

---

## Known issues / history
Expand All @@ -334,4 +379,4 @@ any `LOST`.

- **Pre-run confirmation prompt** — print matched file count + size, then prompt `Proceed? [y/N]`
- **Overlap detection** — warn if files were already archived in a previous run
- **Vault-side manifest** — a `manifest.json` in `fortress_base_dir` listing all archives for the project
- **Vault-side manifest** — a `manifest.json` in `fortress_base_dir` listing all archives for the project. (Size-routed targets already keep a per-target manifest at `{log_dir}/_vault/`; this gap is about a Fortress-side listing for whole-target archives.)