RFC: incremental, split-source Fortress archiving (design only)

docs/RFC_incremental_archive.md

@@ -0,0 +1,144 @@
+# RFC: Incremental, split-source Fortress archiving
+
+**Status:** DRAFT / for sign-off — **design only, no engine changes in this PR.**
+**Author:** Jarrod Doucette (with Claude)
+**Date:** 2026-06-24
+
+This is the root-cause counterpart to florasense-tools #56. That PR stops
+*spurious* depot-tree churn (no-op nightly runs re-writing provenance/HTML). This
+RFC addresses the actual cost driver: **a backup that re-copies an entire
+experiment whenever anything in it changes.**
+
+---
+
+## 1. Problem
+
+`archive.py` walks the whole source tree, zips **every** matched file, and
+`htar_large`-transfers a brand-new timestamped `.tar` to Fortress on every run
+(`archive.py:304` zip, `:563` transfer). There is no change detection: a single
+edited 50 KB XLSX in an experiment of thousands of imagery/omics files triggers a
+full re-zip and re-transfer of the entire experiment, as a *duplicate* tar that
+consumes additional tape quota. The tool's own `CLAUDE.md` lists the missing
+pieces — "overlap detection" and "vault-side manifest" — as known gaps.
+
+**Impact:** redundant tape consumption proportional to (experiment size × number
+of re-archives), and long runs that have themselves contributed to walltime
+pressure on the nightly pipeline.
+
+## 2. What we already have to build on
+
+- **Per-file MD5 is already computed** every run (`file_checksums` passed into
+  `send_to_fortress`, `archive.py:410`) and a JSON manifest/log is already
+  written. The content-hash primitive for incremental diffing exists; it simply
+  isn't used to *skip* unchanged files.
+- **`fs-checksum`** (florasense-tools) records per-experiment content hashes from
+  bytes — a second, authoritative hash source the archiver can reuse rather than
+  recomputing.
+- **Same-filesystem landing-zone + repository** (one device) — relocation is
+  already hardlink-based; no extra copies in staging.
+
+## 3. Constraints (why not just "use restic/borg")
+
+- **Fortress is HPSS tape, not a CoW/dedup filesystem.** No block dedup, no
+  reflinks, no snapshots at the vault. Content-addressed backup tools that assume
+  a random-access dedup store don't apply.
+- `htar_large` is used (not `htar`) to dodge the 64 GB single-file limit; it
+  writes a fresh tar and generates no index.
+- The engine must stay **value-blind** (no reading research-data semantics) and
+  **fail-safe** (never delete/clobber on uncertainty), consistent with the
+  reclaim hardening (#2/#3) and florasense policy.
+
+The realistic design space on tape is therefore **manifest-diff + delta tars**,
+not a dedup store.
+
+## 4. Proposal (two complementary changes)
+
+### 4A. Split volatile metadata from immutable bulk
+
+Most bytes are **write-once-immutable** (imagery, omics, raw sensor cubes); most
+*changes* are in **small volatile files** (XLSX, JSON, txt, result spreadsheets).
+Treat them as separate sources with separate cadences:
+
+- **Bulk (immutable):** archived **once**, verified, then never re-sent. A
+  re-archive is a no-op unless genuinely new bulk files appear.
+- **Volatile (small):** archived on a cheap cadence — and for much of it, **git
+  already is the backup** (the depot metadata repo versions JSON/MD/etc.), so it
+  may not need Fortress at all.
+
+This alone removes the "small file forces big re-copy" failure mode structurally,
+independent of 4B.
+
+*Mechanism:* per-experiment config gains an `immutable: true|false` (or a
+`bulk_pattern` vs `volatile_pattern` split) so the two classes get separate tars
+and separate schedules. Backward-compatible: unset = today's behavior.
+
+### 4B. Manifest-diff incremental archiving
+
+A **vault-side manifest** (`manifest.json` in `fortress_base_dir`) records, per
+archived file: `{relpath, content_md5, size, tar, run_ts}`. Each run:
+
+1. Walk the source, compute (or reuse from `fs-checksum`) per-file MD5.
+2. **Diff against the manifest:** classify each file `new` / `changed`
+   (hash differs) / `unchanged`.
+3. If nothing is new/changed → **no-op** (log "0 new, 0 changed; nothing to
+   archive") and exit. This is the overlap detection the gaps list calls for.
+4. Otherwise zip **only** the new/changed files into a **delta tar**
+   (`<project>_delta_<ts>.tar`), transfer, verify round-trip, and **update the
+   manifest** atomically.
+
+Change detection is by **content hash**, never mtime (mtime is fragile — the
+existing stale-zip guard's mtime check is exactly the kind of thing this
+replaces).
+
+### 4C. Restore + consolidation
+
+Delta archiving means a restore = base tar + the relevant delta tars. To bound
+this:
+
+- The manifest makes restore **planned, not guesswork**: it lists exactly which
+  tar holds the current version of each file.
+- A periodic **consolidation** (e.g. monthly or at N deltas) re-bases an
+  experiment into a fresh full tar and prunes superseded deltas — keeping restore
+  simple and tape tidy. Consolidation is the *only* place a full re-copy happens,
+  on a deliberate cadence rather than on every drift.
+
+## 5. Phased rollout (each phase independently shippable + reversible)
+
+- **Phase 1a — manifest + overlap warning (no behavior change).** Write/maintain
+  the vault-side manifest; log "would skip N unchanged / re-archive M" but still
+  do the full copy. Pure visibility; lets us *measure* the redundancy on real
+  experiments before changing transfer behavior. Lowest risk.
+- **Phase 1b — delta archiving.** Switch the transfer to send only new/changed
+  files once 1a's numbers confirm the model. Add a real-depot dry-run + restore
+  drill before enabling in the nightly path.
+- **Phase 1c — volatile/immutable split config.** Roll out per-experiment, lead
+  with one experiment, verify, then expand.
+
+## 6. Risks & open questions (for sign-off)
+
+1. **Restore complexity vs. tape savings** — is the manifest + monthly
+   consolidation an acceptable trade, or is a simpler "full re-copy only on
+   explicit `--full`, otherwise refuse if unchanged" enough for now?
+2. **Manifest as the new source of truth** — it must be versioned/backed up
+   itself (it's small JSON → git metadata repo is the natural home). A corrupt or
+   lost manifest must **fail safe** (fall back to treating everything as new, i.e.
+   a full archive — never silently skip).
+3. **Reuse `fs-checksum` hashes vs. recompute** — preferred to avoid double I/O
+   over SAMBA, but couples the two tools' hash formats. Worth confirming.
+4. **Consolidation cadence / trigger** — time-based, delta-count-based, or
+   manual?
+5. **Where the volatile/immutable line is drawn** per modality — needs the data
+   stewards' input (which trees are truly write-once).
+
+## 7. Non-goals
+
+- Not changing reclaim or the canonical-root/opt-in safety model.
+- Not introducing a dedup filesystem or a third-party backup tool (tape
+  constraint).
+- Not reading or interpreting research-data values.
+
+---
+
+**Ask:** sign-off on the **phased approach** and a decision on the open questions
+in §6 — particularly whether to start with **Phase 1a (measure-only)**, which is
+zero-risk and tells us the real redundancy before we touch transfer behavior.