From ccd7832ac310374c2810bfcf3fb982814406569c Mon Sep 17 00:00:00 2001 From: "Doucette, Jarrod S" Date: Wed, 24 Jun 2026 15:03:40 -0400 Subject: [PATCH] docs: RFC for incremental, split-source Fortress archiving MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design-only (no engine changes). The root-cause counterpart to florasense-tools #56: that PR stops spurious depot-tree churn; this addresses the actual cost driver — archive.py re-copies an entire experiment whenever anything in it changes. Proposes: (4A) split write-once-immutable bulk from volatile small files so a metadata edit never re-copies imagery/omics; (4B) a vault-side manifest + content-hash diff + delta tars (reusing the per-file MD5 the engine already computes and/or fs-checksum), which also delivers the "overlap detection" and "vault-side manifest" items already on the gaps list; (4C) manifest-driven restore + periodic consolidation to bound delta count. Phased: 1a measure-only (no behavior change) → 1b delta archiving → 1c split config. Value-blind, fail-safe (lost manifest ⇒ full archive, never silent skip). Open questions flagged for sign-off. Co-Authored-By: Claude Opus 4.8 --- docs/RFC_incremental_archive.md | 144 ++++++++++++++++++++++++++++++++ 1 file changed, 144 insertions(+) create mode 100644 docs/RFC_incremental_archive.md diff --git a/docs/RFC_incremental_archive.md b/docs/RFC_incremental_archive.md new file mode 100644 index 0000000..8d70942 --- /dev/null +++ b/docs/RFC_incremental_archive.md @@ -0,0 +1,144 @@ +# RFC: Incremental, split-source Fortress archiving + +**Status:** DRAFT / for sign-off — **design only, no engine changes in this PR.** +**Author:** Jarrod Doucette (with Claude) +**Date:** 2026-06-24 + +This is the root-cause counterpart to florasense-tools #56. That PR stops +*spurious* depot-tree churn (no-op nightly runs re-writing provenance/HTML). This +RFC addresses the actual cost driver: **a backup that re-copies an entire +experiment whenever anything in it changes.** + +--- + +## 1. Problem + +`archive.py` walks the whole source tree, zips **every** matched file, and +`htar_large`-transfers a brand-new timestamped `.tar` to Fortress on every run +(`archive.py:304` zip, `:563` transfer). There is no change detection: a single +edited 50 KB XLSX in an experiment of thousands of imagery/omics files triggers a +full re-zip and re-transfer of the entire experiment, as a *duplicate* tar that +consumes additional tape quota. The tool's own `CLAUDE.md` lists the missing +pieces — "overlap detection" and "vault-side manifest" — as known gaps. + +**Impact:** redundant tape consumption proportional to (experiment size × number +of re-archives), and long runs that have themselves contributed to walltime +pressure on the nightly pipeline. + +## 2. What we already have to build on + +- **Per-file MD5 is already computed** every run (`file_checksums` passed into + `send_to_fortress`, `archive.py:410`) and a JSON manifest/log is already + written. The content-hash primitive for incremental diffing exists; it simply + isn't used to *skip* unchanged files. +- **`fs-checksum`** (florasense-tools) records per-experiment content hashes from + bytes — a second, authoritative hash source the archiver can reuse rather than + recomputing. +- **Same-filesystem landing-zone + repository** (one device) — relocation is + already hardlink-based; no extra copies in staging. + +## 3. Constraints (why not just "use restic/borg") + +- **Fortress is HPSS tape, not a CoW/dedup filesystem.** No block dedup, no + reflinks, no snapshots at the vault. Content-addressed backup tools that assume + a random-access dedup store don't apply. +- `htar_large` is used (not `htar`) to dodge the 64 GB single-file limit; it + writes a fresh tar and generates no index. +- The engine must stay **value-blind** (no reading research-data semantics) and + **fail-safe** (never delete/clobber on uncertainty), consistent with the + reclaim hardening (#2/#3) and florasense policy. + +The realistic design space on tape is therefore **manifest-diff + delta tars**, +not a dedup store. + +## 4. Proposal (two complementary changes) + +### 4A. Split volatile metadata from immutable bulk + +Most bytes are **write-once-immutable** (imagery, omics, raw sensor cubes); most +*changes* are in **small volatile files** (XLSX, JSON, txt, result spreadsheets). +Treat them as separate sources with separate cadences: + +- **Bulk (immutable):** archived **once**, verified, then never re-sent. A + re-archive is a no-op unless genuinely new bulk files appear. +- **Volatile (small):** archived on a cheap cadence — and for much of it, **git + already is the backup** (the depot metadata repo versions JSON/MD/etc.), so it + may not need Fortress at all. + +This alone removes the "small file forces big re-copy" failure mode structurally, +independent of 4B. + +*Mechanism:* per-experiment config gains an `immutable: true|false` (or a +`bulk_pattern` vs `volatile_pattern` split) so the two classes get separate tars +and separate schedules. Backward-compatible: unset = today's behavior. + +### 4B. Manifest-diff incremental archiving + +A **vault-side manifest** (`manifest.json` in `fortress_base_dir`) records, per +archived file: `{relpath, content_md5, size, tar, run_ts}`. Each run: + +1. Walk the source, compute (or reuse from `fs-checksum`) per-file MD5. +2. **Diff against the manifest:** classify each file `new` / `changed` + (hash differs) / `unchanged`. +3. If nothing is new/changed → **no-op** (log "0 new, 0 changed; nothing to + archive") and exit. This is the overlap detection the gaps list calls for. +4. Otherwise zip **only** the new/changed files into a **delta tar** + (`_delta_.tar`), transfer, verify round-trip, and **update the + manifest** atomically. + +Change detection is by **content hash**, never mtime (mtime is fragile — the +existing stale-zip guard's mtime check is exactly the kind of thing this +replaces). + +### 4C. Restore + consolidation + +Delta archiving means a restore = base tar + the relevant delta tars. To bound +this: + +- The manifest makes restore **planned, not guesswork**: it lists exactly which + tar holds the current version of each file. +- A periodic **consolidation** (e.g. monthly or at N deltas) re-bases an + experiment into a fresh full tar and prunes superseded deltas — keeping restore + simple and tape tidy. Consolidation is the *only* place a full re-copy happens, + on a deliberate cadence rather than on every drift. + +## 5. Phased rollout (each phase independently shippable + reversible) + +- **Phase 1a — manifest + overlap warning (no behavior change).** Write/maintain + the vault-side manifest; log "would skip N unchanged / re-archive M" but still + do the full copy. Pure visibility; lets us *measure* the redundancy on real + experiments before changing transfer behavior. Lowest risk. +- **Phase 1b — delta archiving.** Switch the transfer to send only new/changed + files once 1a's numbers confirm the model. Add a real-depot dry-run + restore + drill before enabling in the nightly path. +- **Phase 1c — volatile/immutable split config.** Roll out per-experiment, lead + with one experiment, verify, then expand. + +## 6. Risks & open questions (for sign-off) + +1. **Restore complexity vs. tape savings** — is the manifest + monthly + consolidation an acceptable trade, or is a simpler "full re-copy only on + explicit `--full`, otherwise refuse if unchanged" enough for now? +2. **Manifest as the new source of truth** — it must be versioned/backed up + itself (it's small JSON → git metadata repo is the natural home). A corrupt or + lost manifest must **fail safe** (fall back to treating everything as new, i.e. + a full archive — never silently skip). +3. **Reuse `fs-checksum` hashes vs. recompute** — preferred to avoid double I/O + over SAMBA, but couples the two tools' hash formats. Worth confirming. +4. **Consolidation cadence / trigger** — time-based, delta-count-based, or + manual? +5. **Where the volatile/immutable line is drawn** per modality — needs the data + stewards' input (which trees are truly write-once). + +## 7. Non-goals + +- Not changing reclaim or the canonical-root/opt-in safety model. +- Not introducing a dedup filesystem or a third-party backup tool (tape + constraint). +- Not reading or interpreting research-data values. + +--- + +**Ask:** sign-off on the **phased approach** and a decision on the open questions +in §6 — particularly whether to start with **Phase 1a (measure-only)**, which is +zero-risk and tells us the real redundancy before we touch transfer behavior.