RFC: incremental, split-source Fortress archiving (design only) #4
+144
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Design-only (no engine changes). The root-cause counterpart to florasense-tools #56: that PR stops spurious depot-tree churn; this addresses the actual cost driver — archive.py re-copies an entire experiment whenever anything in it changes. Proposes: (4A) split write-once-immutable bulk from volatile small files so a metadata edit never re-copies imagery/omics; (4B) a vault-side manifest + content-hash diff + delta tars (reusing the per-file MD5 the engine already computes and/or fs-checksum), which also delivers the "overlap detection" and "vault-side manifest" items already on the gaps list; (4C) manifest-driven restore + periodic consolidation to bound delta count. Phased: 1a measure-only (no behavior change) → 1b delta archiving → 1c split config. Value-blind, fail-safe (lost manifest ⇒ full archive, never silent skip). Open questions flagged for sign-off. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Jun 24, 2026
…nged Phase 1a of incremental archiving (PR #4 plan). The repository backup re-zipped and re-htar'd every target on every run; a target whose live source is the same set as its last SUCCESSFUL archive now skips — writing NO new tar and NO new run log, so the prior success log stays the canonical record reclaim/coverage read. Mechanism: before any preflight, archive.py finds the prior success log via skippable_prior_log and checks the live source against it with source_matches_log using EXACTLY reclaim.verify's no-drift conditions — same file count, nothing modified after the archive ran, same total bytes, same file_pattern, same source_folder — reusing reclaim.scan_source / run_datetime so the skip decision can never disagree with reclaim's notion of "still archived". One metadata-only walk; reads no contents. Config flag `skip_unchanged` (default false): the unchanged/changed decision is always computed and logged (free measurement of redundancy), but only ACTED on (skip) when enabled. Bypassed by --fresh / --force-resume. No run-log schema change; downstream reclaim/coverage untouched. Hardened per an adversarial review: - skippable_prior_log selects the latest success by parsed run_timestamp (NOT log-file mtime), agreeing with coverage.classify and immune to out-of-order mtimes (FS migration / cp -p), and returns it ONLY IF it is the newest run overall — a newer running/failed run (crash/retry in flight) blocks the skip. - file_pattern / source_folder scope changes force a re-archive. - Documented the accepted limitation: the stat-only triple can't see an equal-length, mtime-preserved in-place rewrite; bit-level integrity of archived data is fs-checksum-sweep's job (#60), the backstop for that pathological case. Tests (stdlib unittest, suite 37→52 green): skippable_prior_log (newest success; newer failure/running blocks skip; run_timestamp-not-mtime ordering; prefix sibling; no-success); source_matches_log (unchanged match; count/byte/newer-mtime /pattern-change/source-folder-change/missing-source/unparseable-ts → no match); end-to-end subprocess test asserting a skip exits 0 and writes NO new run log. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Jun 25, 2026
Reverts 5df25c6. The in-engine incremental approach (skip-unchanged Phase 1a + the PR #4 RFC) is superseded by Fortress-native incremental/ versioned backup, so the skip_unchanged path and its tests are removed rather than carried as dead, default-off code. No change to the default daily flow (skip_unchanged was default false). Removes archive.py's skippable_prior_log / source_matches_log helpers and the skip-decision block, plus tests/test_skip_unchanged.py. Remaining suite: 37 tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
|
Closing as superseded: the in-engine incremental/split-source approach in this RFC is replaced by Fortress-native incremental/versioned backup. The skip-unchanged Phase 1a code (#5) is being reverted in the companion PR. Rationale and the waste measurements that informed the decision are retained outside the repo. |
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Jun 26, 2026
…ted Path B) Authoritative buildable plan superseding the closed PR #4 RFC and the reverted PR #5 skip-unchanged. Recommends Path A (zip-delta, keep htar_large) now with Path B (GNU tar --listed-incremental + hsi put) deferred behind a Negishi spike. Covers level decision (cheap mtime+size diff), BLONDE + star-topology manifest (reclaim-ready), restore reassembly, leveled reclaim (verify_chain, required not deferred), the accepted same-size/same-mtime in-place-rewrite limitation, and a phased PR plan. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Jul 8, 2026
Wires the slice-1 primitives into an opt-in size-routing archive path. Default-off,
so an unset config is byte-for-byte the whole-target behavior (full suite 172 -> 192).
Config (load_config): size_routing (bool), t_small (int, floored 100 MB), shard_count
(null=derive), shard_target. DURABILITY GUARD (RFC §2.4): when size_routing is on,
refuse a log_dir under $HOME / scratch / the ~/globus_archive_tmp default — the
per-target manifest at {log_dir}/_vault/ must be shared+durable or reclaim can't find
it (routing_log_dir_ok). config.example.json annotated.
Engine:
- enumerate_source_catalog_excluded: one walk -> (survivor catalog, exclusion report),
the routing data source (exclusion applied once, before partitioning).
- route_and_ship (LOCAL): partition survivors -> objects; skip-unchanged (the append
win + resume); ship changed/new objects via the SAME make_zip_files ->
send_to_fortress chain (only_arcnames per object, exclude_spec=None); K frozen from
the manifest after baseline; drop deleted slots; manifest written atomically AFTER
EACH shipped object (crash-safe resume). object_project_name = per-object stem
(invariant #4).
- __main__: refuse size_routing + --globus up front (RFC §2.8); size-routing branch
replaces whole-target Step 1/2 and emails a [SUCCESS] summary.
Tests: tests/test_routing_wiring.py (20) — durability guard, config validation,
survivor enumerator (incl. exclusion drop), baseline/append/solo-change/shard-change/
delete/K-frozen/--fresh/resume-after-crash, exclusion recorded in manifest.
See docs/RFC_incremental_v2.md §2.5 (slice 2). Aggregate reclaim (slice 3) is REQUIRED
before a routed target is reclaimed, else verify() false-DRIFTs each object log.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign in
to join this conversation on GitHub.
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
RFC: incremental, split-source Fortress archiving — design only, no engine changes
The root-cause counterpart to florasense-tools #56. #56 stops spurious depot-tree churn (no-op runs re-writing files); this is the fix for the actual cost driver —
archive.pyre-copies an entire experiment whenever anything in it changes.Full doc:
docs/RFC_incremental_archive.md.TL;DR of the proposal:
manifest.json+ content-hash diff → delta tars of only new/changed files. Reuses the per-file MD5archive.pyalready computes (and/orfs-checksum). Delivers the "overlap detection" + "vault-side manifest" gaps already on the list. Content-hash, never mtime.Phased, each independently shippable + reversible:
Value-blind and fail-safe (a lost/corrupt manifest ⇒ treat everything as new = full archive, never a silent skip).
Ask: sign-off on the phased approach + the open questions in §6 — especially whether to kick off with Phase 1a (measure-only).
🤖 Generated with Claude Code