feat(skip-unchanged): don't re-archive an unchanged target (incremental archiving Phase 1a) #5
+341
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…nged Phase 1a of incremental archiving (PR #4 plan). The repository backup re-zipped and re-htar'd every target on every run; a target whose live source is the same set as its last SUCCESSFUL archive now skips — writing NO new tar and NO new run log, so the prior success log stays the canonical record reclaim/coverage read. Mechanism: before any preflight, archive.py finds the prior success log via skippable_prior_log and checks the live source against it with source_matches_log using EXACTLY reclaim.verify's no-drift conditions — same file count, nothing modified after the archive ran, same total bytes, same file_pattern, same source_folder — reusing reclaim.scan_source / run_datetime so the skip decision can never disagree with reclaim's notion of "still archived". One metadata-only walk; reads no contents. Config flag `skip_unchanged` (default false): the unchanged/changed decision is always computed and logged (free measurement of redundancy), but only ACTED on (skip) when enabled. Bypassed by --fresh / --force-resume. No run-log schema change; downstream reclaim/coverage untouched. Hardened per an adversarial review: - skippable_prior_log selects the latest success by parsed run_timestamp (NOT log-file mtime), agreeing with coverage.classify and immune to out-of-order mtimes (FS migration / cp -p), and returns it ONLY IF it is the newest run overall — a newer running/failed run (crash/retry in flight) blocks the skip. - file_pattern / source_folder scope changes force a re-archive. - Documented the accepted limitation: the stat-only triple can't see an equal-length, mtime-preserved in-place rewrite; bit-level integrity of archived data is fs-checksum-sweep's job (#60), the backstop for that pathological case. Tests (stdlib unittest, suite 37→52 green): skippable_prior_log (newest success; newer failure/running blocks skip; run_timestamp-not-mtime ordering; prefix sibling; no-success); source_matches_log (unchanged match; count/byte/newer-mtime /pattern-change/source-folder-change/missing-source/unparseable-ts → no match); end-to-end subprocess test asserting a skip exits 0 and writes NO new run log. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jun 25, 2026
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Jun 26, 2026
…ted Path B) Authoritative buildable plan superseding the closed PR #4 RFC and the reverted PR #5 skip-unchanged. Recommends Path A (zip-delta, keep htar_large) now with Path B (GNU tar --listed-incremental + hsi put) deferred behind a Negishi spike. Covers level decision (cheap mtime+size diff), BLONDE + star-topology manifest (reclaim-ready), restore reassembly, leveled reclaim (verify_chain, required not deferred), the accepted same-size/same-mtime in-place-rewrite limitation, and a phased PR plan. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jun 26, 2026
Sign in
to join this conversation on GitHub.
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Phase 1a of incremental archiving — skip an unchanged target
Implements Mechanism 1 of the PR #4 RFC plan. The repository backup re-zipped + re-htar'd every target on every run; a target whose live source is the same set as its last successful archive now skips — writing no new tar and no new run-log, so the prior success log stays the canonical record
reclaim/coverageread. (This is the engine's own "overlap detection" gap.)How
Before any preflight,
archive.py:skippable_prior_log(log_dir, project)→ the latest successful run-log, selected by parsed run_timestamp (not file mtime), returned only if it's the newest run overall.source_matches_log(log, source, pattern)→ unchanged iff same file count, nothing newer than the archive ran, same total bytes, same file_pattern, same source_folder — reusingreclaim.scan_source/run_datetime, so the skip decision can't disagree with reclaim's no-drift definition. One metadata-only walk; reads no bytes.skip_unchangedconfig flag, default false → measure-and-log only (free redundancy measurement); skipping is opt-in. Bypassed by--fresh/--force-resume. No run-log schema change; reclaim/coverage untouched.Hardened by an adversarial review (findings folded in)
coverage.classify; immune to out-of-order log mtimes (FS migration /cpwithout-p).running/failedrun blocks the skip, so a retry can re-establish success.file_patternor re-pointedsource_folderforces re-archive.fs-checksum-sweep's job (florasense-tools #60), the backstop for that pathological case.Tests
stdlib
unittest, suite 37 → 52 green:skippable_prior_log(newest success; newer failure/running blocks skip; run_timestamp-not-mtime ordering; prefix-sibling; no-success);source_matches_log(unchanged match; count/byte/newer-mtime/pattern-change/source-folder-change/missing-source/unparseable-ts → no match); end-to-end subprocess test that a skip exits 0 and writes no new run-log.Rollout (measure-first)
Default-off ships safely. Deploy → run one repository-backup cycle in measure-only mode (logs which targets would skip) → confirm
fs-coverage-audit/fs-reclaim-reportstill read every target as COVERED → then enableskip_unchanged. Mechanism 2 (keep churning metadata out of bulk targets) + Mechanism 3 (size-bundling) are follow-up PRs.Known generic-engine interactions noted for later (not live in this deployment):
reclaim --scan --days Nfilters by log mtime, so a long-quiescent skipped target ages out of that windowed view;log_dirdefaulting totmp_diris a foreign-log hazard for ad-hoc configs.🤖 Generated with Claude Code