chore(reclaim): hold AAPF landing-zone raw until promote loop verified #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AAPF raw HSI/RGB deliveries are archived to Fortress before being promoted into the repository by fs-promote-aapf (sibling florasense-tools repo). Until the archive→promote→reclaim loop is confirmed in steady state, reclaim.py must not auto-delete the landing-zone copies. Add a `landing-zone/AAPF` hold, mirroring the existing inbox block. Scoped to the source path (matched via verify()'s `h in source_folder` branch) so it catches every landing-zone/AAPF/<EXP> source but NOT a future promoted repository copy at a different path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Jun 11, 2026
…ip cleanup Root-cause fix for the repository-baseline false-DRIFT storm: verify() walked the live source unscoped, so pattern-scoped archives (e.g. _root targets, pattern ^[^/]+$) compared live=everything against archived=few and reported DRIFTED forever — re-archiving every _root/catalog/metadata target daily (~40 redundant tape writes) and tripping the drain's quarantine on 10 targets whose every run actually succeeded. - reclaim.scan_source(folder, pattern, cutoff): ONE metadata-only walk returns (count, newer, samples, bytes), scoped by the archive's file_pattern; the ^[^/]+$ top-level pattern is fast-pathed non-recursively (the catalog target no longer walks the whole 42TB tree to match 6 root files). Replaces the three separate unscoped walks verify() used to do. - verify(..., file_pattern=None): pattern = arg, else the log's recorded file_pattern, else None -> legacy unscoped behavior (old logs, landing-zone nightly reclaim unchanged). archive.py now records file_pattern in run logs. - write_log: dump the JSON run log on the first event, non-INFO events, and every 200th event (finalize always dumps) instead of on EVERY event — the per-event full rewrite was quadratic on 50k-file targets and being killed mid-dump is what produced the campaign's truncated JSONs. Policy mirrored in module-level should_flush_json() (unit-tested; inline copy because Compute functions cannot reference outer-scope names, invariant #1). - cleanup_zip_on_success (config, default false): remove the staging zip after the round-trip verification passes — for recurring/automated archives where retained zips accumulate without bound (the baseline left 547 zips / ~71TB in scratch). Default preserves invariant #5. Tests: tests/test_pattern_verify.py (14 cases: scan_source scoping/fast-path/ legacy parity, verify pattern-SAFE vs legacy-DRIFT vs real top-level drift, flush policy, config validation). Full suite 19/19 green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Jun 26, 2026
…nge)
First, behavior-neutral phase of the leveled-incremental plan
(docs/RFC_incremental_v2.md). Adds pure, additive helpers that nothing in the
archive flow calls yet:
- enumerate_source_catalog(): per-file {arcname: (mtime_ns, size)} catalog,
walking/matching identically to enumerate_source/make_zip_files so arcname
keys line up with file_checksums + the round-trip verify (invariant #2).
REMOTE-eligible; imports inside the body (dill / invariant #1).
- badge_of(): stable 40-bit base32 target identity (shake_128 of NUL-joined
project/source_folder/file_pattern) — the future manifest key.
- make_blonde()/decode_blonde(): BLONDE id 'badge.quaver.detail.genus' with a
lossless base36-epoch quaver; round-trips exactly.
- floor_s(): ns -> whole-second flooring so catalog mtimes compare cleanly
against second-precision zip timestamps.
Tests (tests/test_incremental_primitives.py, 17 cases): badge determinism /
format / per-field distinctness / NUL-join safety; blonde round-trip + delta
levels + quaver ordering; base36 zero/roundtrip/negative; floor_s + sub-second
jitter; catalog arcname keys, value shape, enumerate_source parity, pattern
scoping, top-level-only pattern, empty match. Full suite: 54 passing (was 37).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Jun 30, 2026
…no behavior change) First slice of docs/EXCLUSION_SPEC.md. Pure, additive helpers — NOTHING in the archive flow calls them yet (behavior-neutral, mirrors PR #8's primitives slice): - compile_exclude_spec / classify_tier — the 3-tier identity classifier (DATA allowlist first + un-overridable, then EXCLUDE, then OPTIONAL first-match, else DATA-default). Rejects bare-extension rules in the subtractive tiers at compile time (the spec's "never classify by extension" rule). DATA precedence + the greenhouse_climate lookahead are enforced. - git_coverage_preconditions / git_file_verdict — the fail-safe git gate. Per-run preconditions (repo+origin present, origin off the protected mount, durable-remote attestation, no git-LFS, fresh fetch, clean subtree) and the per-file proof (L/H/C/P/B/A) returning COVERED|UNTRACKED|DIRTY|STALE|ERROR. ONLY COVERED permits exclusion; every uncertain/error verdict -> back up. - spec_sha256 / _parse_duration helpers. Imports stay inside function bodies (Globus Compute / dill, invariant #1); the compiled spec (regex objects) is local-only, the raw dict crosses the boundary. Tests: tests/test_exclusion_primitives.py (+39) — classifier precedence incl. the greenhouse data-vs-_metadata-twin counterexample, the bare-extension guard, and the full git-gate matrix against real throwaway repos with a local bare remote. Suite 60 -> 99 green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign in
to join this conversation on GitHub.
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
AAPF raw HSI/RGB deliveries are archived to Fortress before being promoted into the repository by fs-promote-aapf (sibling florasense-tools repo). Until the archive→promote→reclaim loop is confirmed in steady state, reclaim.py must not auto-delete the landing-zone copies.
Add a
landing-zone/AAPFhold, mirroring the existing inbox block. Scoped to the source path (matched via verify()'sh in source_folderbranch) so it catches every landing-zone/AAPF/ source but NOT a future promoted repository copy at a different path.