chore(reclaim): hold AAPF landing-zone raw until promote loop verified #1
+8
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AAPF raw HSI/RGB deliveries are archived to Fortress before being promoted into the repository by fs-promote-aapf (sibling florasense-tools repo). Until the archive→promote→reclaim loop is confirmed in steady state, reclaim.py must not auto-delete the landing-zone copies. Add a `landing-zone/AAPF` hold, mirroring the existing inbox block. Scoped to the source path (matched via verify()'s `h in source_folder` branch) so it catches every landing-zone/AAPF/<EXP> source but NOT a future promoted repository copy at a different path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Jun 11, 2026
…ip cleanup Root-cause fix for the repository-baseline false-DRIFT storm: verify() walked the live source unscoped, so pattern-scoped archives (e.g. _root targets, pattern ^[^/]+$) compared live=everything against archived=few and reported DRIFTED forever — re-archiving every _root/catalog/metadata target daily (~40 redundant tape writes) and tripping the drain's quarantine on 10 targets whose every run actually succeeded. - reclaim.scan_source(folder, pattern, cutoff): ONE metadata-only walk returns (count, newer, samples, bytes), scoped by the archive's file_pattern; the ^[^/]+$ top-level pattern is fast-pathed non-recursively (the catalog target no longer walks the whole 42TB tree to match 6 root files). Replaces the three separate unscoped walks verify() used to do. - verify(..., file_pattern=None): pattern = arg, else the log's recorded file_pattern, else None -> legacy unscoped behavior (old logs, landing-zone nightly reclaim unchanged). archive.py now records file_pattern in run logs. - write_log: dump the JSON run log on the first event, non-INFO events, and every 200th event (finalize always dumps) instead of on EVERY event — the per-event full rewrite was quadratic on 50k-file targets and being killed mid-dump is what produced the campaign's truncated JSONs. Policy mirrored in module-level should_flush_json() (unit-tested; inline copy because Compute functions cannot reference outer-scope names, invariant #1). - cleanup_zip_on_success (config, default false): remove the staging zip after the round-trip verification passes — for recurring/automated archives where retained zips accumulate without bound (the baseline left 547 zips / ~71TB in scratch). Default preserves invariant #5. Tests: tests/test_pattern_verify.py (14 cases: scan_source scoping/fast-path/ legacy parity, verify pattern-SAFE vs legacy-DRIFT vs real top-level drift, flush policy, config validation). Full suite 19/19 green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Jun 26, 2026
…nge)
First, behavior-neutral phase of the leveled-incremental plan
(docs/RFC_incremental_v2.md). Adds pure, additive helpers that nothing in the
archive flow calls yet:
- enumerate_source_catalog(): per-file {arcname: (mtime_ns, size)} catalog,
walking/matching identically to enumerate_source/make_zip_files so arcname
keys line up with file_checksums + the round-trip verify (invariant #2).
REMOTE-eligible; imports inside the body (dill / invariant #1).
- badge_of(): stable 40-bit base32 target identity (shake_128 of NUL-joined
project/source_folder/file_pattern) — the future manifest key.
- make_blonde()/decode_blonde(): BLONDE id 'badge.quaver.detail.genus' with a
lossless base36-epoch quaver; round-trips exactly.
- floor_s(): ns -> whole-second flooring so catalog mtimes compare cleanly
against second-precision zip timestamps.
Tests (tests/test_incremental_primitives.py, 17 cases): badge determinism /
format / per-field distinctness / NUL-join safety; blonde round-trip + delta
levels + quaver ordering; base36 zero/roundtrip/negative; floor_s + sub-second
jitter; catalog arcname keys, value shape, enumerate_source parity, pattern
scoping, top-level-only pattern, empty match. Full suite: 54 passing (was 37).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Jun 30, 2026
…no behavior change) First slice of docs/EXCLUSION_SPEC.md. Pure, additive helpers — NOTHING in the archive flow calls them yet (behavior-neutral, mirrors PR #8's primitives slice): - compile_exclude_spec / classify_tier — the 3-tier identity classifier (DATA allowlist first + un-overridable, then EXCLUDE, then OPTIONAL first-match, else DATA-default). Rejects bare-extension rules in the subtractive tiers at compile time (the spec's "never classify by extension" rule). DATA precedence + the greenhouse_climate lookahead are enforced. - git_coverage_preconditions / git_file_verdict — the fail-safe git gate. Per-run preconditions (repo+origin present, origin off the protected mount, durable-remote attestation, no git-LFS, fresh fetch, clean subtree) and the per-file proof (L/H/C/P/B/A) returning COVERED|UNTRACKED|DIRTY|STALE|ERROR. ONLY COVERED permits exclusion; every uncertain/error verdict -> back up. - spec_sha256 / _parse_duration helpers. Imports stay inside function bodies (Globus Compute / dill, invariant #1); the compiled spec (regex objects) is local-only, the raw dict crosses the boundary. Tests: tests/test_exclusion_primitives.py (+39) — classifier precedence incl. the greenhouse data-vs-_metadata-twin counterexample, the bare-extension guard, and the full git-gate matrix against real throwaway repos with a local bare remote. Suite 60 -> 99 green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Jul 11, 2026
archive.py had zero Teams/webhook code — every [FAILED]/[STALE]/[EXCLUSION-ABORT] went out only as email, which gets tuned out. Add a stdlib-only notify_bell() that POSTs a Teams Adaptive Card @mention (title prefixed "ACT NOW:") beside the durable send_alert_email() audit copy — the "backup didn't happen" T1 tier of the settled notification-triage strategy. - notify_bell()/build_teams_card(): urllib.request only, imports inside the body (invariant #1). Env-gated + DEFAULT-OFF on EARLY_TEAMS_WEBHOOK (silent no-op until wired on Negishi); @mention from EARLY_TEAMS_MENTION_ID/_NAME (both required, else posts un-pinged). Only HTTP {200,202} = delivered; NEVER raises (degrades to email-only + logged reason). Webhook is a secret (sig) — never printed/logged. Card shape reused from florasense-tools reship_watch_alert.py. - Fired beside every LOCAL send_alert_email() failure site; the size-routing completion email ([SUCCESS]/[RESHIP-WATCH]) is deliberately NOT belled. - send_to_fortress carries a self-contained inline twin (closes over write_log only, imports inside) for the exclusion companion-guard abort — the one alert that continues the run (so it never reaches the local Step-2 wrapper) and can fire on a compute node under --globus. Other verification [FAILED]s raise → wrapper alerts+bells. - tests/test_notify_bell.py (15 tests): env-gated no-op, payload/@mention construction, ACT NOW prefix (+idempotency), 200/202 success, never-raises on HTTPError/URLError/ generic + non-2xx, and webhook-never-printed secret safety. Full suite 290 green. - CLAUDE.md: new "Failure notifications (Teams Activity-feed bell)" section. Does not touch the reship-watch alert; no --dm/--task modes (one bell surface). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Jul 11, 2026
archive.py had zero Teams/webhook code — every [FAILED]/[STALE]/[EXCLUSION-ABORT] went out only as email, which gets tuned out. Add a stdlib-only notify_bell() that POSTs a Teams Adaptive Card beside the durable send_alert_email() audit copy (title prefixed "ACT NOW:") — the "backup didn't happen" T1 tier of the settled notification-triage strategy. Delivered as a 1:1 Teams DM, NOT a channel @mention (Jarrod's routing decision, 2026-07-11; mirrors florasense-tools PR #106's --dm mode): the card carries no <at> / no msteams.entities, because a DM already notifies and a Power-Automate flow-bot chat post renders <at> as literal text. DM-vs-channel is decided by the flow behind EARLY_TEAMS_WEBHOOK, not the payload — the switch is a Power-Automate web-UI step done separately from this code. - notify_bell()/build_teams_card(): urllib.request only, imports inside the body (invariant #1). Env-gated + DEFAULT-OFF on EARLY_TEAMS_WEBHOOK (silent no-op until wired on Negishi); no mention env vars needed/read. Only HTTP {200,202} = delivered; NEVER raises (degrades to email-only + logged reason). Webhook is a secret (sig) — never printed/logged. Card shape reused from florasense-tools reship_watch_alert.py. - Fired beside every LOCAL send_alert_email() failure site; the size-routing completion email ([SUCCESS]/[RESHIP-WATCH]) is deliberately NOT belled. - send_to_fortress carries a self-contained inline twin (closes over write_log only, imports inside) for the exclusion companion-guard abort — the one alert that continues the run (so it never reaches the local Step-2 wrapper) and can fire on a compute node under --globus. Other verification [FAILED]s raise → wrapper alerts+bells. - tests/test_notify_bell.py (14 tests): env-gated no-op, DM-clean payload (no @mention, mention env vars ignored), ACT NOW prefix (+idempotency), 200/202 success, never-raises on HTTPError/URLError/generic + non-2xx, webhook-never-printed secret safety. Full suite 289 green. - CLAUDE.md: new "Failure notifications (Teams DM bell)" section. Does not touch the reship-watch alert; no --dm/--task modes (one alert surface). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Jul 18, 2026
Every subprocess.run() call to hsi/htar/htar_large/tar had no timeout, so a hung HPSS connection would block until Slurm's own wall-clock limit (up to 48h) - confirmed on Negishi job 40793013, which ran 32h+ before the htar_large create call's connection stalled with zero progress output. A flat overall timeout is wrong here since real transfers in the same batch legitimately ranged from minutes to 22+ hours; the fix instead watches for silence (no new stdout/stderr) via a Popen + non-blocking read/poll loop, killing and raising only when a create/retrieve call goes HTAR_STALL_SECONDS (2h) without output. Short mkdir/ls existence checks get a flat HSI_SHORT_CALL_TIMEOUT_SECONDS (5 min) instead, since they carry no progress signal to watch. ship_object() calls the new module-level _run_with_stall_watch() directly; send_to_fortress() carries a self-contained inline twin (run_watched) since it can run remotely via Globus Compute / dill (invariant #1). tests/test_ship_object_htar_rc.py is updated to drive real fake htar/hsi/tar executables on PATH (exercising the real Popen/select plumbing) instead of mocking subprocess.run directly, which the create/verify calls no longer use. tests/test_stall_detection.py adds direct coverage of both stall-watch modes (PIPE-captured and file-growth) and the short-call timeout path. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
3 tasks
Sign in
to join this conversation on GitHub.
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
AAPF raw HSI/RGB deliveries are archived to Fortress before being promoted into the repository by fs-promote-aapf (sibling florasense-tools repo). Until the archive→promote→reclaim loop is confirmed in steady state, reclaim.py must not auto-delete the landing-zone copies.
Add a
landing-zone/AAPFhold, mirroring the existing inbox block. Scoped to the source path (matched via verify()'sh in source_folderbranch) so it catches every landing-zone/AAPF/ source but NOT a future promoted repository copy at a different path.