chore(reclaim): hold AAPF landing-zone raw until promote loop verified #1
+8
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AAPF raw HSI/RGB deliveries are archived to Fortress before being promoted into the repository by fs-promote-aapf (sibling florasense-tools repo). Until the archive→promote→reclaim loop is confirmed in steady state, reclaim.py must not auto-delete the landing-zone copies. Add a `landing-zone/AAPF` hold, mirroring the existing inbox block. Scoped to the source path (matched via verify()'s `h in source_folder` branch) so it catches every landing-zone/AAPF/<EXP> source but NOT a future promoted repository copy at a different path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Jun 11, 2026
…ip cleanup Root-cause fix for the repository-baseline false-DRIFT storm: verify() walked the live source unscoped, so pattern-scoped archives (e.g. _root targets, pattern ^[^/]+$) compared live=everything against archived=few and reported DRIFTED forever — re-archiving every _root/catalog/metadata target daily (~40 redundant tape writes) and tripping the drain's quarantine on 10 targets whose every run actually succeeded. - reclaim.scan_source(folder, pattern, cutoff): ONE metadata-only walk returns (count, newer, samples, bytes), scoped by the archive's file_pattern; the ^[^/]+$ top-level pattern is fast-pathed non-recursively (the catalog target no longer walks the whole 42TB tree to match 6 root files). Replaces the three separate unscoped walks verify() used to do. - verify(..., file_pattern=None): pattern = arg, else the log's recorded file_pattern, else None -> legacy unscoped behavior (old logs, landing-zone nightly reclaim unchanged). archive.py now records file_pattern in run logs. - write_log: dump the JSON run log on the first event, non-INFO events, and every 200th event (finalize always dumps) instead of on EVERY event — the per-event full rewrite was quadratic on 50k-file targets and being killed mid-dump is what produced the campaign's truncated JSONs. Policy mirrored in module-level should_flush_json() (unit-tested; inline copy because Compute functions cannot reference outer-scope names, invariant #1). - cleanup_zip_on_success (config, default false): remove the staging zip after the round-trip verification passes — for recurring/automated archives where retained zips accumulate without bound (the baseline left 547 zips / ~71TB in scratch). Default preserves invariant #5. Tests: tests/test_pattern_verify.py (14 cases: scan_source scoping/fast-path/ legacy parity, verify pattern-SAFE vs legacy-DRIFT vs real top-level drift, flush policy, config validation). Full suite 19/19 green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Jun 26, 2026
…nge)
First, behavior-neutral phase of the leveled-incremental plan
(docs/RFC_incremental_v2.md). Adds pure, additive helpers that nothing in the
archive flow calls yet:
- enumerate_source_catalog(): per-file {arcname: (mtime_ns, size)} catalog,
walking/matching identically to enumerate_source/make_zip_files so arcname
keys line up with file_checksums + the round-trip verify (invariant #2).
REMOTE-eligible; imports inside the body (dill / invariant #1).
- badge_of(): stable 40-bit base32 target identity (shake_128 of NUL-joined
project/source_folder/file_pattern) — the future manifest key.
- make_blonde()/decode_blonde(): BLONDE id 'badge.quaver.detail.genus' with a
lossless base36-epoch quaver; round-trips exactly.
- floor_s(): ns -> whole-second flooring so catalog mtimes compare cleanly
against second-precision zip timestamps.
Tests (tests/test_incremental_primitives.py, 17 cases): badge determinism /
format / per-field distinctness / NUL-join safety; blonde round-trip + delta
levels + quaver ordering; base36 zero/roundtrip/negative; floor_s + sub-second
jitter; catalog arcname keys, value shape, enumerate_source parity, pattern
scoping, top-level-only pattern, empty match. Full suite: 54 passing (was 37).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Jun 30, 2026
…no behavior change) First slice of docs/EXCLUSION_SPEC.md. Pure, additive helpers — NOTHING in the archive flow calls them yet (behavior-neutral, mirrors PR #8's primitives slice): - compile_exclude_spec / classify_tier — the 3-tier identity classifier (DATA allowlist first + un-overridable, then EXCLUDE, then OPTIONAL first-match, else DATA-default). Rejects bare-extension rules in the subtractive tiers at compile time (the spec's "never classify by extension" rule). DATA precedence + the greenhouse_climate lookahead are enforced. - git_coverage_preconditions / git_file_verdict — the fail-safe git gate. Per-run preconditions (repo+origin present, origin off the protected mount, durable-remote attestation, no git-LFS, fresh fetch, clean subtree) and the per-file proof (L/H/C/P/B/A) returning COVERED|UNTRACKED|DIRTY|STALE|ERROR. ONLY COVERED permits exclusion; every uncertain/error verdict -> back up. - spec_sha256 / _parse_duration helpers. Imports stay inside function bodies (Globus Compute / dill, invariant #1); the compiled spec (regex objects) is local-only, the raw dict crosses the boundary. Tests: tests/test_exclusion_primitives.py (+39) — classifier precedence incl. the greenhouse data-vs-_metadata-twin counterexample, the bare-extension guard, and the full git-gate matrix against real throwaway repos with a local bare remote. Suite 60 -> 99 green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Jul 11, 2026
archive.py had zero Teams/webhook code — every [FAILED]/[STALE]/[EXCLUSION-ABORT] went out only as email, which gets tuned out. Add a stdlib-only notify_bell() that POSTs a Teams Adaptive Card @mention (title prefixed "ACT NOW:") beside the durable send_alert_email() audit copy — the "backup didn't happen" T1 tier of the settled notification-triage strategy. - notify_bell()/build_teams_card(): urllib.request only, imports inside the body (invariant #1). Env-gated + DEFAULT-OFF on EARLY_TEAMS_WEBHOOK (silent no-op until wired on Negishi); @mention from EARLY_TEAMS_MENTION_ID/_NAME (both required, else posts un-pinged). Only HTTP {200,202} = delivered; NEVER raises (degrades to email-only + logged reason). Webhook is a secret (sig) — never printed/logged. Card shape reused from florasense-tools reship_watch_alert.py. - Fired beside every LOCAL send_alert_email() failure site; the size-routing completion email ([SUCCESS]/[RESHIP-WATCH]) is deliberately NOT belled. - send_to_fortress carries a self-contained inline twin (closes over write_log only, imports inside) for the exclusion companion-guard abort — the one alert that continues the run (so it never reaches the local Step-2 wrapper) and can fire on a compute node under --globus. Other verification [FAILED]s raise → wrapper alerts+bells. - tests/test_notify_bell.py (15 tests): env-gated no-op, payload/@mention construction, ACT NOW prefix (+idempotency), 200/202 success, never-raises on HTTPError/URLError/ generic + non-2xx, and webhook-never-printed secret safety. Full suite 290 green. - CLAUDE.md: new "Failure notifications (Teams Activity-feed bell)" section. Does not touch the reship-watch alert; no --dm/--task modes (one bell surface). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Jul 11, 2026
archive.py had zero Teams/webhook code — every [FAILED]/[STALE]/[EXCLUSION-ABORT] went out only as email, which gets tuned out. Add a stdlib-only notify_bell() that POSTs a Teams Adaptive Card beside the durable send_alert_email() audit copy (title prefixed "ACT NOW:") — the "backup didn't happen" T1 tier of the settled notification-triage strategy. Delivered as a 1:1 Teams DM, NOT a channel @mention (Jarrod's routing decision, 2026-07-11; mirrors florasense-tools PR #106's --dm mode): the card carries no <at> / no msteams.entities, because a DM already notifies and a Power-Automate flow-bot chat post renders <at> as literal text. DM-vs-channel is decided by the flow behind EARLY_TEAMS_WEBHOOK, not the payload — the switch is a Power-Automate web-UI step done separately from this code. - notify_bell()/build_teams_card(): urllib.request only, imports inside the body (invariant #1). Env-gated + DEFAULT-OFF on EARLY_TEAMS_WEBHOOK (silent no-op until wired on Negishi); no mention env vars needed/read. Only HTTP {200,202} = delivered; NEVER raises (degrades to email-only + logged reason). Webhook is a secret (sig) — never printed/logged. Card shape reused from florasense-tools reship_watch_alert.py. - Fired beside every LOCAL send_alert_email() failure site; the size-routing completion email ([SUCCESS]/[RESHIP-WATCH]) is deliberately NOT belled. - send_to_fortress carries a self-contained inline twin (closes over write_log only, imports inside) for the exclusion companion-guard abort — the one alert that continues the run (so it never reaches the local Step-2 wrapper) and can fire on a compute node under --globus. Other verification [FAILED]s raise → wrapper alerts+bells. - tests/test_notify_bell.py (14 tests): env-gated no-op, DM-clean payload (no @mention, mention env vars ignored), ACT NOW prefix (+idempotency), 200/202 success, never-raises on HTTPError/URLError/generic + non-2xx, webhook-never-printed secret safety. Full suite 289 green. - CLAUDE.md: new "Failure notifications (Teams DM bell)" section. Does not touch the reship-watch alert; no --dm/--task modes (one alert surface). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Jul 18, 2026
Every subprocess.run() call to hsi/htar/htar_large/tar had no timeout, so a hung HPSS connection would block until Slurm's own wall-clock limit (up to 48h) - confirmed on Negishi job 40793013, which ran 32h+ before the htar_large create call's connection stalled with zero progress output. A flat overall timeout is wrong here since real transfers in the same batch legitimately ranged from minutes to 22+ hours; the fix instead watches for silence (no new stdout/stderr) via a Popen + non-blocking read/poll loop, killing and raising only when a create/retrieve call goes HTAR_STALL_SECONDS (2h) without output. Short mkdir/ls existence checks get a flat HSI_SHORT_CALL_TIMEOUT_SECONDS (5 min) instead, since they carry no progress signal to watch. ship_object() calls the new module-level _run_with_stall_watch() directly; send_to_fortress() carries a self-contained inline twin (run_watched) since it can run remotely via Globus Compute / dill (invariant #1). tests/test_ship_object_htar_rc.py is updated to drive real fake htar/hsi/tar executables on PATH (exercising the real Popen/select plumbing) instead of mocking subprocess.run directly, which the create/verify calls no longer use. tests/test_stall_detection.py adds direct coverage of both stall-watch modes (PIPE-captured and file-growth) and the short-call timeout path. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
3 tasks
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Aug 5, 2026
…rieve too PR #37 (23a7963) fixed ship_object's routed-object retrieve sharing the create-side HTAR_STALL_SECONDS (2h) silence budget, but send_to_fortress - the whole-target and Phase-3 leveled-incremental transport - carries its own fully self-contained inline twin of the stall-watch logic (run_watched, invariant #1: no outer-scope references, since it can run remotely under --globus) and was left on its own hardcoded 7200s default for both the create AND the two retrieve steps (hsi get, then local tar xvf). Confirmed hit independently of X1D: repository_X0H_2_spectral-standoff (job 41417829, 2026-07-31, a leveled-incremental, multi-TB whole-target tar) had a successful create, then its round-trip retrieve was killed at exactly "no output for 2.0h" - the identical bug class, on the unfixed twin. - send_to_fortress now takes retrieve_stall_seconds and REQUIRES it (raises before any htar_large/hsi/tar call) - no code-side fallback, mirroring ship_object's contract exactly. - Threaded into the two retrieve run_watched() calls only (hsi get, tar xvf); the create call keeps the hardcoded 7200s default - a create streams per-file progress, so silence there really is a hang. - Both real call sites (whole-target Step 2, and Phase-3's ship_level) updated to pass the already-loaded config value positionally (run forwards positional args only). - Reused stall_defaults.json's existing retrieve_stall_seconds default (21600s/6h) rather than adding a second knob: the config key was already path-agnostic ("the post-create round-trip verify retrieve"), and an operator can still raise it per-asset for X0H's larger multi-TB tars via the same override mechanism. - CLAUDE.md, stall_defaults.json, and config.example.json comments updated to describe both transports sharing this one bound. Tests: new tests/test_send_to_fortress_retrieve_stall.py - fakes htar_large/hsi/tar as shell-script stand-ins on PATH (matching test_ship_object_htar_rc.py's approach), since run_watched is a closure local to send_to_fortress and can't be intercepted the way archive._run_with_stall_watch is. Each behavioral assertion runs send_to_fortress in a background thread with a bounded join so a regression that falls back to the 2h default fails fast rather than hanging. Covers: hsi-get retrieve honors the configured bound, tar-extract retrieve honors it too, create stays on its own default even with a tiny configured bound, and the absent/invalid-value raises land before any tape work. Full suite: 428 passed (423 + 5 new), no regressions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2 tasks
Jarrod S Doucette (jdoucett)
added a commit
that referenced
this pull request
Aug 12, 2026
A new, separate class of failure has been hitting many repository targets since 2026-07-31 (the day after a Duke Power outage took Fortress offline 07-30/31): htar_large create calls now sometimes queue at HPSS for well over an hour before moving a single byte, occasionally exceeding the hardcoded 2h HTAR_STALL_SECONDS bound and getting killed as a false-positive hang. Confirmed live on repository_X0E_2_spectral-standoff (job 41943831, 2026-08-12): a create sat silent for 4260.7s (71 min), then resumed and completed normally on its own - proving this is legitimate (if degraded) HPSS queueing, not a dead connection, the same "HPSS queues and stages" phenomenon that justified giving the round-trip retrieve its own retrieve_stall_seconds budget. Since 07-31, roughly a dozen repository targets (X0C, X0E, X0F, X1A-X1G, ...) have had a create genuinely killed at exactly "no output for 2.0h" with zero prior progress. Simply raising HTAR_STALL_SECONDS would reopen the original 32h-hang risk this whole stall-watch mechanism exists to catch (a create that HAS started streaming per-file progress going silent again really is a hang). Instead, both stall-watch functions (_run_with_stall_watch, and send_to_fortress's self-contained inline run_watched() twin, invariant #1) now take a second budget, `initial_stall_seconds`, that applies ONLY before the very first byte of output/growth is observed; once any activity has been seen, the bound tightens back to the original stall_seconds for the rest of the call. Passing only stall_seconds (every existing caller) preserves today's exact single-bound behavior. Both create call sites (ship_object, send_to_fortress) now pass initial_stall_seconds=retrieve_stall_seconds - reusing the existing config knob rather than adding a third one, since the underlying phenomenon (HPSS taking a while to start moving data) is the same regardless of read or write direction. Retrieve call sites are unchanged (they already use retrieve_stall_seconds for their whole duration). Found and fixed a real bug in the file-growth mode of both stall-watch functions while building this: a watched redirect file is created/truncated by the shell almost immediately on startup, so the FIRST observation of a path (going from "never seen" to "seen, size 0") was being treated as "activity" even though nothing had actually been written - silently flipping the phase to the tight bound within milliseconds of every call and completely defeating the two-phase design for anything using file-growth mode (send_to_fortress's create, and any other watch_paths=[...] caller). Fixed by only counting real growth (a size that differs from a previously recorded baseline) as activity; a size first observed as 0 just establishes the baseline. Caught by a new direct regression test (test_freshly_created_empty_file_is_not_mistaken_for_activity) before this reached production. Docs updated (CLAUDE.md invariant #7, stall_defaults.json, config.example.json, and the affected docstrings) to describe the three silence regimes: retrieve (whole call, retrieve_stall_seconds), create before first byte (retrieve_stall_seconds), create after first byte (HTAR_STALL_SECONDS). Tests: new two-phase coverage in test_stall_detection.py (PIPE and file-growth modes: initial silence survives within budget, post-activity silence still killed fast, default behavior unchanged, the empty-file regression above); updated test_retrieve_stall_config.py's _CapturingStallWatch and ship_object create-call assertions to also capture/check initial_stall_seconds; updated test_ship_object_retrieve_retry.py's _FastStallWatch to accept the new kwarg; replaced test_send_to_fortress_retrieve_stall.py's now-outdated single-phase create test with two tests matching the new two-phase contract. Full suite: 435 passed (428 before this change + 7 new), no regressions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign in
to join this conversation on GitHub.
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
AAPF raw HSI/RGB deliveries are archived to Fortress before being promoted into the repository by fs-promote-aapf (sibling florasense-tools repo). Until the archive→promote→reclaim loop is confirmed in steady state, reclaim.py must not auto-delete the landing-zone copies.
Add a
landing-zone/AAPFhold, mirroring the existing inbox block. Scoped to the source path (matched via verify()'sh in source_folderbranch) so it catches every landing-zone/AAPF/ source but NOT a future promoted repository copy at a different path.