Skip to content

Conversation

@jdoucett
Copy link
Contributor

Summary

  • A new, separate class of failure has been hitting many repository targets since 2026-07-31 (the day after a Duke Power outage took Fortress offline 07-30/31): htar_large create calls now sometimes queue at HPSS for well over an hour before moving a single byte, occasionally exceeding the hardcoded 2h HTAR_STALL_SECONDS bound and getting killed as a false-positive hang.
  • Live evidence: repository_X0E_2_spectral-standoff (job 41943831, 2026-08-12) — a create sat silent for 4260.7s (71 min), then resumed and completed normally on its own. Meanwhile roughly a dozen other targets (X0C, X0E, X0F, X1A–X1G, ...) have had a create genuinely killed at exactly "no output for 2.0h" with zero prior progress since 07-31.
  • Simply raising HTAR_STALL_SECONDS would reopen the original 32h-hang risk this whole mechanism exists to catch. Instead, both stall-watch functions (_run_with_stall_watch, and send_to_fortress's self-contained inline run_watched() twin) now take a second budget, initial_stall_seconds, that applies ONLY before the very first byte of output/growth is observed — once any activity has been seen, the bound tightens back to the tight stall_seconds for the rest of the call. Omitting it (every existing caller before this PR) preserves today's exact single-bound behavior.
  • Both create call sites (ship_object, send_to_fortress) now pass initial_stall_seconds=retrieve_stall_seconds — reusing the existing config knob (already required/threaded for retrieve) rather than adding a third one, since the underlying phenomenon is the same regardless of read/write direction.
  • Found and fixed a real bug while building this: the file-growth mode's activity detection treated the first observation of a watched redirect file (created empty by the shell almost immediately on startup) as "activity," silently flipping the phase to the tight bound within milliseconds of every call — completely defeating the two-phase design for send_to_fortress's create. Fixed by only counting real growth (a size differing from a previously recorded baseline) as activity. Caught by a new direct regression test before this reached production.
  • Docs (CLAUDE.md invariant docs: RFC — staged-hybrid incremental backup (Phase 2 = size-routing/append) #7, stall_defaults.json, config.example.json, affected docstrings) updated to describe the three silence regimes: retrieve (whole call), create before first byte, create after first byte.

Test plan

  • New two-phase coverage in tests/test_stall_detection.py (both PIPE and file-growth modes): initial silence survives within budget, post-activity silence still killed fast, default (no initial_stall_seconds) behavior unchanged, and a direct regression test for the empty-redirect-file bug.
  • Updated tests/test_retrieve_stall_config.py's capturing wrapper + ship_object create-call assertions to also check initial_stall_seconds.
  • Updated tests/test_ship_object_retrieve_retry.py's fast-stall wrapper to accept the new kwarg.
  • Replaced tests/test_send_to_fortress_retrieve_stall.py's now-outdated single-phase create test with two tests matching the new two-phase contract.
  • Full suite: 435 passed (428 before this change + 7 new), no regressions.

Not merging or deploying — flagging for review per usual process. Related to and building on #38 (merged).

🤖 Generated with Claude Code

A new, separate class of failure has been hitting many repository targets
since 2026-07-31 (the day after a Duke Power outage took Fortress offline
07-30/31): htar_large create calls now sometimes queue at HPSS for well
over an hour before moving a single byte, occasionally exceeding the
hardcoded 2h HTAR_STALL_SECONDS bound and getting killed as a false-positive
hang. Confirmed live on repository_X0E_2_spectral-standoff (job 41943831,
2026-08-12): a create sat silent for 4260.7s (71 min), then resumed and
completed normally on its own - proving this is legitimate (if degraded)
HPSS queueing, not a dead connection, the same "HPSS queues and stages"
phenomenon that justified giving the round-trip retrieve its own
retrieve_stall_seconds budget. Since 07-31, roughly a dozen repository
targets (X0C, X0E, X0F, X1A-X1G, ...) have had a create genuinely killed at
exactly "no output for 2.0h" with zero prior progress.

Simply raising HTAR_STALL_SECONDS would reopen the original 32h-hang risk
this whole stall-watch mechanism exists to catch (a create that HAS started
streaming per-file progress going silent again really is a hang). Instead,
both stall-watch functions (_run_with_stall_watch, and send_to_fortress's
self-contained inline run_watched() twin, invariant #1) now take a second
budget, `initial_stall_seconds`, that applies ONLY before the very first
byte of output/growth is observed; once any activity has been seen, the
bound tightens back to the original stall_seconds for the rest of the call.
Passing only stall_seconds (every existing caller) preserves today's exact
single-bound behavior.

Both create call sites (ship_object, send_to_fortress) now pass
initial_stall_seconds=retrieve_stall_seconds - reusing the existing config
knob rather than adding a third one, since the underlying phenomenon (HPSS
taking a while to start moving data) is the same regardless of read or
write direction. Retrieve call sites are unchanged (they already use
retrieve_stall_seconds for their whole duration).

Found and fixed a real bug in the file-growth mode of both stall-watch
functions while building this: a watched redirect file is created/truncated
by the shell almost immediately on startup, so the FIRST observation of a
path (going from "never seen" to "seen, size 0") was being treated as
"activity" even though nothing had actually been written - silently
flipping the phase to the tight bound within milliseconds of every call and
completely defeating the two-phase design for anything using file-growth
mode (send_to_fortress's create, and any other watch_paths=[...] caller).
Fixed by only counting real growth (a size that differs from a previously
recorded baseline) as activity; a size first observed as 0 just establishes
the baseline. Caught by a new direct regression test
(test_freshly_created_empty_file_is_not_mistaken_for_activity) before this
reached production.

Docs updated (CLAUDE.md invariant #7, stall_defaults.json,
config.example.json, and the affected docstrings) to describe the three
silence regimes: retrieve (whole call, retrieve_stall_seconds), create
before first byte (retrieve_stall_seconds), create after first byte
(HTAR_STALL_SECONDS).

Tests: new two-phase coverage in test_stall_detection.py (PIPE and
file-growth modes: initial silence survives within budget, post-activity
silence still killed fast, default behavior unchanged, the empty-file
regression above); updated test_retrieve_stall_config.py's
_CapturingStallWatch and ship_object create-call assertions to also
capture/check initial_stall_seconds; updated test_ship_object_retrieve_retry.py's
_FastStallWatch to accept the new kwarg; replaced
test_send_to_fortress_retrieve_stall.py's now-outdated single-phase create
test with two tests matching the new two-phase contract. Full suite: 435
passed (428 before this change + 7 new), no regressions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign in to join this conversation on GitHub.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant