fix(stall): give create's initial HPSS silence a two-phase budget #39
+395
−109
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
htar_largecreate calls now sometimes queue at HPSS for well over an hour before moving a single byte, occasionally exceeding the hardcoded 2hHTAR_STALL_SECONDSbound and getting killed as a false-positive hang.repository_X0E_2_spectral-standoff(job 41943831, 2026-08-12) — a create sat silent for 4260.7s (71 min), then resumed and completed normally on its own. Meanwhile roughly a dozen other targets (X0C, X0E, X0F, X1A–X1G, ...) have had a create genuinely killed at exactly "no output for 2.0h" with zero prior progress since 07-31.HTAR_STALL_SECONDSwould reopen the original 32h-hang risk this whole mechanism exists to catch. Instead, both stall-watch functions (_run_with_stall_watch, andsend_to_fortress's self-contained inlinerun_watched()twin) now take a second budget,initial_stall_seconds, that applies ONLY before the very first byte of output/growth is observed — once any activity has been seen, the bound tightens back to the tightstall_secondsfor the rest of the call. Omitting it (every existing caller before this PR) preserves today's exact single-bound behavior.ship_object,send_to_fortress) now passinitial_stall_seconds=retrieve_stall_seconds— reusing the existing config knob (already required/threaded for retrieve) rather than adding a third one, since the underlying phenomenon is the same regardless of read/write direction.send_to_fortress's create. Fixed by only counting real growth (a size differing from a previously recorded baseline) as activity. Caught by a new direct regression test before this reached production.CLAUDE.mdinvariant docs: RFC — staged-hybrid incremental backup (Phase 2 = size-routing/append) #7,stall_defaults.json,config.example.json, affected docstrings) updated to describe the three silence regimes: retrieve (whole call), create before first byte, create after first byte.Test plan
tests/test_stall_detection.py(both PIPE and file-growth modes): initial silence survives within budget, post-activity silence still killed fast, default (noinitial_stall_seconds) behavior unchanged, and a direct regression test for the empty-redirect-file bug.tests/test_retrieve_stall_config.py's capturing wrapper +ship_objectcreate-call assertions to also checkinitial_stall_seconds.tests/test_ship_object_retrieve_retry.py's fast-stall wrapper to accept the new kwarg.tests/test_send_to_fortress_retrieve_stall.py's now-outdated single-phase create test with two tests matching the new two-phase contract.Not merging or deploying — flagging for review per usual process. Related to and building on #38 (merged).
🤖 Generated with Claude Code