Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 40 additions & 15 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,23 +234,48 @@ raise if the call's silence budget passes with no new output — silence, not el
time, is the signal, since legitimate transfers range from minutes to 22+ hours (see
Known issues below).

The budget is **not one number**: a *create* streams per-file progress, so silence there
really is a hang and it keeps `HTAR_STALL_SECONDS` (2h); the post-create *round-trip
verify retrieve* is silent **by design** while HPSS queues and stages the tape, so both
`ship_object` **and** `send_to_fortress` bound it with the config's
**`retrieve_stall_seconds`** instead (default in `stall_defaults.json` — data, not code;
a per-asset config may override it; there is deliberately no code-side fallback, so a
caller that forgets to thread it down raises). Conflating the two is a real, expensive
bug, not a theoretical one: on 2026-08-03 a single 6.62 GB `X1D_3_metabolomics_rawspectra`
shard (routed via `ship_object`) had its healthy retrieve killed at exactly 2h twice per
job (its stall log reads `activity resumed after 7200.1s silence`), so verify never
passed, no success record was written, and every scheduled run re-shipped the whole
shard — 48 identical tars, ~318 GB of duplicate tape, over 2.5 weeks. The identical bug
independently existed in `send_to_fortress`'s own inline `run_watched()` twin (whole-target
and Phase-3 leveled targets) until it was given the same fix: confirmed hit on
The budget is **not one number** — it has THREE regimes, all fed by the same two
config-ish quantities (`HTAR_STALL_SECONDS`, a constant, and the config's
`retrieve_stall_seconds`):

1. **Retrieve, the whole call.** The post-create round-trip verify retrieve is silent
**by design** while HPSS queues and stages the tape, so both `ship_object` **and**
`send_to_fortress` bound its ENTIRE duration with the config's
**`retrieve_stall_seconds`** (default in `stall_defaults.json` — data, not code; a
per-asset config may override it; there is deliberately no code-side fallback, so a
caller that forgets to thread it down raises).
2. **Create, before the first byte moves.** A create used to be assumed to stream
per-file progress continuously from the start, so ANY silence meant a hang. That
assumption broke: 2026-07-31 onward (post a Duke Power outage that took Fortress
offline 07-30/31), creates began legitimately queueing at HPSS for well over an
hour with zero output before transfer actually begins — confirmed live on
`repository_X0E_2_spectral-standoff` (job 41943831, 2026-08-12): a create sat
silent for 4260.7s (71 min), then resumed and completed normally. So the create's
INITIAL silence — before ANY output/growth has ever been observed — is now ALSO
bounded by `retrieve_stall_seconds`, the same "HPSS is queueing/staging" budget
retrieve uses, via `_run_with_stall_watch()`'s / `run_watched()`'s
`initial_stall_seconds` parameter.
3. **Create, after the first byte moves.** Once a create HAS started streaming
per-file progress, the original rationale still holds — further silence really is a
hang — so the bound tightens back to the tight `HTAR_STALL_SECONDS` (2h) for the
rest of that call. This is what still catches a genuine mid-transfer hang fast (see
the 32h incident below) without needing to loosen the whole create's budget.

Conflating any of these is a real, expensive bug, not a theoretical one: on 2026-08-03
a single 6.62 GB `X1D_3_metabolomics_rawspectra` shard (routed via `ship_object`) had
its healthy retrieve killed at exactly 2h twice per job (its stall log reads `activity
resumed after 7200.1s silence`), so verify never passed, no success record was written,
and every scheduled run re-shipped the whole shard — 48 identical tars, ~318 GB of
duplicate tape, over 2.5 weeks. The identical retrieve bug independently existed in
`send_to_fortress`'s own inline `run_watched()` twin (whole-target and Phase-3 leveled
targets) until it was given the same fix: confirmed hit on
`repository_X0H_2_spectral-standoff` (job 41417829, 2026-07-31), a leveled-incremental
target whose round-trip retrieve was killed at exactly "no output for 2.0h" after a
successful create.
successful create. That same 07-31 date also marks the onset of the SEPARATE
create-queueing pattern (regime 2 above) — many repository targets (X0C, X0E, X0F,
X1A–X1G, ...) have since failed with a create killed at exactly "no output for 2.0h"
with zero prior progress, a pattern that lines up with the Duke Power outage rather
than a code bug.

Short, no-progress calls (`hsi mkdir`/`hsi ls`) just need
`timeout=HSI_SHORT_CALL_TIMEOUT_SECONDS` (5 min) on a plain `subprocess.run()`. A new
Expand Down
Loading