fix(stall): bound the verify retrieve separately from the create (X1D re-ship loop)

[Jarrod S Doucette (jdoucett)]

What

ship_object's post-create round-trip verify retrieve shared the create-side HTAR_STALL_SECONDS (2 h) silence budget. It now has its own bound, retrieve_stall_seconds, whose default lives in data (stall_defaults.json, 21600 s = 6 h) and which a per-asset config may override.

Why

A create streams per-file progress, so silence there is a genuine hang. A retrieve is silent by design while HPSS queues and stages the tape. On a large routed shard the shared bound was killing healthy retrieves.

Observed — repository_X1D_3_metabolomics_rawspectra, one 6,624,961,024-byte shard:

• htar -cvf succeeded every run; the retrieve was killed at exactly 7200 s on both attempts (~8 h/job, exit 1);
• its own stall log: poll: 7191.9s since last activity (stall limit 7200s) then activity resumed after 7200.1s silence (+73 bytes) — the retrieve was healthy, 0.1 s past the kill line;
• verify never passed -> no success record -> the wrapper re-shipped the whole shard every scheduled run: 48 identical 6.62 GB tars, ~318 GB of duplicate tape, 2026-07-17 -> 08-03.

(Separate and already resolved: the 2026-08-01 HPSS outage produced a different mode, htar create failed (rc=71). Not conflated here.)

Changes

• stall_defaults.json (new, committed) — the default retrieve budget, with the incident recorded beside it. 6 h is ~3x the one observed silence gap and stays well inside the 48 h Slurm walltime even if both retrieve attempts burn their full budget.
• load_stall_defaults() — fails loud on a missing/malformed/invalid file; no code-side fallback. load_config reads it lazily, so a config that pins retrieve_stall_seconds needs no defaults file, and validates a per-asset override.
• ship_object(..., retrieve_stall_seconds) — required; raises before any htar/hsi call rather than silently reinstating 2 h. Threaded through the ship_one closure (positionally — the local/globus run dispatcher forwards positional args only).
• The create keeps HTAR_STALL_SECONDS, documented at both constants, in config.example.json, and in CLAUDE.md invariant docs: RFC — staged-hybrid incremental backup (Phase 2 = size-routing/append) #7.

Tests

New tests/test_retrieve_stall_config.py: the defaults-file contract (including the default must exceed the create bound — a smaller value would re-open the loop while looking configured), load_config default/override/validation, that the retrieve calls carry the configured bound while the create does not, and that an absent/invalid value raises before any tape work. Existing ship_object callers in three test files updated. Full suite: 423 passed.

Scope / deploy notes

• This alone does not stop a re-ship loop — a repeatedly-failing verify still re-ships. The per-target circuit breaker for that is florasense-tools PR (fortress/repo_backup.py), shipped alongside.
• Deploy is by git pull on the engine checkout, so stall_defaults.json travels with archive.py. Copying archive.py alone would make every config load fail loud — by design, but worth knowing.
• Not touched (out of scope, needs its own gated go/no-go): reclaiming the ~318 GB of duplicate X1D tars. That is a destructive Fortress op — gated-data-remediation protocol, provenance record first, and only after one verify actually passes.
• No merge, no deploy without explicit approval.