fix(stall): thread retrieve_stall_seconds into send_to_fortress's retrieve too

CLAUDE.md

@@ -236,15 +236,21 @@ Known issues below).
 
 The budget is **not one number**: a *create* streams per-file progress, so silence there
 really is a hang and it keeps `HTAR_STALL_SECONDS` (2h); the post-create *round-trip
-verify retrieve* is silent **by design** while HPSS queues and stages the tape, so
-`ship_object` bounds it with the config's **`retrieve_stall_seconds`** instead (default
-in `stall_defaults.json` — data, not code; a per-asset config may override it; there is
-deliberately no code-side fallback, so a caller that forgets to thread it down raises).
-Conflating the two is a real, expensive bug, not a theoretical one: on 2026-08-03 a
-single 6.62 GB `X1D_3_metabolomics_rawspectra` shard had its healthy retrieve killed at
-exactly 2h twice per job (its stall log reads `activity resumed after 7200.1s silence`),
-so verify never passed, no success record was written, and every scheduled run re-shipped
-the whole shard — 48 identical tars, ~318 GB of duplicate tape, over 2.5 weeks.
+verify retrieve* is silent **by design** while HPSS queues and stages the tape, so both
+`ship_object` **and** `send_to_fortress` bound it with the config's
+**`retrieve_stall_seconds`** instead (default in `stall_defaults.json` — data, not code;
+a per-asset config may override it; there is deliberately no code-side fallback, so a
+caller that forgets to thread it down raises). Conflating the two is a real, expensive
+bug, not a theoretical one: on 2026-08-03 a single 6.62 GB `X1D_3_metabolomics_rawspectra`
+shard (routed via `ship_object`) had its healthy retrieve killed at exactly 2h twice per
+job (its stall log reads `activity resumed after 7200.1s silence`), so verify never
+passed, no success record was written, and every scheduled run re-shipped the whole
+shard — 48 identical tars, ~318 GB of duplicate tape, over 2.5 weeks. The identical bug
+independently existed in `send_to_fortress`'s own inline `run_watched()` twin (whole-target
+and Phase-3 leveled targets) until it was given the same fix: confirmed hit on
+`repository_X0H_2_spectral-standoff` (job 41417829, 2026-07-31), a leveled-incremental
+target whose round-trip retrieve was killed at exactly "no output for 2.0h" after a
+successful create.
 
 Short, no-progress calls (`hsi mkdir`/`hsi ls`) just need
 `timeout=HSI_SHORT_CALL_TIMEOUT_SECONDS` (5 min) on a plain `subprocess.run()`. A new

archive.py

@@ -3034,6 +3034,7 @@ def send_to_fortress(zip_path, zip_checksum, file_checksums, members,
                      source_bytes, source_folder, file_pattern,
                      fortress_base_dir, emails, project_name, log_dir,
                      cleanup_zip_on_success=False, exclusion=None,
+                     retrieve_stall_seconds=None,
                      extra_record_fields=None, quiet_success=False):
     """
     On the RCAC compute system: transfer the zip to Fortress via htar_large,
@@ -3047,6 +3048,21 @@ def send_to_fortress(zip_path, zip_checksum, file_checksums, members,
     All steps use full stdout+stderr logging since no binary streaming is involved.
     Log files are named after the zip (e.g. X0F_20260331_140000.txt/.json).
 
+    retrieve_stall_seconds is REQUIRED (no code-side default): the silence budget for
+    the two post-create round-trip verify retrieve calls only (hsi get, then the local
+    tar xvf), threaded down from the config (`retrieve_stall_seconds`, whose own
+    default lives in stall_defaults.json). Passing None raises rather than quietly
+    reinstating the create-side default this function's inline run_watched() otherwise
+    uses (7200s / 2h — kept in sync with HTAR_STALL_SECONDS by inspection, invariant
+    #1), which is far too tight for a large whole-target/leveled tar's tape stage: the
+    same class of bug that burned ~318 GB of duplicate tape on
+    repository_X1D_3_metabolomics_rawspectra via ship_object's twin of this gap (see
+    HTAR_STALL_SECONDS) — confirmed hit here too on
+    repository_X0H_2_spectral-standoff (job 41417829, 2026-07-31): create succeeded,
+    then the round-trip retrieve was killed at exactly "no output for 2.0h". The create
+    call keeps the hardcoded 7200s default: a create streams per-file progress, so
+    silence there really is a hang.
+
     extra_record_fields: optional dict merged into run_record at construction
     (default None is behavior-neutral for the existing whole-target caller).
     Phase 3 (leveled incremental, RFC §3) tags leveled_object/badge/level/
@@ -3076,6 +3092,15 @@ def send_to_fortress(zip_path, zip_checksum, file_checksums, members,
     import smtplib
     from email.mime.text import MIMEText
 
+    if not isinstance(retrieve_stall_seconds, int) \
+            or isinstance(retrieve_stall_seconds, bool) or retrieve_stall_seconds <= 0:
+        raise RuntimeError(
+            f"send_to_fortress: retrieve_stall_seconds must be a positive integer "
+            f"(seconds), got {retrieve_stall_seconds!r} — thread it down from the "
+            f"config (default in stall_defaults.json); there is deliberately no "
+            f"code-side fallback that could silently reinstate the create-side "
+            f"2h bound.")
+
     # hsi lives at /opt/hsi/bin on Negishi but is not always on the compute
     # node PATH (only the login node sources /etc/profile.d snippets that add
     # it). htar_large invokes hsi, so we prepend explicitly to avoid the
@@ -3529,6 +3554,7 @@ def worker():
                    f'>{hsi_get_out} 2>{hsi_get_err}')
     try:
         hsi_result = run_watched(hsi_get_cmd, [hsi_get_out, hsi_get_err],
+                                  stall_seconds=retrieve_stall_seconds,
                                   activity_log=os.path.join(log_dir, f"{zip_ts}.hsiget.stall.log"))
     except RuntimeError as e:
         write_log(f"hsi get stalled: {e}", level="ERROR")
@@ -3590,6 +3616,7 @@ def worker():
     tar_cmd = f'tar xvf {retrieved_tar} -C {retrieve_dir} >{tar_extract_out} 2>{tar_extract_err}'
     try:
         tar_result = run_watched(tar_cmd, [tar_extract_out, tar_extract_err],
+                                  stall_seconds=retrieve_stall_seconds,
                                   activity_log=os.path.join(log_dir, f"{zip_ts}.tarextract.stall.log"))
     except RuntimeError as e:
         write_log(f"tar extraction stalled: {e}", level="ERROR")
@@ -4561,6 +4588,10 @@ def ship_level(level_obj, stem):
                 zip_path, zip_checksum, file_checksums, zip_members,
                 lvl_source_bytes, source_folder, file_pattern, fortress_base,
                 emails, project_name, log_dir, cleanup_zip_on_success, exclusion,
+                # retrieve_stall_seconds is passed POSITIONALLY: `run` is the
+                # local/globus dispatcher (def run(fn, *fn_args)) and forwards
+                # positional args only.
+                retrieve_stall_seconds,
                 {"leveled_object": True,
                  "badge": badge_of(project_name, source_folder, file_pattern),
                  "level": level_obj["level"], "blonde": level_obj["blonde"],
@@ -4800,7 +4831,11 @@ def ship_level(level_obj, stem):
             send_to_fortress,
             zip_path, zip_checksum, file_checksums, members,
             source_bytes, source_folder, file_pattern, fortress_base,
-            emails, project_name, log_dir, cleanup_zip_on_success, exclusion
+            emails, project_name, log_dir, cleanup_zip_on_success, exclusion,
+            # retrieve_stall_seconds is passed POSITIONALLY: `run` is the
+            # local/globus dispatcher (def run(fn, *fn_args)) and forwards
+            # positional args only.
+            retrieve_stall_seconds,
         )
     except Exception as e:
         msg = (

config.example.json

@@ -35,7 +35,7 @@
     "comment_exclude_optional": "Optional (default []). Per-asset opt-out list of Tier-2 (OPTIONAL) rule ids from the exclude_spec (e.g. omics provenance/QC: aux_info, fastp, checksums). Honored only if this asset's file_pattern also ships the raw inputs those artifacts are regenerable from. Omit to back up all OPTIONAL artifacts (safe default).",
     "exclude_optional": [],
 
-    "comment_retrieve_stall_seconds": "Optional override (default: stall_defaults.json's retrieve_stall_seconds, currently 21600 = 6h). How long the post-create round-trip VERIFY RETRIEVE may go without producing any output before it is killed as a hung HPSS connection. Much larger than the create-side bound on purpose: a retrieve is silent while HPSS queues and stages the tape, and a too-tight bound kills a healthy retrieve — which, because verify never passes, makes the next run re-ship the whole object (the 2026-08 X1D shard loop; see stall_defaults.json). Raise it for a target whose tape-stage latency is legitimately longer.",
+    "comment_retrieve_stall_seconds": "Optional override (default: stall_defaults.json's retrieve_stall_seconds, currently 21600 = 6h). How long the post-create round-trip VERIFY RETRIEVE may go without producing any output before it is killed as a hung HPSS connection — applies whichever transport this target ships through (whole-target/leveled-incremental via send_to_fortress, or size-routed objects via ship_object; see stall_defaults.json). Much larger than the create-side bound on purpose: a retrieve is silent while HPSS queues and stages the tape, and a too-tight bound kills a healthy retrieve — which, because verify never passes, makes the next run re-ship the whole object (the 2026-08 X1D shard loop and the X0H whole-target loop; see stall_defaults.json). Raise it for a target whose tape-stage latency is legitimately longer (e.g. a much larger whole-target tar).",
     "retrieve_stall_seconds": 21600,
 
     "comment_size_routing": "Optional (default false). Phase-2 size-routing (docs/RFC_incremental_v2.md §2): instead of one whole-target tar, ship each file >= t_small as its own 'solo' object and bundle smaller files into content-addressed 'shards' (shard = hash(relpath) mod K). Unchanged objects skip, so a GROWING target ships only its new files (the append win). LOCAL mode only — refused under --globus. When true, log_dir MUST be shared+durable (a Depot logs dir), NOT home or scratch: the per-target manifest lives at {log_dir}/_vault/ and reclaim must find it. Omit for the default whole-target behavior.",

stall_defaults.json

@@ -1,6 +1,6 @@
 {
     "comment": "Engine-wide stall-watch defaults. Read by archive.load_config() for EVERY config (per-asset configs may override a key by name). This file is REQUIRED: a missing/malformed/invalid file raises rather than falling back to a value hidden in code, so the operative timeout is always inspectable here.",
 
-    "comment_retrieve_stall_seconds": "How long ship_object's post-create round-trip verify retrieve (htar -xvf, or hsi get + tar xf) may go without producing ANY output before it is killed as a hung HPSS connection. Deliberately much larger than the create-side HTAR_STALL_SECONDS (2h): a create streams per-file progress continuously, so silence there really is a hang, whereas a retrieve of a large routed shard is silent BY DESIGN while HPSS queues and stages the tape. 2026-08-03 (repository_X1D_3_metabolomics_rawspectra, a single 6,624,961,024-byte shard): the retrieve went quiet, was killed at exactly the 2h bound on both attempts, and its own stall log shows activity resuming 0.1s past the kill line ('activity resumed after 7200.1s silence') — legitimate tape-stage latency, misread as a hang. Verify never passed, so no success record was written and the next scheduled run re-shipped the whole shard from scratch: 48 identical 6.62 GB tars (~318 GB of duplicate tape) over 2026-07-17..08-03. 6h gives ~3x margin over the one observed gap while staying well inside the 48h Slurm walltime even if both retrieve attempts burn their full budget.",
+    "comment_retrieve_stall_seconds": "How long a target's post-create round-trip verify retrieve may go without producing ANY output before it is killed as a hung HPSS connection. Shared by BOTH transport paths: ship_object's routed-object retrieve (htar -xvf, or hsi get + tar xf) and send_to_fortress's whole-target/leveled-incremental retrieve (hsi get + tar xvf) via its own self-contained inline run_watched() twin (invariant #1) — one config knob, same semantics, regardless of which path an asset ships through. Deliberately much larger than the create-side HTAR_STALL_SECONDS (2h): a create streams per-file progress continuously, so silence there really is a hang, whereas a retrieve is silent BY DESIGN while HPSS queues and stages the tape. 2026-08-03 (repository_X1D_3_metabolomics_rawspectra, a single 6,624,961,024-byte routed shard): the retrieve went quiet, was killed at exactly the 2h bound on both attempts, and its own stall log shows activity resuming 0.1s past the kill line ('activity resumed after 7200.1s silence') — legitimate tape-stage latency, misread as a hang. Verify never passed, so no success record was written and the next scheduled run re-shipped the whole shard from scratch: 48 identical 6.62 GB tars (~318 GB of duplicate tape) over 2026-07-17..08-03. The identical bug independently existed in send_to_fortress's retrieve (confirmed on repository_X0H_2_spectral-standoff, job 41417829, 2026-07-31 — a leveled-incremental, multi-TB whole-target tar, killed at exactly 'no output for 2.0h') until it was given this same fix. 6h gives ~3x margin over the one precisely-measured gap (X1D's 0.1s overrun) while staying well inside the 48h Slurm walltime even if both retrieve attempts burn their full budget; a per-asset config may raise this further for a target whose tape-stage latency is legitimately longer (e.g. a much larger whole-target tar like X0H's).",
     "retrieve_stall_seconds": 21600
 }