fix(stall-watch): instrument, close MD5-verify gap, retry retrieve stalls

[Jarrod S Doucette (jdoucett)]

Summary

Follow-up from the 2026-07-20 stall investigation (job 40937735,
repository_X1D_3_metabolomics_rawspectra): PR #32's stall-watch reliably killed 4
prior identical htar -xvf retrieve hangs at exactly "no output for 2.0h" — but this
one ran ~30 minutes past that bound with no way, after the fact, to tell whether that
was a genuine mechanism gap or a last-second activity reset (short of a live
strace/py-spy capture, which wasn't available). Implements recommendations 1-4
from that report.

• Instrument the stall-watch (_run_with_stall_watch + send_to_fortress's
  inline twin): a new activity_log path records a timestamped trail of every
  activity/poll event, but only once silence passes half of stall_seconds — a
  healthy multi-hour transfer leaves zero trace, so log_dir doesn't get littered.
  Wired into every create/retrieve call site in both ship_object and
  send_to_fortress. Closes the open question from the report for any recurrence.
• Close the MD5-verify-loop gap: every htar/hsi subprocess call was already
  stall-guarded, but the pure-Python per-file MD5 comparison loop that runs after a
  round-trip retrieve had no timeout of any kind — confirmed real by code read, not
  speculated. Added _run_callable_with_stall_watch / VerifyStallError, a
  thread-based watchdog for blocking reads with no subprocess to terminate. Genuine
  verification failures (mismatch, missing file) still raise/email exactly as
  before; only a real stall raises the new type.
• Retry-with-backoff on the retrieve step: ship_object's round-trip retrieve
  gets one automatic retry (RETRIEVE_STALL_MAX_ATTEMPTS=2, 60s backoff) — but only
  when the failure is specifically a stall-watch timeout, never a genuine extract
  failure. 4 of 5 recent stalls on this object were the create succeeding and only
  the retrieve hanging on a dead HPSS mover session (confirmed via CLOSE_WAIT
  sockets to hpss-mvr03 + a futex-wedged client during the live investigation).
• Also cancelled the confirmed-hung job (40937735) on Negishi as part of the same
  investigation — unrelated to this diff, no code change needed for that.

Test plan

• Full suite green: 409 passed (395 existing + 14 new), stable across 8+
  repeated runs (timing-sensitive tests use real subprocesses/threads with shrunk
  stall_seconds/poll_interval, no mocking of subprocess itself, matching the
  existing suite's convention)
• New tests cover: quiet-vs-logged activity_log behavior in both stall-watch
  modes (pipe + file-growth), the callable watchdog + exception propagation
  (VerifyStallError vs a genuine re-raised failure), and the retry/give-up/
  non-retry boundary cases for ship_object's retrieve step
• python3 -m pyflakes archive.py tests/ clean of new issues

🤖 Generated with Claude Code