fix(archive): kill stalled hsi/htar subprocess calls instead of hanging

[Jarrod S Doucette (jdoucett)]

Summary

• Every subprocess.run() call to hsi/htar/htar_large/tar in archive.py had no timeout= - a hung HPSS connection would block until Slurm's own wall-clock limit (up to 48h). Confirmed on Negishi job 40793013 (repository_X1D_3_metabolomics_rawspectra), which ran 32h+ before manual cancellation; its .out file showed the destination hsi mkdir line and then nothing - the htar_large create call's connection stalled before transferring a byte.
• A flat overall timeout is wrong: real transfers in the same batch legitimately ranged from minutes to 22+ hours (sacct confirmed every other job that batch completed fine - an isolated stall, not a broader HPSS outage). The signal that distinguishes a hang from a merely-long transfer is silence, not elapsed time.
• Adds _run_with_stall_watch() (module-level, used directly by ship_object) and a self-contained inline twin run_watched() inside send_to_fortress (invariant chore(reclaim): hold AAPF landing-zone raw until promote loop verified #1 - dill-safety for --globus). Both drive Popen + a non-blocking read/poll loop, killing the process and raising a distinguishable RuntimeError if HTAR_STALL_SECONDS (2h) passes with no new stdout/stderr.
• Short, no-progress calls (hsi mkdir / hsi ls) get a flat HSI_SHORT_CALL_TIMEOUT_SECONDS (5 min) on plain subprocess.run() instead, since they carry no progress signal to watch.
• Existing failure-handling/retry semantics are unchanged - a stall just raises like any other known failure (own [FAILED] email + log in send_to_fortress; propagates to the caller's [FAILED]/bell handling from ship_object, same as before). Nothing new to retry: skip-unchanged/resume already re-attempt a killed object/target cleanly.
• Documented as invariant docs: RFC — staged-hybrid incremental backup (Phase 2 = size-routing/append) #7 + a "Known issues" entry in CLAUDE.md.

Test plan

• New tests/test_stall_detection.py: direct coverage of _run_with_stall_watch() in both modes (PIPE-captured for ship_object, file-growth for send_to_fortress's twin) - steady output under the stall window completes normally, silence past the window kills the process and raises, a normal nonzero exit is unaffected, and the short-call timeout path is fail-safe.
• tests/test_ship_object_htar_rc.py updated to drive real fake htar/htar_large/hsi/tar executables on PATH (exercising the actual Popen/select plumbing) instead of mocking subprocess.run, since the create/round-trip calls no longer call it directly.
• Full suite green: python3 -m pytest tests/ -q -> 312 passed (303 previously + 9 new).

Not merging without approval per repo convention (production archive/retrieve code path).