diff --git a/CLAUDE.md b/CLAUDE.md index ca32b40..1c90697 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -368,7 +368,24 @@ archives byte-for-byte as before. Authoritative design: `docs/RFC_incremental_v2 whole-target `find_existing_zip` resume machinery does not apply to routed runs. - **Transport per object** (`ship_object`): tars the SOURCE directly (no zip); real `htar` (indexed `.tar.idx`, random-access restore) unless a member ≥ 64 GiB - → `htar_large`. Invariant #6 still applies to retrieval. + **or any member's arcname ≥ `HTAR_ARCNAME_LEN_LIMIT` (240 chars)** → `htar_large`. + The length trigger exists because indexed `htar -L` silently **drops** members + with an overlong relpath (confirmed on Negishi: ≥250 chars omitted, rc=70, + stdout `WARNING: name too long for tar archive- file omitted`) — `htar_large` + has no such ceiling. As defense in depth (both transports, not just the length + case), `ship_object` also scans the create stdout for htar's/tar's own + member-omission messages and raises immediately (naming every dropped file + + reason) rather than falling through to the rc=70-tolerant path below or relying + on round-trip verify to eventually surface a bare `FileNotFoundError`. There is + **no htar flag that turns a skip into an abort** — checked the full `-H + opt[:opt...]` list straight from the binary (no man page on Negishi); `-H + exfile=path` looked promising but came back empty in a live test with both an + overlong-path and an unreadable member dropped, so it does not capture these. + Patterns covered: htar's `name too long for tar archive- file omitted` and + `Not readable: `, and (the one case rc offers **zero** signal for — + confirmed rc=0 on Negishi) tar's own `Cannot open: ` under `htar_large`, + since that wrapper's exit code reflects `hsi put`, not the `tar` process that + hit the error. Invariant #6 still applies to retrieval. - **Per-target manifest** at `{log_dir}/_vault/{badge}.manifest.json` — the durable cross-run state reclaim and restore rebuild the aggregate from. This is why routing requires a shared, durable `log_dir` (guard in `load_config`). @@ -398,6 +415,33 @@ archives byte-for-byte as before. Authoritative design: `docs/RFC_incremental_v2 - **htar 64 GB limit:** `htar` fails if any single file exceeds 64 GB. Use `htar_large` (no limit, but no `.idx` index file). +- **htar 250-char path limit (silent, not fail-fast):** indexed `htar -L` drops + (does not fail) any member whose relpath is ≥ 250 chars — confirmed on Negishi + 2026-07-12, real-world hit: `X0A_Task4_Pendleton_Transcriptomics/shard-0of8` + (Novogene GSEA/KEGG output, long gene-set names in paths). Unlike the 64 GB + case, htar does not pre-flight this — it emits `WARNING: name too long for tar + archive- file omitted` per dropped file and exits rc=70 (same rc as the benign + transient case, so the two must be told apart by the warning text, not rc + alone). Fixed by routing any object with an overlong arcname to `htar_large` + (`HTAR_ARCNAME_LEN_LIMIT = 240`) plus a fail-fast scan for the warning text in + `ship_object`. +- **No htar setting turns a silent skip into a hard failure — checked, not + assumed:** pulled the full `-H opt[:opt...]` sub-option list live from the + htar binary on Negishi (no man page installed); none of `crc`, `verify=`, + `exfile=`, `okfile=`, `rmlocal`, etc. converts a skip into an abort. `-H + exfile=path` ("exceptions list of files not successfully transferred") looked + like the right structural mechanism, but a live test (one overlong-path member + + one `chmod 000` member) came back with an **empty exfile** despite both + being dropped — it doesn't cover pre-flight skips like these. htar_large + (plain `tar | hsi put`) has its own silent-drop mode: a per-file `tar: : + Cannot open: ` error came back with **rc=0**, since the wrapper's exit + code reflects `hsi put`, not the `tar` process that hit the error — the one + case where rc gives no signal at all. `ship_object`'s fail-fast scan (above) + covers htar's "name too long"/"Not readable" and htar_large's/tar's "Cannot + open" — regex-scraping stdout is the only mechanism that actually works here, + confirmed by ruling out the structured alternative. Not yet reproduced live + (documented as a gap, not guessed at): a symlink whose target path is too long, + and a source file modified mid-create. - **htar_large -xvf broken in subprocess:** streams the tar via stdout; redirect breaks extraction. Fixed by `hsi get` + local `tar xvf`. - **Template-capable endpoint 409:** a template-capable endpoint's `stop` did not diff --git a/archive.py b/archive.py index 50932ec..ab9ea8e 100644 --- a/archive.py +++ b/archive.py @@ -761,6 +761,21 @@ def decode_blonde(blonde): # everything else via real htar (indexed, random-access single-member restore). HTAR_MEMBER_LIMIT = 2 ** 36 # 68,719,476,736 bytes +# The indexed `htar -L listfile` transport silently DROPS members whose relpath +# exceeds a hard length ceiling — reproduced on Negishi 2026-07-12 with synthetic +# nested paths (each directory component kept under the filesystem's 255-byte +# basename limit): every relpath <= 249 chars shipped, every relpath >= 250 chars +# was omitted (htar's own stdout: "WARNING: name too long for tar archive- file +# omitted", htar exit rc=70 — see ship_object's create-stdout scan). `htar_large` +# (plain HPSS tar, no -L listfile) has NO such ceiling: confirmed live the same +# day with an absolute Fortress destination and a 431-char relpath (create, +# retrieve via hsi get + tar xf, and per-file content all succeeded). This +# threshold is set to 240 — a ~9-char margin under the confirmed 249-char ceiling, +# enough to absorb small path-construction differences (trailing slashes, unicode +# normalization) without hugging the boundary — to force htar_large before any +# member gets anywhere near the drop threshold. +HTAR_ARCNAME_LEN_LIMIT = 240 + def clamp_t_small(t_small): """ @@ -937,16 +952,23 @@ def budget_warnings(objects, catalog, shard_target=SHARD_TARGET_DEFAULT, def choose_transport(arcnames, catalog): """ - Pick the Fortress transport for an object by its LARGEST member (RFC §2.1): - "htar" — every member < HTAR_MEMBER_LIMIT (2^36) → indexed .tar.idx, - fast listing + random-access single-member restore; - "htar_large" — some member >= HTAR_MEMBER_LIMIT (a huge solo) → plain tar, no idx. - Pure + deterministic. Shards (members < T_small) are always "htar"; only a solo - whose one file hits 64 GiB is "htar_large". A member missing from the catalog - (vanished mid-walk) is treated as 0 bytes. + Pick the Fortress transport for an object (RFC §2.1). "htar_large" (plain tar, + no .tar.idx) is forced if EITHER: + - some member's size >= HTAR_MEMBER_LIMIT (2^36, a huge solo) — htar's 64 GiB + single-file limit; or + - some member's arcname length >= HTAR_ARCNAME_LEN_LIMIT — indexed `htar -L` + silently DROPS overlong relpaths (confirmed on Negishi; see the constant's + comment), so any member near that ceiling routes the whole object away + from indexed htar rather than risk a partial, silently-incomplete tar. + Otherwise "htar" (indexed .tar.idx, fast listing + random-access restore). + Pure + deterministic. A member missing from the catalog (vanished mid-walk) is + treated as 0 bytes for the size check (its arcname length still counts). """ max_size = max((catalog.get(a, (0, 0))[1] for a in arcnames), default=0) - return "htar_large" if max_size >= HTAR_MEMBER_LIMIT else "htar" + max_name_len = max((len(a) for a in arcnames), default=0) + if max_size >= HTAR_MEMBER_LIMIT or max_name_len >= HTAR_ARCNAME_LEN_LIMIT: + return "htar_large" + return "htar" def vault_dir_for(log_dir): @@ -2605,11 +2627,12 @@ def ship_object(source_folder, arcnames, object_stem, catalog, fortress_base_dir tmp_dir, log_dir, emails, cleanup_on_success=False): """ Ship ONE routed object (Phase 2, RFC §2.5) — LOCAL only. Tars the object's members - DIRECTLY from source (no zip), choosing the transport by max member size - (choose_transport): real `htar` (indexed .idx, random-access restore) when every - member < HTAR_MEMBER_LIMIT, else `htar_large` (plain tar, no index) for a >=64 GiB - solo. Computes per-file MD5 from source, round-trip verifies every member off tape, - and writes the object's own {stem}_{ts}.txt/.json run-log pair (invariant #4). + DIRECTLY from source (no zip), choosing the transport via choose_transport: real + `htar` (indexed .idx, random-access restore) when every member is < HTAR_MEMBER_LIMIT + AND every arcname is < HTAR_ARCNAME_LEN_LIMIT, else `htar_large` (plain tar, no + index) for a >=64 GiB solo or an overlong-path member. Computes per-file MD5 from + source, round-trip verifies every member off tape, and writes the object's own + {stem}_{ts}.txt/.json run-log pair (invariant #4). Returns dict(fortress_tar, transport, idx, run_timestamp, n_files, source_bytes, tar_bytes, file_checksums, txt_path, json_path, create_warning). Raises RuntimeError @@ -2617,10 +2640,20 @@ def ship_object(source_folder, arcnames, object_stem, catalog, fortress_base_dir objects are already in the manifest). Imports stay inside the body (consistency with the other workers). - A nonzero htar/htar_large create exit code is NOT immediately fatal: HPSS can return - a transient nonzero rc under concurrent load even though the tar landed correctly - (observed on Negishi, job 40527505 — rc=70, stdout ended "HTAR SUCCESSFUL", tar - confirmed fine). On rc != 0, the full create stdout is written to + Immediately after create, the stdout is scanned for htar's/tar's own member-omission + messages ("name too long for tar archive- file omitted", "Not readable: ", or + tar's "Cannot open: " under htar_large) regardless of returncode — the last + of these has been observed to return rc=0, so the scan cannot be gated on rc. If any + are present, this raises right away naming every dropped file + reason — real bytes + are missing from the tar, so this must never be treated as the harmless-rc case below + (choose_transport's length check should already keep the "name too long" case + unreachable in normal operation; this scan is the defense-in-depth backstop for that + and the other, unrouted omission modes). + + Otherwise, a nonzero htar/htar_large create exit code is NOT immediately fatal: + HPSS can return a transient nonzero rc under concurrent load even though the tar + landed correctly (observed on Negishi, job 40527505 — rc=70, stdout ended "HTAR + SUCCESSFUL", tar confirmed fine). On rc != 0, the full create stdout is written to {stem}_{ts}.htar_create.log, then `hsi ls` checks whether the tar (+ .idx for `htar`) actually landed. If not, this raises exactly as before (genuine failure). If it did land, this proceeds to the round-trip retrieve + per-file MD5 verify below — a @@ -2634,6 +2667,7 @@ def ship_object(source_folder, arcnames, object_stem, catalog, fortress_base_dir verify _lg: hsi get -> ; tar xf ; md5 """ import os + import re import subprocess import hashlib import datetime @@ -2687,9 +2721,57 @@ def ship_object(source_folder, arcnames, object_stem, catalog, fortress_base_dir cmd = ["htar_large", "-cvf", fortress_tar] + arcnames create = subprocess.run(cmd, cwd=source_folder, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) + create_stdout = create.stdout.decode("utf-8", "replace") + + # Fail fast on an explicit member-omission message, BEFORE the rc!=0 + # fall-through below gets a chance to treat this as the benign transient-rc + # case (PR #28) — and regardless of rc at all, since one of these (see the + # htar_large/tar case below) can print rc=0. Each pattern here is htar's or + # tar's own unambiguous statement that a member was NOT written to the + # archive, i.e. those bytes are NOT on Fortress — this must never fall + # through to "maybe fine, check hsi ls + round-trip verify"; that path only + # catches it minutes later, deep in the per-file MD5 loop, as a bare + # FileNotFoundError on whichever omitted file the loop happens to reach + # first (and only reports one, even if many were dropped). + # - htar (indexed, -L listfile): "WARNING: name too long for tar + # archive- file omitted" + " []" on the next line. Should be + # unreachable now that choose_transport routes any object with an + # overlong arcname to htar_large first (HTAR_ARCNAME_LEN_LIMIT) — this + # is the defense-in-depth backstop for anything that slips past that. + # - htar (indexed): "INFO: Not readable: " — a permission/IO + # error on the source file, skipped rather than aborting the run. + # - htar_large (plain `tar | hsi put`): "tar: : Cannot open: ..." + # (or Cannot stat/Cannot read/etc.) — confirmed on Negishi to print + # rc=0 even though the member was dropped, since the wrapper's exit + # code reflects `hsi put`, not the `tar` process that hit the error. + # This is the ONLY transport where rc offers no signal at all. + # Not yet reproduced live (left undetected here rather than guessed at): + # "symlink %s: pointed-to name too long to store in archive", and a file + # modified mid-create ("... has been modified - skipped"). + omitted = ( + [(p, "name too long for tar archive") for p in re.findall( + r"WARNING: name too long for tar archive- file omitted" + r"\s*\r?\n\s*\[(.*?)\]", create_stdout)] + + [(p, "not readable (permission/IO error)") for p in re.findall( + r"(?:INFO: )?Not readable:\s*(\S+)", create_stdout)] + + [(p, f"tar: {reason.strip()}") for p, reason in re.findall( + r"tar: (.+?): Cannot \S+: (.+)", create_stdout)] + ) + if omitted: + debug_log = os.path.join(log_dir, f"{run_timestamp}.htar_create.log") + os.makedirs(log_dir, exist_ok=True) + with open(debug_log, "w") as fh: + fh.write(create_stdout) + fh.write("\n\n=== OMITTED MEMBERS (dropped from the archive) ===\n") + fh.write("\n".join(f"{p} ({reason})" for p, reason in omitted) + "\n") + raise RuntimeError( + f"{transport} create for {run_timestamp} OMITTED {len(omitted)} member(s) " + f"(rc={create.returncode}) — these bytes are NOT on Fortress: " + + "; ".join(f"{p} ({reason})" for p, reason in omitted) + + f". Full create output: {debug_log}") + create_warning = None if create.returncode != 0: - create_stdout = create.stdout.decode("utf-8", "replace") debug_log = os.path.join(log_dir, f"{run_timestamp}.htar_create.log") with open(debug_log, "w") as fh: fh.write(create_stdout) diff --git a/docs/RFC_incremental_v2.md b/docs/RFC_incremental_v2.md index 072e8fd..8f01547 100644 --- a/docs/RFC_incremental_v2.md +++ b/docs/RFC_incremental_v2.md @@ -168,6 +168,36 @@ every object is a self-describing `{stem}.tar`, and `htar -t` now lists the **re paths** instead of the old opaque `{stem}.zip`. Only the routed path changes; the whole-target Phase-1 path keeps `zip → htar_large` untouched. +**Second `htar_large` trigger — overlong member paths (Negishi, 2026-07-12).** Unlike the +size limit, indexed `htar -L` does not fail fast on an overlong relpath: it silently +**drops** the member (stdout: `WARNING: name too long for tar archive- file omitted`, +rc=70) and continues, so a naive rc check alone can't tell this apart from the harmless +transient rc=70 case (§2.5's create-retry note). Reproduced with synthetic nested paths — +every relpath ≤ 249 chars shipped, every relpath ≥ 250 chars was dropped. `htar_large` +has no such ceiling (confirmed live with a 431-char relpath: create, `hsi get` + `tar xf` +retrieve, and per-file MD5 all matched). `choose_transport` therefore also forces +`htar_large` when **any member's arcname length ≥ `HTAR_ARCNAME_LEN_LIMIT` (240)** — a +margin under the 249-char confirmed ceiling — regardless of size. `ship_object` additionally +scans the create stdout for the omission warning and raises immediately, naming every +dropped file, rather than relying solely on routing or the generic round-trip verify to +eventually surface a bare `FileNotFoundError`. + +**No htar setting converts a skip into an abort, and other silent-drop modes exist beyond +path length.** Pulled the full `-H opt[:opt...]` list live from the htar binary (no man page +on Negishi) — none of its options turn a per-file skip into a run failure. `-H exfile=path` +("exceptions list of files not successfully transferred") looked like the right structural +capture mechanism, but a live test with one overlong-path member and one `chmod 000` member +came back with an **empty exfile** despite both being dropped — it does not cover these. +`htar_large` (plain `tar | hsi put`) has its own silent-drop mode, worse than htar's: a +per-file `tar: : Cannot open: ` error came back with **rc=0**, because the +wrapper's exit code reflects `hsi put`, not the `tar` process that hit the error — the one +case rc gives zero signal for. `ship_object`'s stdout scan therefore covers three confirmed +patterns regardless of transport or rc: htar's `name too long for tar archive- file omitted`, +htar's `Not readable: ` (unreadable source file, also silently skipped), and tar's +`Cannot open: ` under `htar_large`. Two more patterns exist in the htar binary's +strings but haven't been reproduced live, so are documented as a gap rather than guessed at: +a symlink whose target path is too long, and a source file modified mid-create. + ### 2.2 Thresholds, K, and budgets (measurement-tuned; all warn-only) | Knob | Default | Basis | diff --git a/tests/test_routing_wiring.py b/tests/test_routing_wiring.py index 172eba4..f9be4ea 100644 --- a/tests/test_routing_wiring.py +++ b/tests/test_routing_wiring.py @@ -95,6 +95,27 @@ def test_max_member_decides(self): def test_missing_member_treated_as_zero(self): self.assertEqual(archive.choose_transport(["gone"], {}), "htar") + def test_long_arcname_forces_htar_large_regardless_of_size(self): + # Negishi 2026-07-12: indexed `htar -L` silently drops members whose + # relpath >= 250 chars. A small file with an overlong path must still + # route to htar_large — size alone must not decide. + long_name = "d/" * 120 + "f.txt" # well past HTAR_ARCNAME_LEN_LIMIT + self.assertGreaterEqual(len(long_name), archive.HTAR_ARCNAME_LEN_LIMIT) + c = {long_name: (1, SMALL)} + self.assertEqual(archive.choose_transport([long_name], c), "htar_large") + + def test_short_arcname_at_limit_boundary(self): + under = "a" * (archive.HTAR_ARCNAME_LEN_LIMIT - 1) + at = "a" * archive.HTAR_ARCNAME_LEN_LIMIT + c = {under: (1, SMALL), at: (1, SMALL)} + self.assertEqual(archive.choose_transport([under], c), "htar") + self.assertEqual(archive.choose_transport([at], c), "htar_large") + + def test_long_arcname_among_short_ones_decides(self): + long_name = "d/" * 120 + "f.txt" + c = {"short": (1, SMALL), long_name: (1, SMALL)} + self.assertEqual(archive.choose_transport(["short", long_name], c), "htar_large") + class TestObjectProjectName(unittest.TestCase): def test_solo_and_shard_stems(self): diff --git a/tests/test_ship_object_htar_rc.py b/tests/test_ship_object_htar_rc.py index b75fbb3..20b02bc 100644 --- a/tests/test_ship_object_htar_rc.py +++ b/tests/test_ship_object_htar_rc.py @@ -144,5 +144,160 @@ def test_zero_rc_success_path_unaffected(self): self.assertFalse(os.path.isfile(debug_log)) +class TestShipObjectNameTooLongOmission(unittest.TestCase): + """ + htar's (and tar's, under htar_large) create stdout can carry explicit, + unambiguous omission messages for a dropped member: htar's "name too long for + tar archive- file omitted" (always alongside rc=70), htar's "Not readable: + " (also indexed htar), and tar's "Cannot open: " under + htar_large — the last one confirmed on Negishi to come back with **rc=0**, + since the wrapper's exit code reflects `hsi put`, not the `tar` process that + hit the error. Unlike the OTHER rc=70 case PR #28 tolerates (a harmless + transient — no omission message present), any of these mean real member bytes + are NOT on Fortress and must fail immediately, before the rc!=0 fall-through + logic gets a chance to treat it as "maybe fine, check further" (or, for the + htar_large/rc=0 case, before it's mistaken for a clean success at all). + """ + + def setUp(self): + self.d = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, self.d, ignore_errors=True) + self.source_folder = os.path.join(self.d, "src") + os.makedirs(os.path.join(self.source_folder, "sub")) + with open(os.path.join(self.source_folder, "a.txt"), "wb") as fh: + fh.write(b"hello world") + with open(os.path.join(self.source_folder, "sub", "b.txt"), "wb") as fh: + fh.write(b"more data here") + self.arcnames = ["a.txt", "sub/b.txt"] + self.catalog = {"a.txt": (0, 11), "sub/b.txt": (0, 14)} + self.tmp_dir = os.path.join(self.d, "tmp") + self.log_dir = os.path.join(self.d, "logs") + self.fortress_base_dir = "/tape/base" + + def _ship(self, fake_run): + with mock.patch("subprocess.run", side_effect=fake_run): + return archive.ship_object( + self.source_folder, self.arcnames, "testproj__shard-0of1", + self.catalog, self.fortress_base_dir, self.tmp_dir, self.log_dir, + ["ops@example.com"]) + + def test_omission_warning_raises_immediately_and_skips_verify(self): + omitted_relpath = "sub/" + ("x" * 260) + ".txt" + stdout = ( + b"HTAR: a a.txt\n" + b"WARNING: name too long for tar archive- file omitted\n" + b" [" + omitted_relpath.encode() + b"]\n" + b"HTAR: HTAR SUCCESSFUL\n" + ) + fake_run, calls = make_fake_run( + self.source_folder, self.arcnames, create_rc=70, create_stdout=stdout, + ls_ok=lambda path: True) + + with self.assertRaises(RuntimeError) as ctx: + self._ship(fake_run) + + self.assertIn(omitted_relpath, str(ctx.exception)) + self.assertIn("too long", str(ctx.exception)) + + # Fails at the create step — hsi ls / round-trip extract must never run. + self.assertFalse(any(c[0][0] == "hsi" and c[0][1:3] == ["-q", "ls"] + for c in calls)) + self.assertFalse(any(c[0][0] == "htar" and c[0][1] == "-xvf" for c in calls)) + + # Full stdout + the omitted-member list are captured durably even though + # the run fails fast. + run_ts_glob = [f for f in os.listdir(self.log_dir) if f.endswith(".htar_create.log")] + self.assertEqual(len(run_ts_glob), 1) + with open(os.path.join(self.log_dir, run_ts_glob[0])) as fh: + debug_log = fh.read() + self.assertIn(omitted_relpath, debug_log) + self.assertIn("name too long", debug_log) + + # No success run-log written for a failed object. + self.assertEqual( + [f for f in os.listdir(self.log_dir) if f.endswith(".json")], []) + + def test_multiple_omissions_all_named(self): + omitted_a = "sub/" + ("a" * 260) + ".txt" + omitted_b = "sub/" + ("b" * 260) + ".txt" + stdout = ( + b"WARNING: name too long for tar archive- file omitted\n" + b" [" + omitted_a.encode() + b"]\n" + b"WARNING: name too long for tar archive- file omitted\n" + b" [" + omitted_b.encode() + b"]\n" + ) + fake_run, _calls = make_fake_run( + self.source_folder, self.arcnames, create_rc=70, create_stdout=stdout, + ls_ok=lambda path: True) + + with self.assertRaises(RuntimeError) as ctx: + self._ship(fake_run) + + self.assertIn(omitted_a, str(ctx.exception)) + self.assertIn(omitted_b, str(ctx.exception)) + + def test_omission_warning_with_rc_zero_still_raises(self): + # Belt-and-braces: the scan runs regardless of returncode. + omitted_relpath = "sub/" + ("z" * 260) + ".txt" + stdout = ( + b"WARNING: name too long for tar archive- file omitted\n" + b" [" + omitted_relpath.encode() + b"]\n" + b"HTAR: HTAR SUCCESSFUL\n" + ) + fake_run, calls = make_fake_run( + self.source_folder, self.arcnames, create_rc=0, create_stdout=stdout, + ls_ok=lambda path: True) + + with self.assertRaises(RuntimeError) as ctx: + self._ship(fake_run) + self.assertIn(omitted_relpath, str(ctx.exception)) + self.assertFalse(any(c[0][0] == "htar" and c[0][1] == "-xvf" for c in calls)) + + def test_not_readable_omission_raises(self): + # htar (indexed) skips unreadable source files with an INFO line rather + # than aborting — confirmed on Negishi 2026-07-12 (chmod 000 file, rc=72 + # that run, i.e. yet another rc value — reinforcing that rc must never be + # the discriminator). + stdout = ( + b"INFO: Not readable: sub/b.txt\n" + b"HTAR: a a.txt\n" + b"HTAR: HTAR SUCCESSFUL\n" + ) + fake_run, calls = make_fake_run( + self.source_folder, self.arcnames, create_rc=72, create_stdout=stdout, + ls_ok=lambda path: True) + + with self.assertRaises(RuntimeError) as ctx: + self._ship(fake_run) + + self.assertIn("sub/b.txt", str(ctx.exception)) + self.assertIn("not readable", str(ctx.exception)) + self.assertFalse(any(c[0][0] == "hsi" and c[0][1:3] == ["-q", "ls"] + for c in calls)) + + def test_htar_large_tar_cannot_open_raises_even_with_rc_zero(self): + # htar_large wraps `tar cvf - | hsi put`; when tar hits a per-file + # error it prints to stdout/stderr and exits nonzero itself, but the rc we + # see is `hsi put`'s — confirmed on Negishi to come back rc=0 even though + # tar dropped a member. This is the one case the existing rc!=0 + # fall-through can NEVER catch, so the stdout scan must run unconditionally. + stdout = ( + b"a.txt\n" + b"tar: sub/b.txt: Cannot open: Permission denied\n" + b"tar: Exiting with failure status due to previous errors\n" + ) + fake_run, calls = make_fake_run( + self.source_folder, self.arcnames, create_rc=0, create_stdout=stdout, + ls_ok=lambda path: True) + + with self.assertRaises(RuntimeError) as ctx: + self._ship(fake_run) + + self.assertIn("sub/b.txt", str(ctx.exception)) + self.assertIn("Permission denied", str(ctx.exception)) + self.assertFalse(any(c[0][0] == "hsi" and c[0][1:3] == ["-q", "ls"] + for c in calls)) + + if __name__ == "__main__": unittest.main()