Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -386,6 +386,30 @@ archives byte-for-byte as before. Authoritative design: `docs/RFC_incremental_v2
confirmed rc=0 on Negishi) tar's own `Cannot open: <reason>` under `htar_large`,
since that wrapper's exit code reflects `hsi put`, not the `tar` process that
hit the error. Invariant #6 still applies to retrieval.
- **This isn't theoretical for the two areas that matter — audited 2026-07-12.**
The two areas actually required to be backed up are `data/repository` and
`data/landing-zone` (legacy per-experiment archives like the old flat X0A–F
layout are being consolidated out of scope separately). A direct relpath-length
scan of both, done to settle whether they're exposed rather than waiting for a
run to fail:
- `data/repository` (46 TB, ~19 categories): **max relpath 237 chars** —
`X0F/4_omics-analysis/.../10_KEGG_enrichment/.../gmx04141...absMaxFC.png` —
only **3 characters under `HTAR_ARCNAME_LEN_LIMIT` (240)**. 13,408 files are
already ≥200 chars. This is real, in-scope production data, not the
deprecated X0A raw-delivery tree that originally broke — same
KEGG/pathway-analysis naming family, just reorganized slightly shorter by
repository's ingest. Nothing has crossed 240 yet, so nothing has silently
dropped here, but the margin is thin enough that the next omics-analysis
pipeline run with a slightly longer comparison name plausibly could without
this fix.
- `data/landing-zone` (Metabolomics + inboxes): max relpath 218 chars, 46,870
files ≥200 chars. Same long-imaging-pipeline-name style, wider (22-char)
margin than repository's.
- Cross-checked against every historical run log: only the legacy
`X0A_Task4_Pendleton_Transcriptomics` target has ever actually hit "name too
long" — repository and landing-zone runs have not silently lost files to
date. The margin found here is why that's true, not evidence it's not a
real risk.
- **Per-target manifest** at `{log_dir}/_vault/{badge}.manifest.json` — the
durable cross-run state reclaim and restore rebuild the aggregate from. This is
why routing requires a shared, durable `log_dir` (guard in `load_config`).
Expand Down
14 changes: 14 additions & 0 deletions docs/RFC_incremental_v2.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,20 @@ htar's `Not readable: <relpath>` (unreadable source file, also silently skipped)
strings but haven't been reproduced live, so are documented as a gap rather than guessed at:
a symlink whose target path is too long, and a source file modified mid-create.

**Exposure audit (2026-07-12), against the areas actually in scope — `data/repository` and
`data/landing-zone`, not the legacy per-experiment X0A–F layout being consolidated
separately.** A direct relpath-length scan (not waiting for a real run to fail) found
`data/repository` (46 TB, ~19 categories) has a max relpath of **237 chars** — 3 characters
under `HTAR_ARCNAME_LEN_LIMIT` — in `X0F/4_omics-analysis/.../10_KEGG_enrichment/...`, the
same pathway-analysis naming family that broke the legacy X0A archive, just reorganized
slightly shorter by repository's ingest; 13,408 files are already ≥200 chars.
`data/landing-zone` maxes at 218 chars (46,870 files ≥200), a wider margin. Neither has
crossed 240 yet — consistent with every historical run log showing only the legacy
`X0A_Task4_Pendleton_Transcriptomics` target ever actually hitting "name too long" — but the
3-char margin on repository means this was close to being a live incident, not a
hypothetical one, and the next omics pipeline run with slightly longer comparison names
could cross it without this fix in place.

### 2.2 Thresholds, K, and budgets (measurement-tuned; all warn-only)

| Knob | Default | Basis |
Expand Down