docs: RFC — staged-hybrid incremental backup (Phase 2 = size-routing/append) #7

Jarrod S Doucette (jdoucett) · 2026-06-26T17:40:02Z

Refreshed: the staged-hybrid RFC (review gate for Phase 2)

This rewrites docs/RFC_incremental_v2.md from the original leveled-only RFC into the
umbrella staged-hybrid design — matching what actually shipped (Phase 1) and what is
being built next (Phase 2). This PR is a review gate: no engine code lands until the
Phase-2 design here is approved.

What changed vs the old RFC

Phase 1 — Exclusion [DONE]: recap + pointer to the authoritative docs/EXCLUSION_SPEC.md (merged, PRs feat(incremental): catalog + BLONDE primitives (PR 1) #8/docs: VOCABULARY.md — shared terms aligned to Bastion #10–feat(exclusion): resume-under-exclusion + post-hoc re-verifier (finish Phase 1) #17, suite 136 green).
Phase 2 — Size-routing / Append [THIS RFC — READY TO BUILD]: the substantive new content.
Phase 3 — Leveled L0/L1/L2 [DEFERRED — DORMANT]: the entire prior leveled RFC is preserved intact under a DEFERRED banner (§3) as the Phase-3 reference — Path A/B recommendation, restore, chain-aware reclaim, the Negishi spike. Nothing was lost.
Appendix: the PR feat(incremental): catalog + BLONDE primitives (PR 1) #8 primitives (badge_of / make_blonde / enumerate_source_catalog / floor_s) reused across phases, with current archive.py line refs.

Phase 2 in one screen

Per target, the post-exclusion survivors are partitioned into independent Fortress
objects, each shipped through the unchanged make_zip_files → send_to_fortress
verify chain:

Solo — one file ≥ T_small (256 MiB, floor 100 MB); its own object; skips if unchanged.
Shard — files < T_small bucketed by k = hash(relpath) mod K (K frozen at baseline); whole-shard re-ship on any member change. No repack cascade.
The append win: a growing target's new big files become new solos; every existing object skips — zero re-read of already-archived bytes. Skip-unchanged also doubles as resume (a half-shipped object simply isn't in the manifest yet).
Per-target AGGREGATE reclaim (verify_target) — REQUIRED in the same slice as the first routed ship, or every routed target perpetually false-DRIFTs (reclaim dedups to the newest log = one object). Sums all objects; requires every object tar present; INV-E exclusion re-gate on the aggregate.
Reship-monitor — [RESHIP-WATCH] email when shard reship ratio > ~25%: the uniform-medium-churn signature and the only trigger to activate Phase 3.
Full restore (§2.9) is the capstone slice — object-model reassembly (disjoint union, so simpler than Phase-3 leveled restore: no overlay/deletion replay), completeness gate + per-object MD5 verify + mandatory reconciliation, --slot selective restore. It's what makes aggregate --delete trustworthy.
Local-mode only — refuses --globus (routing + manifest are on the launching host).
Manifest per target at {log_dir}/_vault/{badge}.manifest.json (dodges reclaim's non-recursive glob).

Design driver (settled, do not re-litigate)

The 2026-07-08 source_files measurement (809 logs, Negishi): standoff imagery is
stable (Phase-1 exclusion covers it); the data modalities GROW by appending
(modified≈0) — so append is the dominant Phase-2 win, which content-addressed
shards + solos deliver for free. Corrects the earlier du-based "omics re-analyzes
everything" read.

Build order (after approval) — same cadence as exclusion

Routing primitives (behavior-neutral) + only_arcnames on make_zip_files.
Archive wiring (the append engine) + config keys + --globus refusal + --fresh re-baseline.
Aggregate reclaim (verify_target, required with slice 2's first routed ship).
Reship-monitor.
Full restore (restore.py) — the capstone.

Tests ship with every slice; python3 -m unittest discover -s tests before merging.

🤖 Generated with Claude Code

…uting/append) Rewrites docs/RFC_incremental_v2.md from the leveled-only RFC into the umbrella staged-hybrid RFC that matches what actually shipped and what is being built next: - Phase 1 (exclusion): DONE/merged; recap + pointer to docs/EXCLUSION_SPEC.md. - Phase 2 (size-routing/APPEND): the new build target, full design — solo objects >= T_small (256 MiB, floor 100 MB), content-addressed shards (hash(relpath) mod K, K frozen at baseline), skip-unchanged (the append win, doubles as resume), per-target AGGREGATE reclaim (verify_target — REQUIRED same slice or every routed target false-DRIFTs), reship-monitor (the Phase-3 trigger), local-mode only (refuses --globus). 4 reviewable slices with tests. - Phase 3 (leveled L0/L1/L2): DEFERRED-DORMANT banner; the prior leveled RFC content (Path A/B, restore, chain-aware reclaim, Negishi spike) preserved intact as the Phase-3 reference. - Appendix: the PR #8 primitives (badge_of/make_blonde/enumerate_source_catalog/ floor_s) reused across phases, with current line refs. Design driver = the 2026-07-08 source_files measurement: standoff imagery is stable (exclusion covers it); the data modalities GROW by appending -> append is the dominant Phase-2 win. Review gate before any engine code. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…pend, all 5 slices [DONE]) Rebased as a doc-only diff on current main (the old branch carried a stale slice-1 code sync that conflicted with the merged engine). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… re-ship loop) ship_object's post-create round-trip verify retrieve shared the create-side HTAR_STALL_SECONDS (2h) silence budget. A create streams per-file progress, so silence there is a genuine hang; a retrieve is silent BY DESIGN while HPSS queues and stages the tape. On a large routed shard the shared bound was killing healthy retrieves — and because verify never passed, ship_object wrote no success record, so the wrapper re-shipped the whole object on every scheduled run. Observed (repository_X1D_3_metabolomics_rawspectra, one 6,624,961,024-byte shard): htar -cvf succeeded every run; the retrieve was killed at exactly 7200s on both attempts (~8h/job, exit 1), and its own stall log records "activity resumed after 7200.1s silence" — 0.1s past the kill line. Result: 48 identical 6.62 GB tars (~318 GB of duplicate tape) between 2026-07-17 and 2026-08-03. - new stall_defaults.json holds the default retrieve budget (21600s = 6h, ~3x the one observed gap, well inside the 48h Slurm walltime even if both retrieve attempts burn it). Defaults live in DATA, not in a function signature. - load_stall_defaults() fails loud on a missing/malformed/invalid file; load_config reads it LAZILY so a config that pins retrieve_stall_seconds needs no defaults file, and validates a per-asset override. - ship_object takes retrieve_stall_seconds and REQUIRES it (raises before any htar/hsi call) — no code-side fallback that could silently reinstate 2h. - create keeps HTAR_STALL_SECONDS, documented at both constants and in CLAUDE.md invariant #7. Tests: new tests/test_retrieve_stall_config.py (defaults-file contract incl. the default must exceed the create bound, load_config default/override/validation, the retrieve calls carry the configured bound while the create does not, and the absent/invalid-value raises land before any tape work). Existing ship_object callers in three test files updated. Full suite: 423 passed. Note: this does NOT by itself stop a re-ship loop — a repeatedly-failing verify still re-ships. The per-target circuit breaker for that lives in florasense-tools (fortress/repo_backup.py), shipped alongside this. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A new, separate class of failure has been hitting many repository targets since 2026-07-31 (the day after a Duke Power outage took Fortress offline 07-30/31): htar_large create calls now sometimes queue at HPSS for well over an hour before moving a single byte, occasionally exceeding the hardcoded 2h HTAR_STALL_SECONDS bound and getting killed as a false-positive hang. Confirmed live on repository_X0E_2_spectral-standoff (job 41943831, 2026-08-12): a create sat silent for 4260.7s (71 min), then resumed and completed normally on its own - proving this is legitimate (if degraded) HPSS queueing, not a dead connection, the same "HPSS queues and stages" phenomenon that justified giving the round-trip retrieve its own retrieve_stall_seconds budget. Since 07-31, roughly a dozen repository targets (X0C, X0E, X0F, X1A-X1G, ...) have had a create genuinely killed at exactly "no output for 2.0h" with zero prior progress. Simply raising HTAR_STALL_SECONDS would reopen the original 32h-hang risk this whole stall-watch mechanism exists to catch (a create that HAS started streaming per-file progress going silent again really is a hang). Instead, both stall-watch functions (_run_with_stall_watch, and send_to_fortress's self-contained inline run_watched() twin, invariant #1) now take a second budget, `initial_stall_seconds`, that applies ONLY before the very first byte of output/growth is observed; once any activity has been seen, the bound tightens back to the original stall_seconds for the rest of the call. Passing only stall_seconds (every existing caller) preserves today's exact single-bound behavior. Both create call sites (ship_object, send_to_fortress) now pass initial_stall_seconds=retrieve_stall_seconds - reusing the existing config knob rather than adding a third one, since the underlying phenomenon (HPSS taking a while to start moving data) is the same regardless of read or write direction. Retrieve call sites are unchanged (they already use retrieve_stall_seconds for their whole duration). Found and fixed a real bug in the file-growth mode of both stall-watch functions while building this: a watched redirect file is created/truncated by the shell almost immediately on startup, so the FIRST observation of a path (going from "never seen" to "seen, size 0") was being treated as "activity" even though nothing had actually been written - silently flipping the phase to the tight bound within milliseconds of every call and completely defeating the two-phase design for anything using file-growth mode (send_to_fortress's create, and any other watch_paths=[...] caller). Fixed by only counting real growth (a size that differs from a previously recorded baseline) as activity; a size first observed as 0 just establishes the baseline. Caught by a new direct regression test (test_freshly_created_empty_file_is_not_mistaken_for_activity) before this reached production. Docs updated (CLAUDE.md invariant #7, stall_defaults.json, config.example.json, and the affected docstrings) to describe the three silence regimes: retrieve (whole call, retrieve_stall_seconds), create before first byte (retrieve_stall_seconds), create after first byte (HTAR_STALL_SECONDS). Tests: new two-phase coverage in test_stall_detection.py (PIPE and file-growth modes: initial silence survives within budget, post-activity silence still killed fast, default behavior unchanged, the empty-file regression above); updated test_retrieve_stall_config.py's _CapturingStallWatch and ship_object create-call assertions to also capture/check initial_stall_seconds; updated test_ship_object_retrieve_retry.py's _FastStallWatch to accept the new kwarg; replaced test_send_to_fortress_retrieve_stall.py's now-outdated single-phase create test with two tests matching the new two-phase contract. Full suite: 435 passed (428 before this change + 7 new), no regressions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Jarrod S Doucette (jdoucett) mentioned this pull request Jun 29, 2026

docs: VOCABULARY.md — shared terms aligned to Bastion #10

Merged

Jarrod S Doucette (jdoucett) force-pushed the docs/incremental-rfc-v2 branch from 87238b4 to 34ad875 Compare July 8, 2026 13:13

Jarrod S Doucette (jdoucett) changed the title ~~docs: RFC — in-engine leveled incremental backup (v2)~~ docs: RFC — staged-hybrid incremental backup (Phase 2 = size-routing/append) Jul 8, 2026

Jarrod S Doucette (jdoucett) force-pushed the docs/incremental-rfc-v2 branch from 34ad875 to b617c58 Compare July 8, 2026 13:14

Jarrod S Doucette (jdoucett) mentioned this pull request Jul 8, 2026

feat(routing): Phase-2 size-routing primitives (slice 1, behavior-neutral) #18

Merged

Jarrod S Doucette (jdoucett) force-pushed the docs/incremental-rfc-v2 branch from e65ed39 to b72c9b3 Compare July 10, 2026 14:18

Jarrod S Doucette (jdoucett) merged commit e9448ac into main Jul 10, 2026

Jarrod S Doucette (jdoucett) deleted the docs/incremental-rfc-v2 branch July 10, 2026 14:23

Jarrod S Doucette (jdoucett) mentioned this pull request Jul 18, 2026

fix(archive): kill stalled hsi/htar subprocess calls instead of hanging #32

Merged

3 tasks

Jarrod S Doucette (jdoucett) mentioned this pull request Aug 4, 2026

fix(stall): bound the verify retrieve separately from the create (X1D re-ship loop) #37

Merged

Jarrod S Doucette (jdoucett) mentioned this pull request Aug 12, 2026

fix(stall): give create's initial HPSS silence a two-phase budget #39

Open

5 tasks

Uh oh!

docs: RFC — staged-hybrid incremental backup (Phase 2 = size-routing/append) #7

docs: RFC — staged-hybrid incremental backup (Phase 2 = size-routing/append) #7

Jarrod S Doucette (jdoucett) commented Jun 26, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

docs: RFC — staged-hybrid incremental backup (Phase 2 = size-routing/append) #7

docs: RFC — staged-hybrid incremental backup (Phase 2 = size-routing/append) #7

Conversation

Jarrod S Doucette (jdoucett) commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Refreshed: the staged-hybrid RFC (review gate for Phase 2)

What changed vs the old RFC

Phase 2 in one screen

Design driver (settled, do not re-litigate)

Build order (after approval) — same cadence as exclusion

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Jarrod S Doucette (jdoucett) commented Jun 26, 2026 •

edited

Loading