Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
115 changes: 115 additions & 0 deletions docs/HOLDS_POLICY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
# Reclaim holds policy — typed holds + promote receipts

**Status:** engine side implemented (`reclaim.py`). The receipt *writer* and the Teams
review card are deployment glue (florasense-tools) and land separately.

`reclaim.py` decides which archived Depot sources are **held** vs **reclaimable**. The
flat `reclaim_holds.txt` conflated distinct reasons for holding, so holds are now **typed**
by *why* a source is held, which yields three review tiers.

## The three tiers

| Tier | `reason` (+ flag) | Verdict | Meaning |
|---|---|---|---|
| **Hold** | `business` | `HOLD` | A human/downstream needs it live (e.g. GeneWiz_X1C until Amanda relocates it). Never auto-reclaimed; `review_date` can drive a nightly nag. |
| **Held (pending receipt)** | `promote-pending`, no valid receipt | `HOLD` | A staging copy of data being promoted into `repository/`. Held until a content-coverage receipt proves the promote landed. |
| **Needs approval** | `promote-pending`, valid receipt | `AWAITING-APPROVAL` | Promotion is **proven**; the source is *eligible*, but a human approves the actual delete. This is what the Teams review card surfaces. |
| **Held (pending record)** | `superseded`, no record | `HOLD` | A corrected/reprocessed/deduped copy (DECISIONS #25). Held until a supersession provenance record exists. |
| **Needs approval** | `superseded`, valid record | `AWAITING-APPROVAL` | The supersession record exists (the "why" is preserved); *eligible*, but a human approves the delete. |

> **There is no auto-reclaim.** Reclaim never deletes on its own: it is **report-only**
> unless `--delete` is passed, and `--delete` confirms **per folder** (unless `--yes`). So
> "don't reclaim by default" is structural, not a flag that could be flipped. A valid-receipt
> promote-pending source becomes `AWAITING-APPROVAL` — eligible, but a human runs the guarded
> delete. (No `auto_reclaim` field; if unattended sweeping is ever wanted it is a deliberate
> future feature with its own review, not a dormant default.)

**Nothing un-promoted is ever reclaimed.** The receipt *is* the "has it landed in
`repository/`?" proof — no valid receipt → `HOLD`, full stop. And even an approved delete
still clears the existing hard gates below.

## `reclaim_holds.json`

```json
{
"_comment": "Typed reclaim holds. reason = business | promote-pending | superseded.",
"holds": [
{ "match": "GeneWiz_X1C", "reason": "business",
"owner": "Amanda", "review_date": "2026-12-01",
"note": "keep on Depot until relocated" },

{ "match": "landing-zone/AAPF", "reason": "promote-pending",
"receipt_dir": "/depot/florasense/etc/logs_fortress/promote_receipts",
"note": "eligible once fs-verify-promoted confirms the copy in repository/" },

{ "match": "X0H_Soy/2_spectral-standoff/hsi/mishap_original", "reason": "superseded",
"record": "/depot/florasense/data/repository/X0H_Soy/2_spectral-standoff/hsi/provenance/X0H_Soy_AAPF_mishap_PROVENANCE.md",
"note": "AAPF barcode-swap; eligible once the supersession record exists (DECISIONS #25)" }
]
}
```

- `match` — exact `project_name` **or** a substring of `source_folder` (same matching as the
old flat list).
- Legacy **`reclaim_holds.txt`** still works: each line is treated as a `business` hold, so
existing deployments are unchanged until they move to JSON.

## The receipt (content-coverage)

Written by the deployment's promote-verify tool (florasense-tools `fs-verify-promoted`) into
`receipt_dir` as JSON:

```json
{ "source_folder": "/depot/.../landing-zone/AAPF",
"file_count": 68382, "all_present": true,
"verified_at": "2026-07-09T14:30:00", "tool": "fs-verify-promoted" }
```

`fs-verify-promoted` proves `set(md5 of every landing-zone file) ⊆ set(md5 under
repository/)` (a real byte-coverage check; it must **reject symlinks** as a "promoted copy",
or a `repository/` symlink back into the landing-zone would falsely pass). `reclaim.py` is
content-agnostic — it trusts the receipt but re-checks it is **current**:
`receipt_valid()` requires `all_present` **AND** `file_count == live count` **AND** no live
file newer than `verified_at`. Any problem (no receipt, stale, count mismatch) → held.

## Hard gates that still apply to every delete

The tiering only decides eligibility; a `--delete` additionally requires **all** of:

1. the source is **archived + round-trip-verified on Fortress** (recoverable from tape);
2. not under a **canonical root** (`/depot/florasense/data/repository` — the curated tree,
never deletable, no override);
3. a **snapshot** proves recoverability — **every live file present in the newest NetApp
snapshot with matching size and mtime** (not merely a file-count ≥ check; that missed a
file *replaced* since the snapshot, same count → old bytes, and renames). Override only
with `--allow-no-snapshot`;
4. a path-depth guard (under `/depot/`, ≥ 4 levels).

## Related deployment gates (florasense-tools)

Two deployed gates wrap the engine's reclaim and this policy should be read alongside them:

- **Supersession-record gate** (florasense `DECISIONS #25` + `docs/data-supersession.md` +
`docs/templates/supersession-record-template.md`). Deleting a **superseded** copy
(corrected / re-delivered / reprocessed / deduped data — *distinct* from a routine
single-copy reclaim of a stale-but-unchanged source) requires a human-authored
**supersession record**, written in **both** the Fortress archive **and** the modality's
repository `provenance/` dir, carrying the integrity/MD5 verdict that justifies the delete.
**The engine now enforces this** via the `superseded` tier above: a matching source stays
`HOLD` until its `record` (the repository `provenance/` copy — it lives under the canonical
root, so it persists) exists, is non-empty, and is actually filled in (not a raw template).
The record's *content* (the MD5 verdict) is human judgment the engine trusts; its
*existence* is the machine-checkable gate #25 requires. The Fortress-archive copy is ensured
by the archive step (it is on tape, not re-checked here).
- **`fs-coverage-audit`** (weekly; `docs/negishi-cron-setup.md`) covers reclaim's blind spot:
reclaim only sees sources that have an archive log, so it cannot flag **never-archived /
single-copy / FAILED** experiments. The trio: coverage-audit = "never backed up",
`fs-reclaim-report` = "backup is stale", reclaim = consumes both.

## Division of labor

- **Engine (`reclaim.py`, this repo):** typed-hold parsing, the tiering + `AWAITING-APPROVAL`
verdict, the content-agnostic receipt hook, the hardened snapshot guard.
- **Deployment (florasense-tools):** `fs-verify-promoted` (writes receipts), and the
review-only Teams card (extends `fs-reclaim-report`, posts the `AWAITING-APPROVAL` +
`HOLD` tiers nightly-when-non-empty). Approve out-of-band via the guarded `--delete`.
Loading