diff --git a/docs/VOCABULARY.md b/docs/VOCABULARY.md new file mode 100644 index 0000000..23ef2f5 --- /dev/null +++ b/docs/VOCABULARY.md @@ -0,0 +1,168 @@ +# VOCABULARY — fortress-archive + +Shared definitions for this project. Where **Bastion** (Nathan's sibling backup +tool) already named a concept, we adopt his term so the two tools speak the same +language — marked **(Bastion)**. Terms marked *(proposed)* are designed but not yet +built in the engine. + +**Conventions settled (collisions we deliberately resolved):** +- **"Zone" is a policy/grouping concept only.** How an asset's files become archive + objects (**solo / bundle / shard**) is a separate, per-asset concern and is *never* + called a zone. +- **"asset"** is the canonical word for our unit of archiving. ("target" was the old + informal name; don't use it.) +- **"drift" is two different things, kept distinct:** the *policy* key **drift** + (staleness before a full is forced) vs. **source drift** (the live source changed + since it was archived — a reclaim concept). +- **"genus"** means a backup *lineage* (Bastion's meaning), not a cosmetic tag. + +> Note on the code: the engine config still uses the keys `project_name`, +> `source_folder`, `file_pattern` — together they *define one asset*. The code has +> not been renamed to this vocabulary yet; these terms are the conceptual model. + +--- + +## Hierarchy & policy — adopted from Bastion: Site → Zone → Asset + +Policy is set at any level and **inherited downward, overridable at the level below**. + +- **Site** (Bastion) — a top-level system/host. The outermost grouping. +- **Zone / Resource Zone** (Bastion) — a named grouping of assets under a site, with + a filesystem **root** path. A zone is a **policy container + path mount point** + (e.g. `landing-zone` vs `repository`). It does **NOT** affect how files are bundled + — packing is per-asset. It exists so a set of assets can share policy and a root. +- **Asset** (Bastion) — one logical dataset that gets backed up: **our unit of + archiving**. In the engine an asset is defined by one config + (`project_name` + `source_folder` + `file_pattern`). One config = one asset. + +**Policy keys** (settable at site/zone/asset; inherit downward; any level overrides): + +| Key | Controls | Unit / default | +|---|---|---| +| **history** | backup generations to retain before releasing an old one | rotations / 2 | +| **drift** | staleness before a fresh FULL (anchor) is forced instead of a differential | levels ≈ days / 2 | +| **cadence** | the *intended* backup interval — **informational**: the external scheduler reads it and invokes the backup; the tool runs no clock of its own | days / 1 | +| **vault** | which storage backend to write to | name / "fortress" | + +So **what a zone affects** = these four, inherited by every asset under it. +`landing-zone` vs `repository` are two zones with different +`history` / `drift` / `cadence` / `vault`. + +--- + +## Asset addressing & identity — adopted from Bastion (full id set) + +- **ARK** — *Asset Resource Key*: the canonical identity of an asset, the triple + `(site, zone, asset)`. Everything about an asset keys off its ARK. +- **CURIE** — *Compact URI*: the textual form of an ARK, `[@site:zone/asset]`. +- **badge** (Bastion: *Slug40*) — an 8-char base32 hash (SHAKE128) of the ARK's + CURIE. The stable, compact asset key — names the manifest and groups an asset's + archives. Same asset → same badge across runs. +- **RDN** — *Relatively Distinguishing Name*: an optional short (≈8-char) + human-friendly label for an asset; defaults to the badge when unset. +- **HALO** — *Host Area Location*: the asset's absolute path on the live filesystem + (`zone.root / asset`). Where the source data lives (our `source_folder`). +- **ZOLO** — *Zone Location*: the asset's path *relative to its zone root* — the + portable, host-independent locator. +- **vault** — a storage backend + protocol an asset's archives are written to + (HPSS/Fortress, SFTP, local copy…). **Fortress is one vault.** + +### Snapshot identity + +- **BLONDE** (Bastion) — *BLOb Name and Description Encoded*: a compact id for ONE + backup snapshot of an asset: `badge + Quantim + detail + genus`. The Quantim + portion is the 8-char form (`str(Quantim)`). +- **Quantim** (Bastion) — *Quantized Time*: a compact timestamp encoding a date plus + the time-of-day **rounded to ~67-second buckets** (1,296 per day, two base-36 + digits) — i.e. ~1-minute precision *by deliberate design* (ample for a backup + stamp; you never snapshot the same asset twice in a minute) in only 7–8 chars. + `str(Quantim)` = `YYYMDDQQ` (8 chars, 3-digit year, valid to 2999) — what BLONDE embeds. +- **quaver** (Bastion) — a Quantim property: the 7-char 2-digit-year form (`YYMDDQQ`, + valid 2000–2099). "QUAntim VERsioning stamp." +- **quake** (Bastion) — the 8-char 3-digit-year form (valid to 2999); what Quantim + uses for ordering/comparison. We use this (3-digit) form so the id is good past 2099. +- **detail** — the BLONDE's backup-level character: full (anchor) vs differential level. +- **genus** (Bastion) — the **lineage id** of a backup thread: an anchor and all the + differentials taken against it share one genus. Identifies which chain a snapshot + belongs to (used by the deferred incremental design to tie differentials to their full). + +--- + +## Fortress / HPSS storage (one vault) + +- **Fortress** — Purdue RCAC's HPSS tape archive; our default vault. +- **hsi** — the only command-line client for Fortress (no daemon, no root). +- **htar / htar_large** — write a tar directly onto Fortress. `htar` also writes a + `.idx` index but fails on any single member > 64 GB; `htar_large` has no size limit + but writes no `.idx`. We use `htar_large`. +- **Object** — one independently-stored thing on Fortress (one tar, or one solo file). + Fortress dislikes *many* objects (per-object overhead); keeping the count low is a + core constraint. +- **Class of Service (COS)** — Fortress's per-file storage policy, auto-picked by + size: **COS 10** = small (< 100 MB, disk-fronted), **COS 11** = large (≥ 100 MB, tape). +- **Disk cache / tape / migration** — files < 100 MB keep their primary copy on disk; + ≥ 100 MB go to tape; small files migrate disk → tape (durable, not exposed). +- **.idx index** — the table-of-contents `htar` writes beside a tar. `htar_large` + omits it; we reconstruct the equivalent in our own manifest. +- **hsi checksum** — `hsi put -c on -H md5` + `hashlist` / `hashverify`: Fortress + stores and verifies an MD5 server-side (confirmed working) — integrity check + without downloading the archive. + +--- + +## Packing — how ONE asset's files become objects (per-asset; NOT a zone) *(proposed)* + +The size-routing model. Happens within a single asset, independent of zone. + +- **Small file** — below the threshold; always bundled. (Fortress calls < 30–50 MB + "small"; the disk/tape line is 100 MB.) +- **Solo / large file** — at/above the threshold; ships as its own object + (single-file restore; skipped on later runs if unchanged). +- **T_small** — the bundle-vs-solo size cutoff; floor 100 MB (Fortress's disk/tape + line), default in the 100–256 MB band; per-asset configurable. +- **Bundle** — a tar/zip of many small files shipped as one object. +- **Shard** — a bundle when there's more than one. A file's shard is + *content-addressed*: `shard = hash(relative_path) mod K`, so adding/changing one + file touches exactly one shard (no repacking cascade). Solo = the one-file-per-shard limit. +- **Lane** — which path a file takes: the **solo lane** or the **bundle lane**. +- **Skip-unchanged** — a size+mtime comparison to the last run; unchanged solo files + and unchanged shards are not re-shipped. Blind spot: a rewrite preserving *both* + size and mtime is missed. +- **Object budget** — an optional cap on objects per asset, with a warning/stop. +- **Data shape** — a named archetype of an asset's size/count/churn distribution + (mixed, many-medium, single-huge, …) that selects default packing rules. + +--- + +## Incremental / leveled model *(designed, DEFERRED)* + +- **Anchor / full** (Bastion: *anchor*) — a complete snapshot. A new anchor is forced + when elapsed ≥ policy **drift**. +- **Differential / delta** (Bastion: *amend*) — only what changed since the anchor. + Differentials off one anchor share a **genus**. +- **history** — retain N generations before releasing one (the retain-N policy key). +- **re-anchor** — start a fresh full when drift is exceeded. +- Status: on the shelf; the packing model + skip-unchanged handles the common case. + +--- + +## Reclamation + +- **Reclaim** — free Depot source space by deleting data that's provably, currently + archived on Fortress. +- **Source drift** — the live source no longer matching what was archived + (count/size/mtime changed); reclaim refuses to delete a source-drifted asset. + (Distinct from the policy key **drift**.) +- **Holds** — assets that are archived + safe but must NOT be deleted (a downstream + consumer still needs them). +- **Snapshot / path guard** — pre-delete safety checks (a filesystem snapshot covers + every live file; the path is deep under `/depot`). +- **Generation** — one archived version of an asset over time; **history** keeps N. + +--- + +## Verbs (Bastion) + +- **Push** — back up an asset to its vault (full or differential). +- **Pull** — retrieve a backup (a BLONDE) from the vault. = our **restore**. +- **Amend** — make a differential backup off the most recent lineage (genus).