Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
168 changes: 168 additions & 0 deletions docs/VOCABULARY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,168 @@
# VOCABULARY — fortress-archive

Shared definitions for this project. Where **Bastion** (Nathan's sibling backup
tool) already named a concept, we adopt his term so the two tools speak the same
language — marked **(Bastion)**. Terms marked *(proposed)* are designed but not yet
built in the engine.

**Conventions settled (collisions we deliberately resolved):**
- **"Zone" is a policy/grouping concept only.** How an asset's files become archive
objects (**solo / bundle / shard**) is a separate, per-asset concern and is *never*
called a zone.
- **"asset"** is the canonical word for our unit of archiving. ("target" was the old
informal name; don't use it.)
- **"drift" is two different things, kept distinct:** the *policy* key **drift**
(staleness before a full is forced) vs. **source drift** (the live source changed
since it was archived — a reclaim concept).
- **"genus"** means a backup *lineage* (Bastion's meaning), not a cosmetic tag.

> Note on the code: the engine config still uses the keys `project_name`,
> `source_folder`, `file_pattern` — together they *define one asset*. The code has
> not been renamed to this vocabulary yet; these terms are the conceptual model.

---

## Hierarchy & policy — adopted from Bastion: Site → Zone → Asset

Policy is set at any level and **inherited downward, overridable at the level below**.

- **Site** (Bastion) — a top-level system/host. The outermost grouping.
- **Zone / Resource Zone** (Bastion) — a named grouping of assets under a site, with
a filesystem **root** path. A zone is a **policy container + path mount point**
(e.g. `landing-zone` vs `repository`). It does **NOT** affect how files are bundled
— packing is per-asset. It exists so a set of assets can share policy and a root.
- **Asset** (Bastion) — one logical dataset that gets backed up: **our unit of
archiving**. In the engine an asset is defined by one config
(`project_name` + `source_folder` + `file_pattern`). One config = one asset.

**Policy keys** (settable at site/zone/asset; inherit downward; any level overrides):

| Key | Controls | Unit / default |
|---|---|---|
| **history** | backup generations to retain before releasing an old one | rotations / 2 |
| **drift** | staleness before a fresh FULL (anchor) is forced instead of a differential | levels ≈ days / 2 |
| **cadence** | the *intended* backup interval — **informational**: the external scheduler reads it and invokes the backup; the tool runs no clock of its own | days / 1 |
| **vault** | which storage backend to write to | name / "fortress" |

So **what a zone affects** = these four, inherited by every asset under it.
`landing-zone` vs `repository` are two zones with different
`history` / `drift` / `cadence` / `vault`.

---

## Asset addressing & identity — adopted from Bastion (full id set)

- **ARK** — *Asset Resource Key*: the canonical identity of an asset, the triple
`(site, zone, asset)`. Everything about an asset keys off its ARK.
- **CURIE** — *Compact URI*: the textual form of an ARK, `[@site:zone/asset]`.
- **badge** (Bastion: *Slug40*) — an 8-char base32 hash (SHAKE128) of the ARK's
CURIE. The stable, compact asset key — names the manifest and groups an asset's
archives. Same asset → same badge across runs.
- **RDN** — *Relatively Distinguishing Name*: an optional short (≈8-char)
human-friendly label for an asset; defaults to the badge when unset.
- **HALO** — *Host Area Location*: the asset's absolute path on the live filesystem
(`zone.root / asset`). Where the source data lives (our `source_folder`).
- **ZOLO** — *Zone Location*: the asset's path *relative to its zone root* — the
portable, host-independent locator.
- **vault** — a storage backend + protocol an asset's archives are written to
(HPSS/Fortress, SFTP, local copy…). **Fortress is one vault.**

### Snapshot identity

- **BLONDE** (Bastion) — *BLOb Name and Description Encoded*: a compact id for ONE
backup snapshot of an asset: `badge + Quantim + detail + genus`. The Quantim
portion is the 8-char form (`str(Quantim)`).
- **Quantim** (Bastion) — *Quantized Time*: a compact timestamp encoding a date plus
the time-of-day **rounded to ~67-second buckets** (1,296 per day, two base-36
digits) — i.e. ~1-minute precision *by deliberate design* (ample for a backup
stamp; you never snapshot the same asset twice in a minute) in only 7–8 chars.
`str(Quantim)` = `YYYMDDQQ` (8 chars, 3-digit year, valid to 2999) — what BLONDE embeds.
- **quaver** (Bastion) — a Quantim property: the 7-char 2-digit-year form (`YYMDDQQ`,
valid 2000–2099). "QUAntim VERsioning stamp."
- **quake** (Bastion) — the 8-char 3-digit-year form (valid to 2999); what Quantim
uses for ordering/comparison. We use this (3-digit) form so the id is good past 2099.
- **detail** — the BLONDE's backup-level character: full (anchor) vs differential level.
- **genus** (Bastion) — the **lineage id** of a backup thread: an anchor and all the
differentials taken against it share one genus. Identifies which chain a snapshot
belongs to (used by the deferred incremental design to tie differentials to their full).

---

## Fortress / HPSS storage (one vault)

- **Fortress** — Purdue RCAC's HPSS tape archive; our default vault.
- **hsi** — the only command-line client for Fortress (no daemon, no root).
- **htar / htar_large** — write a tar directly onto Fortress. `htar` also writes a
`.idx` index but fails on any single member > 64 GB; `htar_large` has no size limit
but writes no `.idx`. We use `htar_large`.
- **Object** — one independently-stored thing on Fortress (one tar, or one solo file).
Fortress dislikes *many* objects (per-object overhead); keeping the count low is a
core constraint.
- **Class of Service (COS)** — Fortress's per-file storage policy, auto-picked by
size: **COS 10** = small (< 100 MB, disk-fronted), **COS 11** = large (≥ 100 MB, tape).
- **Disk cache / tape / migration** — files < 100 MB keep their primary copy on disk;
≥ 100 MB go to tape; small files migrate disk → tape (durable, not exposed).
- **.idx index** — the table-of-contents `htar` writes beside a tar. `htar_large`
omits it; we reconstruct the equivalent in our own manifest.
- **hsi checksum** — `hsi put -c on -H md5` + `hashlist` / `hashverify`: Fortress
stores and verifies an MD5 server-side (confirmed working) — integrity check
without downloading the archive.

---

## Packing — how ONE asset's files become objects (per-asset; NOT a zone) *(proposed)*

The size-routing model. Happens within a single asset, independent of zone.

- **Small file** — below the threshold; always bundled. (Fortress calls < 30–50 MB
"small"; the disk/tape line is 100 MB.)
- **Solo / large file** — at/above the threshold; ships as its own object
(single-file restore; skipped on later runs if unchanged).
- **T_small** — the bundle-vs-solo size cutoff; floor 100 MB (Fortress's disk/tape
line), default in the 100–256 MB band; per-asset configurable.
- **Bundle** — a tar/zip of many small files shipped as one object.
- **Shard** — a bundle when there's more than one. A file's shard is
*content-addressed*: `shard = hash(relative_path) mod K`, so adding/changing one
file touches exactly one shard (no repacking cascade). Solo = the one-file-per-shard limit.
- **Lane** — which path a file takes: the **solo lane** or the **bundle lane**.
- **Skip-unchanged** — a size+mtime comparison to the last run; unchanged solo files
and unchanged shards are not re-shipped. Blind spot: a rewrite preserving *both*
size and mtime is missed.
- **Object budget** — an optional cap on objects per asset, with a warning/stop.
- **Data shape** — a named archetype of an asset's size/count/churn distribution
(mixed, many-medium, single-huge, …) that selects default packing rules.

---

## Incremental / leveled model *(designed, DEFERRED)*

- **Anchor / full** (Bastion: *anchor*) — a complete snapshot. A new anchor is forced
when elapsed ≥ policy **drift**.
- **Differential / delta** (Bastion: *amend*) — only what changed since the anchor.
Differentials off one anchor share a **genus**.
- **history** — retain N generations before releasing one (the retain-N policy key).
- **re-anchor** — start a fresh full when drift is exceeded.
- Status: on the shelf; the packing model + skip-unchanged handles the common case.

---

## Reclamation

- **Reclaim** — free Depot source space by deleting data that's provably, currently
archived on Fortress.
- **Source drift** — the live source no longer matching what was archived
(count/size/mtime changed); reclaim refuses to delete a source-drifted asset.
(Distinct from the policy key **drift**.)
- **Holds** — assets that are archived + safe but must NOT be deleted (a downstream
consumer still needs them).
- **Snapshot / path guard** — pre-delete safety checks (a filesystem snapshot covers
every live file; the path is deep under `/depot`).
- **Generation** — one archived version of an asset over time; **history** keeps N.

---

## Verbs (Bastion)

- **Push** — back up an asset to its vault (full or differential).
- **Pull** — retrieve a backup (a BLONDE) from the vault. = our **restore**.
- **Amend** — make a differential backup off the most recent lineage (genus).