Feat/glance production ingestion

PRODUCTION_GLANCE_INGESTION_AND_ML_REQUIREMENTS.md

@@ -649,3 +649,134 @@ maintenance operation: it reads the authoritative project plus global Postgres
 history, disables the CSV fallback, fits against the current snapshot instead
 of reusing a stale active model, and fails the durable follow-up when no
 proposal batch can be produced.
+
+## 16. Tool 3 production activation record
+
+On 2026-08-06 the Tool 3 direct-PostgreSQL connector was activated for the
+registered equipment
+`updated_plasmatherm_versaline_deep_silicon_etcher` and the private project
+`glance_db3_dse_paul_rich_pilot_705fa109`. The source tool was verified directly
+as `VLN-11304-CTC-PM1` (`idtools=3`). The mapping is stored only in the Geddes
+Secret; no credential or deployable mapping JSON is committed to Git.
+
+The initial image exposed an entry-point defect: launching
+`scripts/run_glance_poll.py` as a file did not place the sibling `common`
+package on Python's import path. The container now launches
+`python -m scripts.run_glance_poll` from `/app/azure`. The first failed Job was
+stopped by its bounded retry policy before it could query GLANCE or write DT
+data.
+
+The activation gates produced the following evidence:
+
+- Approved source run `1` created one DT identity with 77 distinct samples and
+  7 distinct events. Its acknowledged cursor advanced from 0 to 1.
+- An identical run-1 replay returned `no_data`, proving the pilot cursor was
+  durable and did not create a duplicate.
+- The complete 13-run facility allowlist committed 13 distinct source
+  identities, 6,044 samples, 153,696 parameter values, and 103 events. No run
+  was rejected and no local connector failure occurred.
+- A fresh source audit found 3,517 Tool 3 runs, a current maximum run ID of
+  12,164, and a latest start time of 2026-08-06 12:02:00.619830 in the source's
+  timezone-naive timestamp convention.
+- Current run 12,164 committed 351 samples, 23,868 values, and 6 events. It
+  produced two non-fatal registry warnings; unmatched source signals remain
+  visible under their preserved parameter IDs rather than being silently
+  aliased or discarded.
+- The exact normal-mode canary idempotently refreshed run 12,164, discovered
+  new run 12,165, and advanced the live cursor to 12,165. Each source run still
+  had exactly one DT identity.
+
+The live CronJob has no backfill allowlist, uses run 12,164 as its bounded live
+baseline, and runs on `*/10 * * * *` with `concurrencyPolicy: Forbid`. Automatic
+executions at 17:30, 17:40, and 17:50 UTC all succeeded with zero rejected runs
+and zero local failures. They advanced the cursor through run 12,167 while
+preserving exactly one DT identity for each of runs 12,164 through 12,167.
+
+This activation does not authorize or perform an unbounded historical import.
+The approximately 3,500 earlier Tool 3 runs remain a separate backfill project
+and must be copied in reviewed, bounded batches with source/destination counts.
+The Git CronJob manifest remains suspended and placeholder-only by design;
+production activation and its secret-backed mapping are operational state.
+
+### 16.1 Permanent-project cutover
+
+On 2026-08-07 the Tool 3 production mapping was cut over from the temporary
+DB3 pilot to the private permanent project `glance_live_dse_0297fa75` (GLANCE
+Live DSE). The Secret retains the required equipment-keyed mapping for
+`updated_plasmatherm_versaline_deep_silicon_etcher`, source tool 3, and live
+baseline run 12,164. No connection credential or deployable mapping value is
+stored in Git.
+
+The scheduler was suspended with no active Job during the change. The 13
+direct live identities from source runs 12,164 through 12,183 were then
+replayed through the production ingestion endpoint. The stable source identity
+upsert moved those rows to GLANCE Live DSE without inserting duplicates. The
+13 approved historical canary identities (runs 1 through 169) and all 39
+`glance_db3` pilot records remained in the temporary project.
+
+The post-cutover audit recorded:
+
+- 13 permanent-project live identities, 10,822 samples, and 79 events;
+- 13 temporary-project direct canary identities, 6,044 samples, and 103
+  events;
+- unchanged combined totals of 26 direct identities, 16,866 samples, and 182
+  events;
+- no duplicate direct GLANCE source identities; and
+- successful row-level visibility for both `hosler0` and `will3042`, including
+  all 1,772 samples from source run 12,183.
+
+One initial replay Job rejected the configuration before querying GLANCE or
+writing DT data because the mapping had temporarily been serialized as a JSON
+array. The mapping was restored to the connector's required equipment-keyed
+object, independently parsed by the API, and the replacement exact-ID replay
+completed with 13 accepted runs and zero local failures or rejections.
+
+A normal-mode overlap poll and the first automatically scheduled poll after
+resume both completed successfully with cursor 12,183, zero rejected runs, and
+zero local failures. The CronJob remains active on `*/10 * * * *`. This
+cutover does not move the DB3 pilot data and does not authorize an unbounded
+historical Tool 3 backfill.
+
+## 17. Trace export and equipment recipe viewer
+
+The production acceptance test was completed by facility staff using Tool 3
+run 12,213 (`PolyEtch_7loops`) on 2026-08-09. Digital Twin preserved the run in
+GLANCE Live DSE with 87 trace samples, 68 parameters, six events, and recipe
+2,964. Fifty-nine parameters carried GLANCE units; parameters without a source
+unit must be labeled `unit not provided` rather than assigned an inferred unit.
+
+Both the run-detail trace explorer and cross-run Trace Analysis must display a
+prominent Y-axis parameter/unit label and an explicit relative-seconds or
+absolute-source-time X-axis label. They must export the selected parameter to
+CSV with catalog/source run identity, equipment/tool identity, parameter
+identity, source timestamp, derived relative seconds, sample identity, unit,
+and the original recorded value. Edge smoothing and trendlines are display-only
+and must never alter the CSV. Plotly's image action remains available for PNG
+graph export.
+
+The Recipe Viewer is equipment-scoped. It must not compare or combine recipes
+from different equipment. Within an equipment item, `idrecipes` is the source
+identity; recipe name, source hash, and source timestamp distinguish versions
+with repeated names. Recipe discovery, detail, and export use the same project
+row-level visibility as Catalog, return 404 for both absent and inaccessible
+recipes, and never return the Base64 recipe file to the browser.
+
+The server decodes `recipes.recipefile` from strict Base64 into bounded UTF-8
+XML, rejects DTD/entity declarations, verifies the connector-recorded decoded
+size and SHA-256, and caps recipe size, steps, attributes, and cell length.
+Unknown attributes are preserved after the standard GLANCE rows with a visible
+warning; unknown match-mode codes are preserved rather than guessed. The
+decoded matrix and CSV were verified byte-for-byte against the facility's
+recipe-2,964 fixture: 11 steps, 123 parameter rows, and a 9,454-byte CSV.
+Facility source files remain external validation fixtures and are not committed
+to Git.
+
+Production deployment completed on 2026-08-11 from commit `371da73`. The API
+is pinned to `sha256:a9cd36d051b2aada3dd5e7f843cbaac83998711dbd205f66935357e3a0bde483`
+and the web application is pinned to
+`sha256:77251decd1d31729309749331e0801fd071cac315d2a9612ad7ba104930b12f9`.
+An authenticated, non-admin acceptance request as facility user `hosler0`
+resolved recipe 2,964 to run 12,213, returned 11 steps and 123 parameter rows,
+exposed no Base64 source, and produced the exact 9,454-byte reference CSV. The
+10-minute GLANCE CronJob remained enabled and completed successfully after the
+rollout.