Namespace migration retrain fix

.dockerignore

@@ -0,0 +1,30 @@
+.git
+.github
+.agents
+.codex
+
+**/__pycache__
+**/*.py[cod]
+**/.pytest_cache
+**/.venv
+**/node_modules
+**/.next
+
+**/.env
+**/.env.*
+**/local.settings.json
+**/appsettings.json
+**/*kubeconfig*
+
+incoming
+**/*.db
+**/*.db3
+**/*.sqlite
+**/*.sqlite3
+nanohub/confidential
+nanohub/config
+nanohub/data
+uploader/staging
+
+.DS_Store
+*.log

.gitignore

@@ -58,6 +58,8 @@ debug_*.py
 !api/tests/test_phase3_security.py
 !api/tests/test_e2e_admin_flow.py
 !api/tests/test_e2e_platform.py
+!api/tests/test_glance_db3_trace_import.py
+!api/tests/test_retrain.py
 
 # Development files
 temp.py
@@ -90,6 +92,11 @@ process_etcher_data/OUTPUT_DATASET.csv
 *.db
 *.sqlite
 *.sqlite3
+*.db3
+
+# Local source exports and conversion scratch data. GLANCE DB3 files can contain
+# proprietary equipment traces and must never be committed.
+/incoming/
 
 # Cache directories
 .cache/
@@ -122,6 +129,8 @@ temp_azure/
 *.ps1
 *.sh
 !scripts/check_deployment_state.sh
+!geddes/k8s/pilots/db3/deploy.sh
+!geddes/k8s/pilots/db3/rollback.sh
 
 # VS Code settings
 .vscode/settings.json

DB3_UPLOAD_PIPELINE_REVIEW.md

@@ -0,0 +1,135 @@
+# Temporary GLANCE DB3 Test Import
+
+## Decision confirmed with Paul
+
+One catalog row represents one physical GLANCE machine run. It does **not**
+represent one timestamped sample.
+
+Every sample row and every parameter value from that run is retained in a linked
+time-series trace. The source absolute timestamp remains authoritative. Relative
+seconds are calculated only when a user reads or plots one or more traces, with
+each run independently starting at its earliest sample.
+
+No input/output summaries are invented in this branch. The physical catalog run
+has empty scalar input/output objects until Paul and the team define scientifically
+valid summary rules.
+
+## Test-only scope
+
+Uploading a `.db3` is a temporary way to test this data model and plotting flow.
+It is opt-in and must not be treated as the long-term GLANCE integration. A future
+production integration should read from the authoritative GLANCE service or an
+approved export pipeline rather than depend on users moving SQLite files around.
+
+DB3 upload is default-off. It requires all three temporary pilot controls:
+
+- `DT_ENABLE_DB3_UPLOAD=true`;
+- the authenticated nanoHUB ID in `DT_DB3_ALLOWED_USER_IDS`;
+- `DT_DB3_EQUIPMENT_TOOL_MAP` explicitly mapping the selected registered
+  equipment ID to the GLANCE tool name embedded in the export.
+
+```json
+{
+  "<registered-equipment-id>": ["VLN-11304-CTC-PM1"]
+}
+```
+
+This mapping is intentional. It prevents an export from one machine from being
+silently attached to another machine as more GLANCE tools are tested. The default
+DB3 limit is 64 MB and can be adjusted with `DT_DB3_UPLOAD_MAX_MB`. The UI reads
+an authenticated runtime capability endpoint, so users outside the allowlist do
+not see DB3 as an accepted upload type or see DB3 upload records.
+
+## Workflow
+
+1. The user selects an approved equipment registration and uploads one `.db3`.
+2. The API streams the file to upload storage in bounded chunks. It does not read
+   the complete DB3 into one in-memory bytes object and does not preprocess inside
+   the async request handler.
+3. The upload appears as `success`. The user selects a disposable test project and
+   clicks **Process**.
+4. The existing synchronous background task reads SQLite in read-only mode and
+   extracts one object per physical run. A malformed run is reported and skipped
+   without discarding other valid physical runs.
+5. One database transaction upserts the physical catalog parents and replaces all
+   linked samples. A retry is idempotent by source upload plus GLANCE run ID.
+6. The run detail page retrieves a trace through
+   `GET /dataset/v2/runs/{run_id}/trace` and can compare up to six runs using an
+   absolute/source or dynamically calculated relative time axis.
+
+Processing uses an in-process background task for this pilot. A timestamped
+processing lease is refreshed by a heartbeat while work is active. If the
+worker disappears, polling returns the abandoned DB3 to a retryable state after
+the configured stale interval; trace synchronization remains idempotent.
+
+The old 52 generated input/output CSV records are gone. A single DB3 now requires
+one Process action.
+
+## Persisted model
+
+`equipment_runs` stores one parent per physical run, including:
+
+- lot name for display;
+- GLANCE run, tool, and recipe identifiers;
+- tool, recipe, material, start/end, and status provenance;
+- source filename, upload ID, size, and SHA-256;
+- parameter metadata keyed by the stable GLANCE parameter ID (`p<ID>`).
+
+`equipment_run_trace_samples` stores every sample linked to its parent:
+
+- source sample-record ID;
+- source GLANCE tool ID recorded on the sample;
+- deterministic sample index;
+- exact `sample_time_raw` text;
+- parsed timezone-naive wall time for ordering and relative-time arithmetic;
+- a JSON object containing every value present at that timestamp.
+
+The source export does not declare a timezone, so the API reports
+`timestamp_timezone: null` and never labels the values UTC. Missing values remain
+null and empty sample rows remain `{}` so plots show real gaps.
+
+Lot name is not a technical identity. Paul's sample contains repeated lot names,
+including many physical runs named `FM-2'`. The durable source identity is the
+data-upload UUID plus GLANCE `idruns`; lot name remains searchable display data.
+
+## Paul's sample expectation
+
+For `incoming/paul_db3_pipeline/MyDatabaseSample.db3`, the extractor asserts:
+
+- 27 rows in the source `runs` table;
+- 26 physical runs with sample records, therefore 26 catalog parents;
+- 11,697 linked sample rows;
+- 11,696 samples with values and one retained empty-value sample;
+- 242 source parameter definitions;
+- 795,328 parameter values;
+- one source tool: `VLN-11304-CTC-PM1`.
+
+These counts intentionally differ from the discarded CSV conversion, which
+removed the empty sample and expanded samples into catalog rows.
+
+## Verification
+
+The focused extractor test builds a small SQLite fixture at runtime. It covers
+duplicate lot names, fractional and irregular timestamps, missing values, an
+empty-value sample, a no-sample run, source checksum/provenance, and per-run
+failure isolation.
+
+```bash
+api/.venv/bin/python -m pytest -q api/tests/test_glance_db3_trace_import.py
+api/.venv/bin/python -m py_compile \
+  api/glance_db3/trace_import.py \
+  api/data_loader_pg.py \
+  api/routers/upload.py \
+  api/routers/dataset_v2.py
+
+cd web
+npx tsc --noEmit
+npm run lint
+```
+
+The local `incoming/` directory and all `*.db3` files are ignored because source
+exports can contain proprietary equipment traces. No generated CSV/intermediate
+directories are created. The Geddes pilot stores raw uploads on its own encrypted
+PVC, isolated from the normal deployment manifest. The removable deployment and
+digest-pinned rollback procedure live in `geddes/k8s/pilots/db3/`; that package
+must not be merged into the final production platform.

api/FAIR_ARCHIVAL_WALKTHROUGH.md

@@ -55,7 +55,7 @@ Comprehensive rewrite with researcher-friendly guide: setup, usage, architecture
 
 1. **Run the migration**:
    ```bash
-   kubectl exec -n nanohub dt-db-v2-0 -- psql -U postgres -d digital_twin -f /tmp/add_fair_columns.sql
+   kubectl exec -n ncn-digitaltwins-zchen dt-db-v2-0 -- psql -U postgres -d digital_twin -f /tmp/add_fair_columns.sql
    ```
 
 2. **Add PURR env vars** to the API deployment: