Enforce row-level security for experiment data

api/ROW_LEVEL_SECURITY.md

@@ -0,0 +1,159 @@
+# Row-level security
+
+Postgres row-level security (RLS) enforces who can read and change
+experiment data and project membership, in the database, in addition to the
+API's own checks. This page covers the access model, the database roles, and
+how to roll it out or roll it back.
+
+## Database roles
+
+| Role | Used by | Privileges |
+|---|---|---|
+| `api_client` | Every user request, via `get_pg_connection(user)` | Table grants only; subject to RLS. The connection carries the caller in session settings. |
+| `api_worker` | Background and system jobs, via `get_pg_worker_connection()`: ingestion, optimizer jobs, model registry, audits | Data access on all tables, explicit `TO api_worker` policies on RLS tables. Not superuser, not `BYPASSRLS`, owns nothing, cannot run DDL. |
+| owner (`postgres`) | Schema migrations only, via `get_pg_migration_connection()` | Table owner. Never used to serve requests. |
+
+A user-scoped connection sets three session settings that the policies read:
+`app.current_user` (user id), `app.current_user_role` (`admin`, `pi`,
+`equipment_owner`, `researcher`) and `app.current_user_org`. They are set at
+session scope and committed, so a commit or rollback part-way through a
+request cannot drop the caller's identity.
+
+Until `PG_WORKER_PASS` is configured, `get_pg_worker_connection()` falls back
+to the owner credentials and logs a warning once, so existing deployments keep
+working.
+
+## Access rules
+
+The policies mirror the API's existing checks
+(`api/scripts/add_row_level_security.sql`):
+
+- **Projects:** members and admins see private projects; anyone signed in sees
+  open projects. PIs and admins create projects; a project's PI or an admin
+  deletes it.
+- **Project members:** visible to anyone who can read the project (as before)
+  and to admins. A PI adds plain members; the creator of a project enrols
+  themselves as its PI in the same transaction that created it; only admins
+  change PI assignments. A PI removes non-PI members. Always enforced.
+- **Experiment definitions:** readable by the owner, anyone who can read the
+  project, and facility managers (equipment owners) for execution requests on
+  their equipment. Created by the owner into projects they can write, with a
+  template they can read. A trigger fixes an experiment's owner, project,
+  equipment, template and request id after creation (admins and background
+  jobs excepted), and limits facility managers' updates to the execution
+  status columns.
+- **Experiment templates:** published templates are public; otherwise the
+  owner, shared templates, the owner's organization, admins, and anyone who can
+  read a project or experiment that uses the template. Only the owner or an
+  admin edits or deletes. Version history follows template visibility.
+- **Samples and sample links:** follow project access; facility managers can
+  read samples linked to execution requests they manage.
+- **Recipe batches and proposals:** readable with the experiment; written only
+  by the proposal worker; status updates by the experiment owner, project
+  members or admins.
+- **Run tables** (`etcher_runs`, `equipment_runs`, trace tables, run files):
+  unchanged member/open read policies, plus admin read and worker access.
+
+Two operations deliberately cross tenants and are implemented as narrow
+`SECURITY DEFINER` functions that re-check permission in the database:
+`app_detach_experiment_type` (deleting a template clears references in other
+projects) and `app_experiment_project_name` (a facility manager sees the name
+of the project behind an execution request, and nothing else about it).
+
+Operations that intentionally act across all tenants run on the worker
+connection after the API's own authorization: ingestion and run sync, the
+optimizer and follow-up loops, equipment deletion (which detaches projects),
+equipment fleet metrics, process-definition reads, identity merges and model
+registry work.
+
+## Configuration
+
+| Variable | Where | Meaning |
+|---|---|---|
+| `PG_WORKER_USER`, `PG_WORKER_PASS` | API | `api_worker` credentials for background jobs. |
+| `PG_SUPER_USER`, `PG_SUPER_PASS` | migrations (and the API until the worker role exists) | Owner credentials. |
+| `DB_MIGRATE_ON_STARTUP` | API | `false` skips migrations at startup; default runs them when owner credentials are present. |
+| `DB_RLS_ENFORCEMENT` | wherever migrations run | `enforce` turns RLS on for the nine experiment tables; `disable` turns it off for them (rollback); unset leaves them unchanged. `project_members` and the six original tables are always enforced. |
+| `DB_MIGRATION_LOCK_TIMEOUT` | wherever migrations run | Maximum wait for a migration lock (default `30s`). |
+| `PROPOSAL_JOB_SWEEP_SECONDS` | API | How often each API worker runs queued/expired proposal jobs (default 60, `0` disables). |
+| `PROPOSAL_JOB_LEASE_SECONDS`, `PROPOSAL_JOB_MAX_ATTEMPTS`, `PROPOSAL_JOB_MAX_AGE_HOURS` | API | Proposal job lease (900 s), retry limit (3) and maximum age (72 h). |
+
+## Rollout
+
+Each step is safe on its own and can be verified before the next.
+
+1. **Deploy the new API image.** Startup migrations add the new columns,
+   helper functions, grants, policies and guards, and enforce RLS on
+   `project_members` (together with the membership write grants). Enforcement
+   on the nine experiment tables stays off (`DB_RLS_ENFORCEMENT` unset); the
+   API's own checks, including the new template-visibility checks, still
+   apply, as do the atomic experiment creation, durable proposal jobs and
+   idempotent retries. Check `GET /api/health` → `postgres.row_level_security`:
+   `projects` and `project_members` are `true`, the experiment tables `false`.
+2. **Provision the worker role.** As `postgres`:
+   `ALTER ROLE api_worker WITH LOGIN PASSWORD '<strong-password>';`
+   Add `PG_WORKER_USER`/`PG_WORKER_PASS` to `dt-api-secret` and restart the API.
+   Health shows `runtime.postgres_worker_role_configured: true`, and the
+   "falling back to migration credentials" warning disappears from the logs.
+3. **Enforce RLS.** Set `DB_RLS_ENFORCEMENT=enforce` in `dt-api-secret` (or in
+   the migration Job's secret) and restart / re-run migrations. Health shows
+   `row_level_security` all `true`. Smoke-test as a PI, a researcher, an
+   equipment owner and an admin: projects, experiments, samples, recipe
+   proposals and the facility execution queue.
+4. **Take owner credentials out of the API.** Create `dt-db-migration-secret`
+   (see `geddes/k8s/01-secrets.yaml.example`), run migrations with
+   `geddes/k8s/09-db-migrate-job.yaml` before each rollout, set
+   `DB_MIGRATE_ON_STARTUP=false` and remove `PG_SUPER_USER`/`PG_SUPER_PASS`
+   from `dt-api-secret`. Health stays `ok` without them.
+
+## Rollback
+
+- **RLS causing problems:** set `DB_RLS_ENFORCEMENT=disable` and re-run
+  migrations (restart the API while it still runs them, or run the Job). This
+  turns RLS off on the nine experiment tables; `project_members` and the six
+  original tables stay enforced, and the API's own checks remain.
+- **Worker role problems:** remove `PG_WORKER_PASS`; background jobs fall back
+  to the owner credentials (requires `PG_SUPER_PASS` in the API secret).
+- **Rolling back to an API image from before this change:** turn experiment
+  enforcement off *first*, while the new image is still running
+  (`DB_RLS_ENFORCEMENT=disable`, restart), then roll back the image. The old
+  code creates experiments and templates without a user identity, which the
+  enforced policies reject. If the new image is already gone, run as the owner:
+
+  ```sql
+  ALTER TABLE experiment_definitions DISABLE ROW LEVEL SECURITY;
+  ALTER TABLE experiment_types DISABLE ROW LEVEL SECURITY;
+  ALTER TABLE experiment_type_versions DISABLE ROW LEVEL SECURITY;
+  ALTER TABLE project_experiments DISABLE ROW LEVEL SECURITY;
+  ALTER TABLE samples DISABLE ROW LEVEL SECURITY;
+  ALTER TABLE experiment_samples DISABLE ROW LEVEL SECURITY;
+  ALTER TABLE run_samples DISABLE ROW LEVEL SECURITY;
+  ALTER TABLE experiment_recipe_batches DISABLE ROW LEVEL SECURITY;
+  ALTER TABLE experiment_recipe_proposals DISABLE ROW LEVEL SECURITY;
+  ```
+
+  After rollout step 4 (owner credentials removed from the API and
+  `DB_MIGRATE_ON_STARTUP=false`), restarting the API no longer changes
+  enforcement: set `DB_RLS_ENFORCEMENT=disable` in `dt-db-migration-secret`
+  and run the migration Job (or the SQL above as the owner), confirm health
+  shows the nine tables as `false`, and put `PG_SUPER_USER`/`PG_SUPER_PASS`
+  back into `dt-api-secret`, because older images require them to start.
+  Roll the `dt-api-retrain` CronJob image back together with the Deployment.
+
+  The old code works with everything else this migration leaves behind,
+  including enforced `project_members` (verified by running the previous API
+  code against a migrated database).
+
+## Applying the SQL by hand
+
+Prefer `python -m db_migrations`, which applies the script in one
+transaction. If you apply it with psql, keep it atomic:
+`psql --single-transaction -v ON_ERROR_STOP=1 -f api/scripts/add_row_level_security.sql`
+
+## Verifying locally
+
+`api/tests/test_row_level_security_pg.py` runs the migrations with enforcement
+on and exercises the API functions as different users against a disposable
+Postgres 15. It is skipped unless `DT_RLS_TEST_PG_PORT` is set and refuses to
+run against a database without an `rls_test_marker` table. See the module
+docstring for setup.

api/ai_readiness.py

@@ -93,14 +93,11 @@ def export_ml_matrix(
             "column_types": {"run_id": "int", ...},
         }
     """
-    from data_loader_pg import get_pg_connection, get_pg_superuser_connection
+    from data_loader_pg import _scoped_connection
     from psycopg2.extras import RealDictCursor
 
     try:
-        if is_admin:
-            conn = get_pg_superuser_connection()
-        else:
-            conn = get_pg_connection(nanohub_user_id)
+        conn = _scoped_connection(nanohub_user_id, is_admin)
 
         # Build the flat query — no nested JSON, just raw columns
         select_cols = METADATA_COLUMNS + FEATURE_COLUMNS + TARGET_COLUMNS if include_metadata else FEATURE_COLUMNS + TARGET_COLUMNS
@@ -220,7 +217,7 @@ def record_dataset_snapshot(
     SHA-256 hash of the full dataset. This lets you trace any model back
     to the exact data that trained it.
     """
-    from data_loader_pg import get_pg_superuser_connection
+    from data_loader_pg import get_pg_worker_connection
 
     snapshot_record = {
         "snapshot_hash": snapshot_hash,
@@ -231,7 +228,7 @@ def record_dataset_snapshot(
     }
 
     try:
-        conn = get_pg_superuser_connection()
+        conn = get_pg_worker_connection()
         with conn.cursor() as cur:
             cur.execute(
                 """
@@ -271,11 +268,11 @@ def record_dataset_snapshot(
 
 def get_latest_snapshots(limit: int = 10) -> list[dict[str, Any]]:
     """Fetch the most recent dataset snapshots."""
-    from data_loader_pg import get_pg_superuser_connection
+    from data_loader_pg import get_pg_worker_connection
     from psycopg2.extras import RealDictCursor
 
     try:
-        conn = get_pg_superuser_connection()
+        conn = get_pg_worker_connection()
         with conn.cursor(cursor_factory=RealDictCursor) as cur:
             cur.execute(
                 """
@@ -312,11 +309,11 @@ def generate_data_card(
       - Temporal coverage (runs per month)
       - Known limitations / bias warnings
     """
-    from data_loader_pg import get_pg_superuser_connection
+    from data_loader_pg import get_pg_worker_connection
     from psycopg2.extras import RealDictCursor
 
     try:
-        conn = get_pg_superuser_connection()
+        conn = get_pg_worker_connection()
 
         with conn.cursor(cursor_factory=RealDictCursor) as cur:
             # ── Overview statistics ──────────────────────────────────────