Enforce row-level security for experiment data

[Navid Gholizadeh (ngholiza)]

Summary

Enforces experiment data isolation in Postgres with row-level security (RLS), separates the API's database roles, and fixes the experiment-creation and proposal-generation issues found in review. This code is already running in production (dt-api:rls-b02e9ac); merging makes dev match what is deployed.

Database isolation

• RLS policies for experiment definitions, templates and versions, project template attachments, samples and sample links, recipe batches and proposals, and project membership. Admin, worker and write policies are added to the tables that already had RLS.
• Enforcement on the nine experiment tables is controlled by DB_RLS_ENFORCEMENT (enforce / disable). project_members is always enforced, together with its new write grants.
• A trigger blocks non-admin changes to an experiment's owner, project, equipment, template and request id, and limits equipment owners to execution-status fields.

Database roles

• User requests: api_client, carrying the caller's id, role and organization for the whole session.
• Background jobs: a new api_worker role (not superuser, cannot bypass RLS, no DDL).
• Owner credentials are used only by migrations, which now run as a separate Job (python -m db_migrations, geddes/k8s/09-db-migrate-job.yaml).

Fixes

• Template visibility is checked when creating, forking, attaching, viewing versions, using processes and seeding workflow examples.
• Experiment creation writes its initial proposal status in the same transaction. A retry with the same client_request_id returns the original experiment; a reused key with a different payload returns 409.
• Proposal generation is a durable job queue with leases, bounded retries and results saved atomically with completion, plus a sweeper in each API worker.
• init_db.sql now works on a fresh database; three stale security tests are aligned with the project-workflow change in 004d902.
• Pytest points database connections at a dead port unless DT_ALLOW_LIVE_DB_TESTS=1, so unit tests can't reach a port-forwarded production database.

Operations docs

• api/ROW_LEVEL_SECURITY.md: access model, rollout steps and tested rollback order.
• geddes/k8s/02-api.yaml: deployed image and settings, plus a description of the DB3 pilot layer on the live Deployment.
• geddes/k8s/GLANCE_INGESTION_OPERATIONS.md: how GLANCE run 12822 was deferred, and how to defer a run in future.

Production state (2026-09-24/25)

• The dt-api Deployment and dt-api-retrain CronJob run dt-api:rls-b02e9ac, with DB_RLS_ENFORCEMENT=enforce and DB_MIGRATE_ON_STARTUP=false.
• api_worker is provisioned. The owner credentials were moved from dt-api-secret to dt-db-migration-secret.
• GLANCE ingestion is unblocked. It had failed since 2026-09-04 on tool 3 run 12822 (11 h, about 97.6% idle logging). The run was deferred, not deleted, and polling has resumed.

Test plan

• Unit tests: 143 passed, 0 failed.
• RLS integration suite (api/tests/test_row_level_security_pg.py): 36/36 on a disposable Postgres 15 with enforcement on. With enforcement off, only the isolation-only assertions fail.
• Rollback: the previous dev code works against the migrated database while enforcement is off.
• Production smoke test: every read endpoint as all 17 active users (423 requests) was identical to the pre-deploy baseline after deploying, after enforcing, and after removing owner credentials.
• Production isolation: per-role row counts, plus write probes in transactions that were rolled back. A PI can create a project and an experiment; a researcher writing into a PI's project and a PI claiming someone else's project are both rejected.
• The 02:00 UTC retrain ran as api_worker and succeeded.
• Reviewed by a second agent (Codex) over 9 rounds; all findings resolved, final verdict approve.

🤖 Generated with Claude Code