Birck Digital Twin

An end-to-end platform for optimizing semiconductor etching processes through automated data ingestion, live machine learning, and an interactive Digital Twin dashboard.

Overview

The Birck Digital Twin platform has evolved beyond static analysis into a live, production-ready ecosystem. It integrates local equipment data with cloud-based ML to provide real-time recipe optimization:

Data Ingestion (Azure & Uploader): Captures raw experiment data from local equipment and syncs it to a central PostgreSQL database.
ML Engine (FastAPI): A live backend that trains Gaussian Process Regression (GPR) models on the production database to provide Pareto-optimal recipe suggestions.
Digital Twin Dashboard (Next.js): A premium web interface for monitoring equipment stats, visualizing model parity (LOOCV), and generating recipe proposals.

System Architecture

Data & Workflow Pipeline

Uploader: Runs on equipment machines to package experiments into RO-Crate metadata.
PostgreSQL (Geddes): The source of truth for all experiment runs (etcher_runs) and project-based access control (RLS).
ML API: Consumes Postgres data to perform real-time training and Pareto optimization batches.
Dashboard: Directly interfaces with the ML API for live analytics and recipe generation.

Component Breakdown

Component	Path	Technology	Purpose
API	`api/`	FastAPI, Scikit-Learn	Real-time ML engine and data access layer.
Web App	`web/`	Next.js, Tailwind, Auth.js	The primary "Digital Twin" interface for researchers.
Uploader	`uploader/`	Python	Local equipment data packaging and sync.
Azure Ingestion	`azure/`	Python Azure Functions	Cloud-side processing and metric extraction.
Optimization	`nanohub/`	Python	Legacy Pareto search and batch processing logic.

Quick Start (Production)

1. Environment Configuration

You must set up the shared DT_SYSTEM_TOKEN, INGESTION_TOKEN, and Postgres credentials in your environment. Copy the template to get started:

cd geddes/k8s
cp 01-secrets.yaml.example 01-secrets.yaml
# Edit with your PG_SUPER_PASS and INGESTION_TOKEN
# Also set DT_SYSTEM_TOKEN and PG_PASS before starting the API or web app

2. Deployment (Geddes Kubernetes)

The platform is optimized for the Purdue Geddes cluster. Build the images for the linux/amd64 platform:

# Build & Push API
docker build --platform linux/amd64 -f api/Dockerfile -t geddes-registry.rcac.purdue.edu/sdx/dt-api:v5 .
docker push geddes-registry.rcac.purdue.edu/sdx/dt-api:v5

# Build & Push Web
cd web
docker build --platform linux/amd64 -t geddes-registry.rcac.purdue.edu/sdx/dt-web:v5 .
docker push geddes-registry.rcac.purdue.edu/sdx/dt-web:v5

# Apply Manifests
kubectl apply -f geddes/k8s/

The API pod is stateless: every durable artifact (equipment metadata, runs, projects, uploads) lives in the dt-db-v2 Postgres database. There is no PersistentVolumeClaim mounted into the API container. Approved-equipment JSON snapshots and /data/upload blobs land on the container's ephemeral filesystem and are intentionally lost on pod restart — Postgres is the source of truth, and the snapshot files are a regenerable cache. If you ever need durable user-uploaded blobs, route them through Azure Blob Storage (the same path the Azure ingestion pipeline already uses) rather than re-introducing a PVC.

Data Management

Database Seeding

To migrate existing legacy data (e.g., from full_dataset.csv) to the live production database:

python scripts/sync_csv_to_postgres.py

Ingestion Security

Communication between the Azure-side ingestion and the Geddes-side API is secured via a shared INGESTION_TOKEN. Ensure this matches in both the Azure Function configuration and the K8s secrets. The Azure Function must also be configured with an explicit INGESTION_API_URL ending in /api/dataset/v2/runs/sync; the etcher ingestion flow now fails closed and will not advance blob state if the DT sync step is not configured or returns an error.

Platform Authentication

The API and Next.js proxy require DT_SYSTEM_TOKEN to be set explicitly. The API also requires PG_PASS and PG_SUPER_PASS at startup; it no longer falls back to development credentials.

Etcher Smoke Test

To verify the Azure sync -> API runs listing -> ML proposals contract on this branch without needing the cluster, run:

api/.venv/bin/python api/tests/test_etcher_realtime_smoke.py

Project Structure

Birck-Digital-Twin/
├── api/                    # Live ML Engine (FastAPI)
├── web/                    # Digital Twin Dashboard (Next.js)
├── azure/                  # Azure Ingestion Pipeline
├── uploader/               # Equipment Metadata Uploader
├── geddes/                 # K8s Deployment Manifests
├── scripts/                # Database Sync & Maintenance Utilities
└── docs/                   # Full Dataset & Historical Records

Documentation

Deployment Runbook: Step-by-step instructions for live setup.
API V2 Docs: Python interface for Postgres data access.
Web Auth Docs: NanoHUB OAuth2 integration details.

will3042/Birck-Digital-Twin

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Contributors

Uh oh!

Languages