Birck Digital Twin

A comprehensive system for optimizing semiconductor etching processes through data processing, machine learning, and iterative Pareto optimization.

Overview

The Birck Digital Twin system consists of two main components that work together to process semiconductor etching data and optimize recipe parameters:

Azure Functions: Processes raw etching data from PostgreSQL database, computes step averages and etch metrics, and stores processed datasets in Azure Blob Storage.
nanoHUB: Performs iterative Pareto optimization using machine learning models to predict optimal recipe parameters that balance multiple objectives (etch rate, uniformity, etc.).

System Architecture

High-Level Workflow

The system follows a sequential process from data preparation (Azure) to advanced analytics and active learning (nanoHUB).

Detailed System Architecture

Data Flow

Azure Functions fetches raw run data from PostgreSQL (via PostgREST API)
Processes FIMAP files to extract etch metrics
Computes step averages and detects outliers
Uploads processed dataset (full_dataset.csv) to Azure Blob Storage
nanoHUB downloads the processed dataset
Trains ML models and performs Pareto optimization
Generates recipe proposals and stores results in SQLite database
Sends Teams notifications for new proposals

Components

Azure Functions (`azure/`)

HTTP-triggered Azure Function that processes semiconductor etching data:

Fetches run data from PostgreSQL database
Synchronizes FIMAP files from Azure Blob Storage
Computes step averages using recipe setpoints or threshold methods
Parses FIMAP files to extract etch metrics (etch rate, uniformity, etc.)
Detects outliers (physical limits violations, site-level outliers, calibration drift)
Uploads processed dataset to Azure Blob Storage
Sends Teams notifications via Logic Apps

See Azure README for detailed documentation.

nanoHUB (`nanohub/`)

Comprehensive Pareto optimization system for semiconductor etching processes:

Downloads processed dataset from Azure Blob Storage
Trains ML models (Random Forest, Extra Trees, Gaussian Process Regression)
Generates candidate recipes using Sobol sequences or random sampling
Performs Pareto optimization to identify optimal recipes
Generates recipe proposals for experimental validation
Creates comprehensive visualizations and reports
Stores all results in SQLite database for reproducibility
Provides Streamlit web application for exploration and management
Supports Kubernetes deployment on nanoHUB infrastructure

See nanoHUB README for detailed documentation.

Quick Start

Prerequisites

Python 3.9+
Azure account with:
- Azure Functions App
- Azure Storage Account
- Azure Logic Apps (for Teams notifications)
PostgreSQL database with PostgREST API access
Microsoft Graph API credentials (for SharePoint access)
Access to nanoHUB platform (for nanoHUB deployment)

Installation

Clone the repository:

git clone <repository-url>
cd birck-digital-twin

Install dependencies:
```
pip install -r requirements.txt
```

Configure Azure Functions:

cd azure
cp local.settings.json.example local.settings.json
# Edit local.settings.json with your Azure credentials

Configure nanoHUB:

cd nanohub
cp confidential/config/development.env.example confidential/config/development.env
# Edit confidential/config/development.env with your credentials

Running the System

Deploy Azure Function (see Azure README for details):

cd azure
func start  # Local development
# Or deploy to Azure

Run nanoHUB optimization (see nanoHUB README for details):
```
cd nanohub
python run.py
```

Project Structure

birck-digital-twin/
├── azure/                    # Azure Functions component
│   ├── process_etcher_data/  # Main Azure Function
│   ├── host.json             # Azure Functions host config
│   ├── requirements.txt      # Azure dependencies
│   └── README.md            # Azure documentation
│
├── nanohub/                  # nanoHUB optimization component
│   ├── code/                 # GitHub-pushable code
│   ├── confidential/         # Confidential config (NOT in GitHub)
│   ├── data/                 # Data files (NOT in GitHub)
│   ├── tests/               # Test files (NOT in GitHub)
│   ├── run.py               # Main entry point
│   └── README.md            # nanoHUB documentation
│
├── requirements.txt         # Combined dependencies
└── README.md               # This file

Dependencies

The root requirements.txt combines dependencies for both components:

Azure Functions: Azure SDK, database connectivity, HTTP clients
nanoHUB: Scientific computing (NumPy, Pandas, SciPy), ML (scikit-learn), visualization (Matplotlib, Seaborn), Excel handling

See requirements.txt for the complete list.

Configuration

Azure Functions

Configuration is done through environment variables (Application Settings in Azure):

Azure Storage connection string
PostgREST API URL and token
Microsoft Graph API credentials
Logic Apps webhook URLs

See Azure README for complete configuration details.

nanoHUB

Configuration is done through environment files in nanohub/confidential/config/:

Azure Storage connection string
Microsoft Graph API credentials
Teams notification webhooks
Processing modes and cache management settings

See nanoHUB README for complete configuration details.

Development

Code Organization

Azure code: All Azure-specific code is in azure/
nanoHUB code: All nanoHUB-specific code is in nanohub/code/
Confidential files: Stored in nanohub/confidential/ and excluded from Git
Data files: Stored in nanohub/data/ and excluded from Git

Testing

Azure Functions: Test scripts in azure/tests/ (not deployed)
nanoHUB: Test scripts in nanohub/tests/tests/ (not in GitHub)

Security

Never commit confidential files to Git:
- nanohub/confidential/ - Environment variables with credentials
- azure/local.settings.json - Azure Function settings
Use Azure Key Vault for sensitive credentials in production
Rotate secrets regularly
See component-specific READMEs for security best practices

Documentation

Azure README: Complete Azure Functions documentation
nanoHUB README: Complete nanoHUB optimization documentation
Component-specific documentation in azure/ and nanohub/code/docs/

Contributing

Follow the code organization structure
Keep confidential files out of Git
Update documentation when making changes
Test changes thoroughly before committing

StrachanGroup/Birck-Digital-Twin

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Contributors 2

Languages