Dynamic Low-light Imaging with Quanta Image Sensors

This is the implementation for Dynamic Low-light Imaging with Quanta Image Sensors, Yiheng Chi, Abhiram Gnanasambandam, Vladlen Koltun, Stanley H. Chan, in ECCV, 2020.

Getting Started

• Download the repository

Prerequisites

• Keras
• PIL
• numpy

Running

We used the PASCAL VOC 2008 dataset for training and synthetic testing. Other standard image datasets should be compatible.

Training

• Prepare the pretrained models of motion teacher and denoising teacher.
  • The motion teacher used in our implementation is Burst Denoising with Kernel Prediction Networks, Ben Mildenhall, Jonathan T. Barron, Jiawen Chen, Dillon Sharlet, Ren Ng, Robert Carroll, in CVPR, 2018.
  • The denoising teacher used in our implementation is Image Reconstruction for Quanta Image Sensors Using Deep Neural Networks, Joon Hee Choi, Omar A. Elgendy, Stanley H. Chan, in ICASSP, 2019.
• Prepare the feature extraction functions to load features from teachers (See utils.py).
• Usage:

python e2_main.py --phase train

Testing

• A model pretrained using the default setting (photon level = 2 ppp, motion magnitude = 4 pixels) is available in the repository.
• Usage:

python e2_main.py --phase test

Usage

usage: e2_main.py [-h] [--phase PHASE] [--patches NUM_PATCH]
                  [--patchsize PATCH_SZ] [--burstsize BURST_SZ]
                  [--batchsize BATCH_SZ] [--epochs EPOCHS] [--lr LR]
                  [--jit JIT] [--J J] [--alpha ALPHA] [--ckpt_dir CKPT_DIR]
                  [--train_dir TRAIN_DIR] [--valid_dir VALID_DIR]
                  [--test_dir TEST_DIR] [--output_dir OUTPUT_DIR]
                  [--ckpt_file CKPT_FILE] [--shift_ckpt SHIFT_CKPT]
                  [--noisy_ckpt NOISY_CKPT] [--seed SEED]

optional arguments:
  -h, --help            show this help message and exit
  --phase               train or test
  --patches             number of patches extracted from an image
  --patchsize           patch size, height/width of frames in each burst
  --burstsize           burst size, number of frames in each burst
  --batchsize           batch size, number of bursts in each batch
  --epochs              number of epoch
  --lr                  learning rate
  --jit                 jit
  --J                   downsample ratio J
  --alpha               alpha
  --ckpt_dir            the directory that models are saved
  --train_dir           the directory that training images are saved
  --valid_dir           the directory that validation images are saved
  --test_dir            the directory that test images are saved
  --output_dir          the directory to save processed test images
  --ckpt_file           the file that models are saved
  --shift_ckpt          the file of shift model
  --noisy_ckpt          the file of noisy model
  --seed                random seed number

Citation

@inproceedings{chi2020dynamic,
  title={Dynamic low-light imaging with quanta image sensors},
  author={Chi, Yiheng and Gnanasambandam, Abhiram and Koltun, Vladlen and Chan, Stanley H},
  booktitle={Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XXI 16},
  pages={122--138},
  year={2020},
  organization={Springer}
}