README.md

# Dynamic Low-light Imaging with Quanta Image Sensors
<img src="./img/real_results.png" width="800">

This is the implementation for [Dynamic Low-light Imaging with Quanta Image Sensors](https://arxiv.org/abs/2007.08614), Yiheng Chi, Abhiram Gnanasambandam, Vladlen Koltun, Stanley H. Chan, in ECCV, 2020.


## Getting Started
- Download the repository

### Prerequisites
- Keras
- PIL
- numpy


## Running
We used the [PASCAL VOC 2008 dataset](http://host.robots.ox.ac.uk/pascal/VOC/voc2008/) for training and synthetic testing. Other standard image datasets should be compatible.

### Training
- Prepare the pretrained models of motion teacher and denoising teacher.
  - The motion teacher used in our implementation is [Burst Denoising with Kernel Prediction Networks](https://ieeexplore.ieee.org/document/8578363), Ben Mildenhall, Jonathan T. Barron, Jiawen Chen, Dillon Sharlet, Ren Ng, Robert Carroll, in CVPR, 2018.
  - The denoising teacher used in our implementation is [Image Reconstruction for Quanta Image Sensors Using Deep Neural Networks](https://ieeexplore.ieee.org/document/8461685), Joon Hee Choi, Omar A. Elgendy, Stanley H. Chan, in ICASSP, 2019.
- Prepare the feature extraction functions to load features from teachers (See utils.py).
- Usage:
```bash
python e2_main.py --phase train
```

### Testing
- A model pretrained using the default setting (photon level = 2 ppp, motion magnitude = 4 pixels) is available in the repository.
- Usage:
```bash
python e2_main.py --phase test
```

### Usage
```
usage: e2_main.py [-h] [--phase PHASE] [--patches NUM_PATCH]
                  [--patchsize PATCH_SZ] [--burstsize BURST_SZ]
                  [--batchsize BATCH_SZ] [--epochs EPOCHS] [--lr LR]
                  [--jit JIT] [--J J] [--alpha ALPHA] [--ckpt_dir CKPT_DIR]
                  [--train_dir TRAIN_DIR] [--valid_dir VALID_DIR]
                  [--test_dir TEST_DIR] [--output_dir OUTPUT_DIR]
                  [--ckpt_file CKPT_FILE] [--shift_ckpt SHIFT_CKPT]
                  [--noisy_ckpt NOISY_CKPT] [--seed SEED]
```

```
optional arguments:
  -h, --help            show this help message and exit
  --phase               train or test
  --patches             number of patches extracted from an image
  --patchsize           patch size, height/width of frames in each burst
  --burstsize           burst size, number of frames in each burst
  --batchsize           batch size, number of bursts in each batch
  --epochs              number of epoch
  --lr                  learning rate
  --jit                 jit
  --J                   downsample ratio J
  --alpha               alpha
  --ckpt_dir            the directory that models are saved
  --train_dir           the directory that training images are saved
  --valid_dir           the directory that validation images are saved
  --test_dir            the directory that test images are saved
  --output_dir          the directory to save processed test images
  --ckpt_file           the file that models are saved
  --shift_ckpt          the file of shift model
  --noisy_ckpt          the file of noisy model
  --seed                random seed number
```


## Citation
```
@inproceedings{chi2020dynamic,
  title={Dynamic low-light imaging with quanta image sensors},
  author={Chi, Yiheng and Gnanasambandam, Abhiram and Koltun, Vladlen and Chan, Stanley H},
  booktitle={Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XXI 16},
  pages={122--138},
  year={2020},
  organization={Springer}
}
```