Skip to content

Latest commit

 

History

History
executable file
·
239 lines (203 loc) · 6.71 KB

develop.md

File metadata and controls

executable file
·
239 lines (203 loc) · 6.71 KB

Development for evaluation and training

Datasets

First, please prepare standard datasets for evaluation and training.

We present most of prevailing datasets in video frame interpolation, though some are not used in our project. Hope this collection could help your research.

Dataset 🔗 Source Train/Eval Arbitrary/Fixed
Vimeo90k ToFlow (IJCV 2019) Both Fixed
ATD-12K AnimeInterp (CVPR 2021) Both Fixed
SNU-FILM CAIN (AAAI 2021) Eval Fixed
UCF101 Google Driver Eval Fixed
HD MEMC-Net (TPAMI 2018)/Google Driver Eval Fixed
Xiph-2k/-4k SoftSplat (CVPR 2020) Eval Fixed
MiddleBury MiddleBury Eval Fixed
GoPro GoPro Both Arbitrary
Adobe240fps DBN (CVPR 2017) Both Arbitrary
X4K1000FPS XVFI (ICCV 2021) Both Arbitrary

Pretrained Models

Dataset 🔗 Download Links Config file Trained on Arbitrary/Fixed
AMT-S [Google Driver][Baidu Cloud] [cfgs/AMT-S] Vimeo90k Fixed
AMT-L [Google Driver][Baidu Cloud] [cfgs/AMT-L] Vimeo90k Fixed
AMT-G [Google Driver][Baidu Cloud] [cfgs/AMT-G] Vimeo90k Fixed
AMT-S [Google Driver][Baidu Cloud] [cfgs/AMT-S_gopro] GoPro Arbitrary

Evaluation

Before evaluation, you should:

  1. Check the dataroot is organized as follows:
./data
├── Adobe240
│   ├── original_high_fps_videos
│   └── test_frames # using ffmpeg to extract 240 fps frames from `original_high_fps_videos`
├── GOPRO
│   ├── test
│   └── train
├── SNU_FILM
│   ├── GOPRO_test
│   ├── test-easy.txt
│   ├── test-extreme.txt
│   ├── test-hard.txt
│   ├── test-medium.txt
│   └── YouTube_test
├── ucf101_interp_ours
│   ├── 1
│   ├── 1001
│   └── ...
└── vimeo_triplet
    ├── readme.txt
    ├── sequences
    ├── tri_testlist.txt
    └── tri_trainlist.txt
  1. Download the provided pretrained models.

Then, you can perform evaluation as follows:

  • Run all benchmarks for fixed-time models.

    sh ./scripts/benchmark_fixed.sh [CFG] [CKPT_PATH]
## e.g.
sh ./scripts/benchmark_fixed.sh cfgs/AMT-S.yaml pretrained/amt-s.pth

  • Run all benchmarks for arbitrary-time models.

    sh ./scripts/benchmark_arbitrary.sh [CFG] [CKPT_PATH]
## e.g.
sh ./scripts/benchmark_arbitrary.sh cfgs/AMT-S.yaml pretrained/gopro_amt-s.pth

  • Run a single benchmark for fixed-time models. You can custom data paths in this case.

    python [BENCHMARK] -c [CFG] -p [CKPT_PATH] -r [DATAROOT]
## e.g.
python benchmarks/vimeo90k.py -c cfgs/AMT-S.yaml -p pretrained/amt-s.pth -r data/vimeo_triplet

  • Run the inference speed & model size comparisons using:

    python speed_parameters.py -c [CFG]
## e.g.
python speed_parameters.py -c cfgs/AMT-S.yaml

Training

Before training, please first prepare the optical flows (which are used for supervision).

We need to install cupy first before flow generation:

conda activate amt # satisfying `requirement.txt`
conda install -c conda-forge cupy

After installing cupy, we can generate optical flows by the following command:

python flow_generation/gen_flow.py -r [DATA_ROOT]
## e.g.
python flow_generation/gen_flow.py -r data/vimeo_triplet

After obtaining the optical flow of the training data, run the following commands for training (DDP mode):

 sh ./scripts/train.sh [NUM_GPU] [CFG] [MASTER_PORT]
 ## e.g.
 sh ./scripts/train.sh 2 cfgs/AMT-S.yaml 14514

Our training configuration files are provided in cfgs. Please carefully check the dataset_dir is suitable for you.

Note:

  • If you intend to turn off DDP training, you can switch the key distributed from true to false in the config file.

  • If you do not use wandb, you can switch the key logger.use_wandb from true to false in the config file.