This project is the implement of "RIFT - Robust VIdeo Frame Interpolation with MulTiple Intermediate Flow" 见缝插帧. The paper is still under preparation.
git clone https://github.com/askerlee/rift
cd rift
pip3 install -r requirements.txt
Video Frame Interpolation
You can use demo.mp4 or your own video.
python3 inference_video.py --exp=1 --video=video.mp4
(generate video_2X_xxfps.mp4)
python3 inference_video.py --exp=2 --video=video.mp4
(for 4X interpolation)
python3 inference_video.py --exp=1 --video=video.mp4 --scale=0.5
(If your video has very high resolution such as 4K, we recommend set --scale=0.5 (default 1.0). If you generate disordered pattern on your videos, try set --scale=2.0. This parameter control the process resolution for optical flow model.)
python3 inference_video.py --exp=2 --img=input/
(to read video from pngs, like input/0.png ... input/612.png, ensure that the png names are numbers)
python3 inference_video.py --exp=2 --video=video.mp4 --fps=60
(add slomo effect, the audio will be removed)
python3 inference_video.py --video=video.mp4 --montage --png
(if you want to montage the origin video and save the png format output)
Image Interpolation
python3 inference_img.py --img img0.png img1.png --exp=4
(2^4=16X interpolation results) After that, you can use pngs to generate mp4:
ffmpeg -r 10 -f image2 -i output/img%d.png -s 448x256 -c:v libx264 -pix_fmt yuv420p output/slomo.mp4 -q:v 0 -q:a 0
You can also use pngs to generate gif:
ffmpeg -r 10 -f image2 -i output/img%d.png -s 448x256 -vf "split[s0][s1];[s0]palettegen=stats_mode=single[p];[s1][p]paletteuse=new=1" output/slomo.gif
UCF101: Download UCF101 dataset at ./UCF101/ucf101_interp_ours/
Vimeo90K: Download Vimeo90K dataset at ./vimeo_triplet
MiddleBury: Download MiddleBury OTHER dataset at ./middlebury/other-data and ./middlebury/other-gt-interp
HD: Download HD dataset at ./HD_dataset. The RIFE authors also provide a google drive download link.
# RIFT
python3 benchmark/UCF101.py --cp checkpoints/rift-04041959.pth
# "PSNR: 35.356 SSIM: 0.969"
python3 benchmark/Vimeo90K.py --cp checkpoints/rift-04041959.pth
# "PSNR: 35.798 SSIM: 0.979"
python3 benchmark/MiddleBury_Other.py --cp checkpoints/rift-04041959.pth
# "PSNR: 37.670 SSIM: 0.985 IE: 1.961"
python3 benchmark/HD.py --cp checkpoints/rift-04041959.pth
# "PSNR: 32.327. 544*1280: 25.694, 720p: 33.741, 1080p: 37.898"
Download Vimeo90K dataset.
We use 2 GPUs and 20G RAM each GPU for training:
torchrun train.py --consshiftprob 0.1 --multi 8,8,4
We have not released the RIFT paper on arxiv yet. Before that, please consider citing our precursor, the RIFE paper as follows:
@article{huang2020rife,
title={RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation},
author={Huang, Zhewei and Zhang, Tianyuan and Heng, Wen and Shi, Boxin and Zhou, Shuchang},
journal={arXiv preprint arXiv:2011.06294},
year={2020}
}
Our code is based on RIFE. We thank hzwer for his tremendous help during the development.