Generalizable Novel-View Synthesis using a Stereo Camera
Haechan Lee*, Wonjoon Jin*, Seung-Hwan Baek, Sunghyun Cho
- Our paper is accepted to CVPR 2024!
- Check out our Project page!
- The StereoNVS dataset and the BlendedMVS-Stereo dataset are released. Check out our Google Drive Link!
- Code id released!
In this paper, we propose the first generalizable view synthesis approach that specifically targets multi-view stereo-camera images. Since recent stereo matching has demonstrated accurate geometry prediction, we introduce stereo matching into novel-view synthesis for high-quality geometry reconstruction. To this end, this paper proposes a novel framework, dubbed StereoNeRF, which integrates stereo matching into a NeRF-based generalizable view synthesis approach. StereoNeRF is equipped with three key components to effectively exploit stereo matching in novel-view synthesis: a stereo feature extractor, a depth-guided plane-sweeping, and a stereo depth loss. Moreover, we propose the StereoNVS dataset, the first multi-view dataset of stereo-camera images, encompassing a wide variety of both real and synthetic scenes. Our experimental results demonstrate that StereoNeRF surpasses previous approaches in generalizable view synthesis.
Tested with Docker image (pytorch/pytorch:1.9.1-cuda11.1-cudnn8-devel).
To install python library dependencies, run the code below:
pip install -r requirements.txt
If you want to reproduce the results of StereoNeRF:
- Adjust data paths in the config files located in the 'configs' folder.
- Download the pretrained UniMatch weights to the 'pretrained_weights' folder.
- Prepare disparity maps of StereoNVS-Real predicted by UniMatch.
You can download the pretrained weights and rendered StereoNVS-Real images of StereoNeRF from our Google Drive Link.
(Our code is heavily based on GeoNeRF).
To train the model, run:
python run_stereo_nerf.py --config configs/config_general_stereo_real.txt
Set the ckpt_file path properly at line 667 in 'run_stereo_nerf.py', then run:
python run_stereo_nerf.py --config configs/config_general_stereo_real.txt --eval
To calculate the scores, run:
python calc_score.py --pred /path/to/evaluation/folder/in/prediction
- StereoNVS-Real: CC BY-NC 3.0 DEED (Non-commercial use)
- BlendedMVS-stereo: CC BY-NC 3.0 DEED (Non-commercial use)
If you want to contact the author, email to: gocks8@gmail.com