by Laura Fink, Linus Franke, Bernhard Egger, Joachim Keinert, and Marc Stamminger
Official repository | Link to project page | My home page
Abstract: Accurate depth estimation is at the core of many applications in computer graphics, vision, and robotics. Current state-of-the-art monocular depth estimators, trained on extensive datasets, generalize well but lack metric accuracy needed for many applications. In this paper, we combine the strength of those general monocular depth estimation techniques with multi-view data by framing this as an analysis-by-synthesis optimization problem to lift and refine such relative depth maps to accurate error-free depth maps. After an initial global scale estimation through structure-from-motion point clouds, we further refine the depth map through optimization enforcing multi-view consistency via photometric and geometric losses with differentiable rendering of the meshed depth map. In a two-stage optimization, first scaling is further refined, and afterwards artifacts and errors in the depth map are corrected via nearby-view photometric supervision. Our evaluation shows that our method is able to generate detailed, high-quality, metrically accurate depth maps, also in challenging indoor scenarios, and outperforms state-of-the-art multi-view depth reconstruction approaches on such datasets.
@article{fink2024refdepth,
title={Refinement of Monocular Depth Maps via Multi-View Differentiable Rendering},
author={Laura Fink and Linus Franke and Bernhard Egger and Joachim Keinert and Marc Stamminger},
journal={arXiv preprint arXiv:2410.03861},
year = {2024}
}
Before you begin, make sure you have the following installed:
- Miniconda or Anaconda
- CUDA Toolkit 12.1 (Tested with 12.1) (or install locally in your environment see 3.)
git clone --recurse-submodules https://github.com/lorafib/ref_depth.git
cd ref_depthor
git clone https://github.com/lorafib/ref_depth.git
git submodule update --init --recursive
cd ref_depthCreate a new Conda environment (adjust version as needed):
conda create -n ref_depth python=3.11
conda activate ref_depthOptional: If you don't have a system-wide CUDA installation available, do
conda install -c nvidia/label/cuda-12.1.1 cuda cuda-toolkit
Then, install all required Python packages from requirements.txt:
git clone https://github.com/NVlabs/nvdiffrast
pip install -r requirements.txtIn case you have issues with versions, check out depth-refinement.yml for my (uncleaned) environment including version numbers.
Download and unzip the provided example data. Place it into this repos root dir.
conda activate ref_depth
python refine_depth.py --config ./example_data/scannetpp_7831862f02/config_scannetpp.json
# or
python refine_depth.py --config ./example_data/Ignatius/config_tnt_ignatius.jsonThis project is licensed under the Fraunhofer Research License.