From b0db2f70c022ebacc1dc41f5985eff45f564fcdd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Johannes=20Sch=C3=B6nberger?= Date: Fri, 9 Aug 2024 13:52:03 +0200 Subject: [PATCH 01/29] More content for documentation about visualization and settings (#51) * More content for documentation about visualization and recommended settings * Update README.md --------- Co-authored-by: Linfei Pan <36349740+lpanaf@users.noreply.github.com> --- README.md | 18 +++++++++++++++ docs/getting_started.md | 50 +++++++++++++++++++++++++++++++---------- 2 files changed, 56 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 7404695..11ac132 100644 --- a/README.md +++ b/README.md @@ -37,6 +37,8 @@ glomap mapper --database_path DATABASE_PATH --output_path OUTPUT_PATH --image_pa ``` For more details on the command line interface, one can type `glomap -h` or `glomap mapper -h` for help. +We also provide a guide on improving the obtained reconstruction, which can be found [here](docs/getting_started.md) + Note: - GLOMAP depends on two external libraries - [COLMAP](https://github.com/colmap/colmap) and [PoseLib](https://github.com/PoseLib/PoseLib). With the default setting, the library is built automatically by GLOMAP via `FetchContent`. @@ -80,6 +82,22 @@ glomap mapper \ --output_path ./output/south-building/sparse ``` +### Visualize and use the results + +The results are written out in the COLMAP sparse reconstruction format. Please +refer to [COLMAP](https://colmap.github.io/format.html#sparse-reconstruction) +for more details. + +The reconstruction can be visualized using the COLMAP GUI, for example: +```shell +colmap gui --import_path ./output/south-building/sparse/0 +``` +Alternatives like [rerun.io](https://rerun.io/examples/3d-reconstruction/glomap) +also enable visualization of COLMAP and GLOMAP outputs. + +If you want to inspect the reconstruction programmatically, you can use +`pycolmap` in Python or link against COLMAP's C++ library interface. + ### Notes - For larger scale datasets, it is recommended to use `sequential_matcher` or diff --git a/docs/getting_started.md b/docs/getting_started.md index 653b436..6ae0651 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -1,20 +1,46 @@ # Getting started -### Installation and End-to-End Examples -Please refer to the main `README.md` -### Recommended practice +## Installation and end-to-end examples + +Please refer to the main `README.md`. + +## Recommended settings + The default parameters do not always gaurantee satisfying reconstructions. -Regarding this, there are several things which can generally help +Regarding this, there are several things which can generally help. + +### Share camera parameters + +If images are known to be taken with the same physical camera under identical +camera settings, or images are well organized and known to be taken by several +cameras, it is higly recommended to share the camera intrinsics as appropriate. +To achieve this, one can set `--ImageReader.single_camera_per_folder` or +`--ImageReader.single_camera_per_image` in `colmap feature_extractor`. -#### Share camera parameters as much as possible -If images are known to be taken with the same camera, or images are well organized and known to be taken by several cameras, it is higly recommended to share the camera intrinsics -To achieve this, one can set `--ImageReader.single_camera_per_folder` or `--ImageReader.single_camera_per_image` in `colmap feature_extractor` to be 1. +### Handle high-resolution or blurry images -#### Allow larger epipolar error -If images are of high resolution, or are blurry, it is worth trying to increase the allowed epipolar error by modifying `--RelPoseEstimation.max_epipolar_error`. For example, make it 4, or 10. +If images have high resolution or are blurry, it is worth trying to increase the +allowed epipolar error by modifying `--RelPoseEstimation.max_epipolar_error`. +For example, increase it to 4 or 10. + +### Speedup reconstruction process #### Cap the number of tracks -If the number of images and points are large, the run-time of global bundle adjustment can be long. In this case, to further speed up the overall reconstruction process, the total number of points can be capped, by changing `--TrackEstablishment.max_num_tracks`. Typically, one image should not need more than 1000 tracks to achieve good performance, so this number can be adjusted to $1000 \times n$. -Afterwards, if a full point cloud is desired (for example, to initialize a Gaussian Splatting), points can be triangulated directly by calling `colmap point_triangulator`. -Note, if the `--skip_retriangulation` is not set true when calling `glomap mapper`, retriangulation should already been performed. +If the number of images and points are large, the run-time of global bundle +adjustment can be long. In this case, to further speed up the overall +reconstruction process, the total number of points can be capped, by changing +`--TrackEstablishment.max_num_tracks`. Typically, one image should not need more +than 1000 tracks to achieve good performance, so this number can be adjusted to +$1000 \times n$. Afterwards, if a full point cloud is desired (for example, to +initialize a Gaussian Splatting), points can be triangulated directly by calling +`colmap point_triangulator`. + +Note, if the `--skip_retriangulation` is not set when calling `glomap mapper`, +retriangulation should already been performed. + +#### Limit optimization iterations + +The number of global positioning and bundle adjustment iterations can be limited +using the `--GlobalPositioning.max_num_iterations` and +`--BundleAdjustment.max_num_iterations` options. From 42dc2efb773aa102adcf865c977f5e66382d979f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Johannes=20Sch=C3=B6nberger?= Date: Fri, 9 Aug 2024 19:49:43 +0200 Subject: [PATCH 02/29] Document pre-compiled Windows binary source (#52) --- README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 11ac132..e5f4455 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ If you use this project for your research, please cite ## Getting Started -To install GLOMAP, first install [COLMAP](https://colmap.github.io/install.html#build-from-source) +To build GLOMAP, first install [COLMAP](https://colmap.github.io/install.html#build-from-source) dependencies and then build GLOMAP using the following commands: ```shell mkdir build @@ -31,6 +31,9 @@ cd build cmake .. -GNinja ninja && ninja install ``` +Pre-compiled Windows binaries can be downloaded from the official +[release page](https://github.com/colmap/glomap/releases). + After installation, one can run GLOMAP by (starting from a database) ```shell glomap mapper --database_path DATABASE_PATH --output_path OUTPUT_PATH --image_path IMAGE_PATH From e062598b7cc136a9867bd8ba4532909420adf722 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Johannes=20Sch=C3=B6nberger?= Date: Mon, 12 Aug 2024 09:43:34 +0200 Subject: [PATCH 03/29] Update PoseLib to support OPENCV_FISHEYE camera model (#54) --- cmake/FindDependencies.cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/FindDependencies.cmake b/cmake/FindDependencies.cmake index 824f8b0..2425a52 100644 --- a/cmake/FindDependencies.cmake +++ b/cmake/FindDependencies.cmake @@ -27,7 +27,7 @@ endif() include(FetchContent) FetchContent_Declare(PoseLib GIT_REPOSITORY https://github.com/PoseLib/PoseLib.git - GIT_TAG b3691b791bcedccd5451621b2275a1df0d9dcdeb + GIT_TAG 0439b2d361125915b8821043fca9376e6cc575b9 EXCLUDE_FROM_ALL ) message(STATUS "Configuring PoseLib...") From 18c750806399bf5ad6c0fe2e52ce321100b971ff Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Johannes=20Sch=C3=B6nberger?= Date: Mon, 12 Aug 2024 13:59:59 +0200 Subject: [PATCH 04/29] Fix invalid reference to scales auxiliary variable in global positioning (#59) * Fix invalid reference to scales auxiliary variable in global positioning * d --- glomap/estimators/global_positioning.cc | 58 +++++++++++++++---------- glomap/estimators/global_positioning.h | 10 ++--- 2 files changed, 41 insertions(+), 27 deletions(-) diff --git a/glomap/estimators/global_positioning.cc b/glomap/estimators/global_positioning.cc index e693674..0b94dd0 100644 --- a/glomap/estimators/global_positioning.cc +++ b/glomap/estimators/global_positioning.cc @@ -42,8 +42,8 @@ bool GlobalPositioner::Solve(const ViewGraph& view_graph, LOG(INFO) << "Setting up the global positioner problem"; - // Initialize the problem - Reset(); + // Setup the problem. + SetupProblem(view_graph, tracks); // Initialize camera translations to be random. // Also, convert the camera pose translation to be the camera center. @@ -81,11 +81,24 @@ bool GlobalPositioner::Solve(const ViewGraph& view_graph, return summary.IsSolutionUsable(); } -void GlobalPositioner::Reset() { +void GlobalPositioner::SetupProblem( + const ViewGraph& view_graph, + const std::unordered_map& tracks) { ceres::Problem::Options problem_options; problem_options.loss_function_ownership = ceres::DO_NOT_TAKE_OWNERSHIP; problem_ = std::make_unique(problem_options); + // Allocate enough memory for the scales. One for each residual. + // Due to possibly invalid image pairs or tracks, the actual number of + // residuals may be smaller. scales_.clear(); + scales_.reserve( + view_graph.image_pairs.size() + + std::accumulate(tracks.begin(), + tracks.end(), + 0, + [](int sum, const std::pair& track) { + return sum + track.second.observations.size(); + })); } void GlobalPositioner::InitializeRandomPositions( @@ -145,8 +158,9 @@ void GlobalPositioner::AddCameraToCameraConstraints( images.find(image_id2) == images.end()) continue; - track_t counter = scales_.size(); - scales_.insert(std::make_pair(counter, 1)); + CHECK_GT(scales_.capacity(), scales_.size()) + << "Not enough capacity was reserved for the scales."; + double& scale = scales_.emplace_back(1); Eigen::Vector3d translation = -(images[image_id2].cam_from_world.rotation.inverse() * @@ -158,9 +172,9 @@ void GlobalPositioner::AddCameraToCameraConstraints( options_.loss_function.get(), images[image_id1].cam_from_world.translation.data(), images[image_id2].cam_from_world.translation.data(), - &(scales_[counter])); + &scale); - problem_->SetParameterLowerBound(&(scales_[counter]), 0, 1e-5); + problem_->SetParameterLowerBound(&scale, 0, 1e-5); } if (options_.verbose) @@ -246,14 +260,14 @@ void GlobalPositioner::AddTrackToProblem( ceres::CostFunction* cost_function = BATAPairwiseDirectionError::Create(translation); - track_t counter = scales_.size(); - if (options_.generate_scales || !tracks[track_id].is_initialized) { - scales_.insert(std::make_pair(counter, 1)); - } else { - Eigen::Vector3d trans_calc = + CHECK_GT(scales_.capacity(), scales_.size()) + << "Not enough capacity was reserved for the scales."; + double& scale = scales_.emplace_back(1); + if (!options_.generate_scales && tracks[track_id].is_initialized) { + const Eigen::Vector3d trans_calc = tracks[track_id].xyz - image.cam_from_world.translation; - double scale = translation.dot(trans_calc) / trans_calc.squaredNorm(); - scales_.insert(std::make_pair(counter, std::max(scale, 1e-5))); + scale = std::max(1e-5, + translation.dot(trans_calc) / trans_calc.squaredNorm()); } // For calibrated and uncalibrated cameras, use different loss functions @@ -263,14 +277,14 @@ void GlobalPositioner::AddTrackToProblem( loss_function_ptcam_calibrated_.get(), image.cam_from_world.translation.data(), tracks[track_id].xyz.data(), - &(scales_[counter])) + &scale) : problem_->AddResidualBlock(cost_function, loss_function_ptcam_uncalibrated_.get(), image.cam_from_world.translation.data(), tracks[track_id].xyz.data(), - &(scales_[counter])); + &scale); - problem_->SetParameterLowerBound(&(scales_[counter]), 0, 1e-5); + problem_->SetParameterLowerBound(&scale, 0, 1e-5); } } @@ -284,8 +298,8 @@ void GlobalPositioner::AddCamerasAndPointsToParameterGroups( options_.solver_options.linear_solver_ordering.get(); // Add scale parameters to group 0 (large and independent) - for (auto& [i, scale] : scales_) { - parameter_ordering->AddElementToGroup(&(scales_[i]), 0); + for (double& scale : scales_) { + parameter_ordering->AddElementToGroup(&scale, 0); } // Add point parameters to group 1. @@ -332,8 +346,8 @@ void GlobalPositioner::ParameterizeVariables( // If do not optimize the scales, set the scales to be constant if (!options_.optimize_scales) { - for (auto& [i, scale] : scales_) { - problem_->SetParameterBlockConstant(&(scales_[i])); + for (double& scale : scales_) { + problem_->SetParameterBlockConstant(&scale); } } @@ -358,4 +372,4 @@ void GlobalPositioner::ConvertResults( } } -} // namespace glomap \ No newline at end of file +} // namespace glomap diff --git a/glomap/estimators/global_positioning.h b/glomap/estimators/global_positioning.h index 24260dc..6a9b323 100644 --- a/glomap/estimators/global_positioning.h +++ b/glomap/estimators/global_positioning.h @@ -61,8 +61,8 @@ class GlobalPositioner { GlobalPositionerOptions& GetOptions() { return options_; } protected: - // Reset the problem - void Reset(); + void SetupProblem(const ViewGraph& view_graph, + const std::unordered_map& tracks); // Initialize all cameras to be random. void InitializeRandomPositions(const ViewGraph& view_graph, @@ -98,17 +98,17 @@ class GlobalPositioner { // center Convert the results back to camera poses void ConvertResults(std::unordered_map& images); - // Data members GlobalPositionerOptions options_; std::mt19937 random_generator_; std::unique_ptr problem_; - // loss functions for reweighted terms + // Loss functions for reweighted terms. std::shared_ptr loss_function_ptcam_uncalibrated_; std::shared_ptr loss_function_ptcam_calibrated_; - std::unordered_map scales_; + // Auxiliary scale variables. + std::vector scales_; }; } // namespace glomap From da77885ed90a8b5ab846ff10bb788662b7d7c244 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Johannes=20Sch=C3=B6nberger?= Date: Mon, 12 Aug 2024 15:10:05 +0200 Subject: [PATCH 05/29] Robustly handle undistortion failures (#58) * Robustly handle undistortion failures * f * d * f --------- Co-authored-by: Linfei Pan --- glomap/estimators/cost_function.h | 20 ++++------- glomap/estimators/global_positioning.cc | 46 ++++++++++++++++--------- glomap/estimators/global_positioning.h | 2 +- glomap/estimators/relpose_estimation.cc | 25 +++++++++----- glomap/processors/track_filter.cc | 2 +- glomap/scene/image.h | 7 ++-- 6 files changed, 57 insertions(+), 45 deletions(-) diff --git a/glomap/estimators/cost_function.h b/glomap/estimators/cost_function.h index d0a8ce4..64c3e46 100644 --- a/glomap/estimators/cost_function.h +++ b/glomap/estimators/cost_function.h @@ -14,7 +14,7 @@ namespace glomap { // from two positions such that t_ij - scale * (c_j - c_i) is minimized. struct BATAPairwiseDirectionError { BATAPairwiseDirectionError(const Eigen::Vector3d& translation_obs) - : translation_obs_(translation_obs){}; + : translation_obs_(translation_obs) {} // The error is given by the position error described above. template @@ -22,19 +22,11 @@ struct BATAPairwiseDirectionError { const T* position2, const T* scale, T* residuals) const { - Eigen::Matrix translation; - translation[0] = position2[0] - position1[0]; - translation[1] = position2[1] - position1[1]; - translation[2] = position2[2] - position1[2]; - - Eigen::Matrix residual_vec; - - residual_vec = translation_obs_.cast() - scale[0] * translation; - - residuals[0] = residual_vec(0); - residuals[1] = residual_vec(1); - residuals[2] = residual_vec(2); - + Eigen::Map> residuals_vec(residuals); + residuals_vec = + translation_obs_.cast() - + scale[0] * (Eigen::Map>(position2) - + Eigen::Map>(position1)); return true; } diff --git a/glomap/estimators/global_positioning.cc b/glomap/estimators/global_positioning.cc index 0b94dd0..5590903 100644 --- a/glomap/estimators/global_positioning.cc +++ b/glomap/estimators/global_positioning.cc @@ -155,14 +155,15 @@ void GlobalPositioner::AddCameraToCameraConstraints( const image_t image_id1 = image_pair.image_id1; const image_t image_id2 = image_pair.image_id2; if (images.find(image_id1) == images.end() || - images.find(image_id2) == images.end()) + images.find(image_id2) == images.end()) { continue; + } CHECK_GT(scales_.capacity(), scales_.size()) << "Not enough capacity was reserved for the scales."; double& scale = scales_.emplace_back(1); - Eigen::Vector3d translation = + const Eigen::Vector3d translation = -(images[image_id2].cam_from_world.rotation.inverse() * image_pair.cam2_from_cam1.translation); ceres::CostFunction* cost_function = @@ -244,7 +245,7 @@ void GlobalPositioner::AddPointToCameraConstraints( } void GlobalPositioner::AddTrackToProblem( - const track_t& track_id, + track_t track_id, std::unordered_map& cameras, std::unordered_map& images, std::unordered_map& tracks) { @@ -255,8 +256,19 @@ void GlobalPositioner::AddTrackToProblem( Image& image = images[observation.first]; if (!image.is_registered) continue; - Eigen::Vector3d translation = image.cam_from_world.rotation.inverse() * - image.features_undist[observation.second]; + const Eigen::Vector3d& feature_undist = + image.features_undist[observation.second]; + if (feature_undist.array().isNaN().any()) { + LOG(WARNING) + << "Ignoring feature because it failed to undistort: track_id=" + << track_id << ", image_id=" << observation.first + << ", feature_id=" << observation.second; + continue; + } + + const Eigen::Vector3d translation = + image.cam_from_world.rotation.inverse() * + image.features_undist[observation.second]; ceres::CostFunction* cost_function = BATAPairwiseDirectionError::Create(translation); @@ -272,17 +284,19 @@ void GlobalPositioner::AddTrackToProblem( // For calibrated and uncalibrated cameras, use different loss functions // Down weight the uncalibrated cameras - (cameras[image.camera_id].has_prior_focal_length) - ? problem_->AddResidualBlock(cost_function, - loss_function_ptcam_calibrated_.get(), - image.cam_from_world.translation.data(), - tracks[track_id].xyz.data(), - &scale) - : problem_->AddResidualBlock(cost_function, - loss_function_ptcam_uncalibrated_.get(), - image.cam_from_world.translation.data(), - tracks[track_id].xyz.data(), - &scale); + if (cameras[image.camera_id].has_prior_focal_length) { + problem_->AddResidualBlock(cost_function, + loss_function_ptcam_calibrated_.get(), + image.cam_from_world.translation.data(), + tracks[track_id].xyz.data(), + &scale); + } else { + problem_->AddResidualBlock(cost_function, + loss_function_ptcam_uncalibrated_.get(), + image.cam_from_world.translation.data(), + tracks[track_id].xyz.data(), + &scale); + } problem_->SetParameterLowerBound(&scale, 0, 1e-5); } diff --git a/glomap/estimators/global_positioning.h b/glomap/estimators/global_positioning.h index 6a9b323..0eb1185 100644 --- a/glomap/estimators/global_positioning.h +++ b/glomap/estimators/global_positioning.h @@ -80,7 +80,7 @@ class GlobalPositioner { std::unordered_map& tracks); // Add a single track to the problem - void AddTrackToProblem(const track_t& track_id, + void AddTrackToProblem(track_t track_id, std::unordered_map& cameras, std::unordered_map& images, std::unordered_map& tracks); diff --git a/glomap/estimators/relpose_estimation.cc b/glomap/estimators/relpose_estimation.cc index 89bdab3..c53f572 100644 --- a/glomap/estimators/relpose_estimation.cc +++ b/glomap/estimators/relpose_estimation.cc @@ -47,15 +47,22 @@ void EstimateRelativePoses(ViewGraph& view_graph, inliers.clear(); poselib::CameraPose pose_rel_calc; - poselib::estimate_relative_pose( - points2D_1, - points2D_2, - ColmapCameraToPoseLibCamera(cameras[image1.camera_id]), - ColmapCameraToPoseLibCamera(cameras[image2.camera_id]), - options.ransac_options, - options.bundle_options, - &pose_rel_calc, - &inliers); + try { + poselib::estimate_relative_pose( + points2D_1, + points2D_2, + ColmapCameraToPoseLibCamera(cameras[image1.camera_id]), + ColmapCameraToPoseLibCamera(cameras[image2.camera_id]), + options.ransac_options, + options.bundle_options, + &pose_rel_calc, + &inliers); + } catch (const std::exception& e) { + LOG(ERROR) << "Error in relative pose estimation: " << e.what(); + image_pair.is_valid = false; + continue; + } + // Convert the relative pose to the glomap format for (int i = 0; i < 4; i++) { image_pair.cam2_from_cam1.rotation.coeffs()[i] = diff --git a/glomap/processors/track_filter.cc b/glomap/processors/track_filter.cc index 226af02..c3a78f7 100644 --- a/glomap/processors/track_filter.cc +++ b/glomap/processors/track_filter.cc @@ -37,7 +37,7 @@ int TrackFilter::FilterTracksByReprojection( // If the reprojection error is smaller than the threshold, then keep it if (reprojection_error < max_reprojection_error) { - observation_new.emplace_back(std::make_pair(image_id, feature_id)); + observation_new.emplace_back(image_id, feature_id); } } if (observation_new.size() != track.observations.size()) { diff --git a/glomap/scene/image.h b/glomap/scene/image.h index a191371..1497efa 100644 --- a/glomap/scene/image.h +++ b/glomap/scene/image.h @@ -47,11 +47,10 @@ struct Image { // Gravity information GravityInfo gravity_info; - // Features + // Distorted feature points in pixels. std::vector features; - std::vector - features_undist; // store the normalized features, can be obtained by - // calling UndistortImages + // Normalized feature rays, can be obtained by calling UndistortImages. + std::vector features_undist; // Methods inline Eigen::Vector3d Center() const; From 8ba27074df20250cc38731c0ca4cc223e19312fb Mon Sep 17 00:00:00 2001 From: Asad Mehboob Ali <86931093+Asadali242@users.noreply.github.com> Date: Wed, 14 Aug 2024 05:34:32 -0400 Subject: [PATCH 06/29] Add complete installation guide for COLMAP and GLOMAP on macOS (#63) * Add complete installation guide for COLMAP and GLOMAP on macOS * Update INSTALL_MAC.md --------- Co-authored-by: Linfei Pan <36349740+lpanaf@users.noreply.github.com> --- docs/INSTALL_MAC.md | 193 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 193 insertions(+) create mode 100644 docs/INSTALL_MAC.md diff --git a/docs/INSTALL_MAC.md b/docs/INSTALL_MAC.md new file mode 100644 index 0000000..638f574 --- /dev/null +++ b/docs/INSTALL_MAC.md @@ -0,0 +1,193 @@ +We would like to thank [Asadali242](https://github.com/Asadali242) for providing the installation guide for MAC. +Leave your comments at [Issue #62](https://github.com/colmap/glomap/issues/62) if you encounter any problems. + +## Installing the COLMAP: + +*1. Open the Terminal and install the brew dependencies:* +``` +brew install \ +cmake \ +ninja \ +boost \ +eigen \ +flann \ +libomp \ #(Install Libomp as well) +freeimage \ +metis \ +glog \ +googletest \ +ceres-solver \ +qt@5 \ +glew \ +cgal \ +sqlite3 +``` + +*2. Clone the COLMAP repository:* +``` +git clone https://github.com/colmap/colmap.git +cd colmap +``` + +*3. Ensure Qt5 is in your PATH:* +``` +export PATH="/opt/homebrew/opt/qt@5/bin:$PATH" +``` + +*4. Create a build directory:* +``` +mkdir build +cd build +``` + +*5. After installing, link Qt5 to make sure it’s accessible:* +``` +brew link qt@5 --force +``` + +*6. Run CMake with the specific paths for ARM Mac(M1 and above):* +``` +cmake .. -GNinja \ + -DCMAKE_PREFIX_PATH="/opt/homebrew/opt/flann;/opt/homebrew/opt/metis;/opt/homebrew/opt/suite-sparse;/opt/homebrew/opt/qt@5;/opt/homebrew/opt/freeimage" +``` + +*7. Build and install COLMAP:* +``` +ninja +sudo ninja install +``` + +*8. Confirm COLMAP installation by running:* +``` +colmap -h +colmap gui +``` + +**This is the first part and will install Colmap on your device.** + + + +## Installing the GLOMAP: + +*1. Clone the GitHub repository:* +``` +git clone https://github.com/colmap/glomap +cd glomap +``` + +*2. Create a build directory:* +``` +mkdir build +cd build +``` + +*3. Export Environment Variables Again:* +``` +export PATH="/opt/homebrew/opt/qt@5/bin:$PATH" +export LDFLAGS="-L/opt/homebrew/opt/libomp/lib" +export CPPFLAGS="-I/opt/homebrew/opt/libomp/include" +export CMAKE_PREFIX_PATH="/opt/homebrew/opt/qt@5;/opt/homebrew/opt/libomp" +export PKG_CONFIG_PATH="/opt/homebrew/opt/qt@5/lib/pkgconfig" +``` + +*4. Run the CMake Command:* +``` +cmake -DCMAKE_PREFIX_PATH="/opt/homebrew/Cellar/qt@5/5.15.13_1;/opt/homebrew/opt/libomp" \ +-DOpenMP_C_FLAGS="-Xclang -fopenmp" \ +-DOpenMP_C_LIB_NAMES="libomp" \ +-DOpenMP_CXX_FLAGS="-Xclang -fopenmp" \ +-DOpenMP_CXX_LIB_NAMES="libomp" \ +-DOpenMP_C_INCLUDE_DIRS="/opt/homebrew/opt/libomp/include" \ +-DOpenMP_CXX_INCLUDE_DIRS="/opt/homebrew/opt/libomp/include" \ +-DOpenMP_libomp_LIBRARY=/opt/homebrew/opt/libomp/lib/libomp.dylib \ +-DOpenMP_INCLUDE_DIR=/opt/homebrew/opt/libomp/include \ +.. -GNinja +``` + +*5. Build the Project:* +``` +ninja +``` + +**NOTE: If at this point, there are build errors related to ‘cholmod.h’ or ‘omp.h’, clean the build and then re-run the make with the following commands:** +``` +cmake -DCMAKE_PREFIX_PATH="/opt/homebrew/Cellar/qt@5/5.15.13_1;/opt/homebrew/opt/libomp" \ +-DOpenMP_C_FLAGS="-Xclang -fopenmp -I/opt/homebrew/opt/libomp/include" \ +-DOpenMP_C_LIB_NAMES="libomp" \ +-DOpenMP_CXX_FLAGS="-Xclang -fopenmp -I/opt/homebrew/opt/libomp/include" \ +-DOpenMP_CXX_LIB_NAMES="libomp" \ +-DOpenMP_libomp_LIBRARY=/opt/homebrew/opt/libomp/lib/libomp.dylib \ +.. -GNinja +``` + +**After the build is successful:** + +*6. Install the Built Project:* +``` +sudo ninja install +``` + +*7. Test the Installation:* +``` +glomap -h +``` + +**It should display something like:** +``` +GLOMAP -- Global Structure-from-Motion +Usage: +glomap mapper --database_path DATABASE --output_path +MODEL +glomap mapper_resume --input_path MODEL_INPUT --output_path MODEL_OUTPUT +Available commands: +help +mapper +mapper_resume +``` + + +## Testing with the end-to-end examples provided: + +*1. Open the end-to-end example database link provided:* +https://lpanaf.github.io/eccv24_glomap/ + +*2. Download one of the datasets provided and extract the zip file.* + +*3. Create a new directory named ‘data’ in the root directory ‘glomap’.* + +*4. Place the extracted dataset in the directory ‘data’.* + +*5. Now navigate back to the project directory ‘glomap’.* + +**NOTE: Following commands are assuming the dataset to be south-building:** + +*6. Extract Features with COLMAP:* +``` +colmap feature_extractor \ +--image_path ./data/south-building/images \ +--database_path ./data/south-building/database.db +``` + +*7. Match Features with COLMAP:* +``` +colmap exhaustive_matcher \ +--database_path ./data/south-building/database.db +``` + +*8. Run GLOMAP Mapper:* +``` +glomap mapper \ +--database_path ./data/south-building/database.db \ +--image_path ./data/south-building/images \ +--output_path ./output/south-building/sparse +``` + +**This should generate a new directory named ‘output’ in the project directory.** + +*9. Visualize the Results:* +``` +colmap gui \ +--database_path ./data/south-building/database.db \ +--image_path ./data/south-building/images \ +--import_path ./output/south-building/sparse/0 +``` From 4210f9392567c4f38dfd1300db0126b69606fc0e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Johannes=20Sch=C3=B6nberger?= Date: Sun, 25 Aug 2024 20:20:44 +0200 Subject: [PATCH 07/29] Enable verbose logging options through command-line flags (#77) --- glomap/controllers/global_mapper_test.cc | 7 ------ glomap/controllers/option_manager.cc | 3 +++ glomap/estimators/bundle_adjustment.cc | 4 ++-- glomap/estimators/global_positioning.cc | 24 ++++++++----------- .../estimators/global_rotation_averaging.cc | 22 +++++++---------- glomap/estimators/global_rotation_averaging.h | 3 --- glomap/estimators/optimization_base.h | 5 +--- glomap/estimators/view_graph_calibration.cc | 17 ++++--------- 8 files changed, 29 insertions(+), 56 deletions(-) diff --git a/glomap/controllers/global_mapper_test.cc b/glomap/controllers/global_mapper_test.cc index 044c917..b1a4062 100644 --- a/glomap/controllers/global_mapper_test.cc +++ b/glomap/controllers/global_mapper_test.cc @@ -40,13 +40,6 @@ void ExpectEqualReconstructions(const colmap::Reconstruction& gt, GlobalMapperOptions CreateTestOptions() { GlobalMapperOptions options; - // Control the verbosity of the global sfm - options.opt_vgcalib.verbose = false; - options.opt_ra.verbose = false; - options.opt_gp.verbose = false; - options.opt_ba.verbose = false; - - // Control the flow of the global sfm options.skip_view_graph_calibration = false; options.skip_relative_pose_estimation = false; options.skip_rotation_averaging = false; diff --git a/glomap/controllers/option_manager.cc b/glomap/controllers/option_manager.cc index 39c17f8..d1e0304 100644 --- a/glomap/controllers/option_manager.cc +++ b/glomap/controllers/option_manager.cc @@ -17,6 +17,9 @@ OptionManager::OptionManager(bool add_project_options) { Reset(); desc_->add_options()("help,h", ""); + + AddAndRegisterDefaultOption("log_to_stderr", &FLAGS_logtostderr); + AddAndRegisterDefaultOption("log_level", &FLAGS_v); } void OptionManager::AddAllOptions() { diff --git a/glomap/estimators/bundle_adjustment.cc b/glomap/estimators/bundle_adjustment.cc index a03bea5..32204ba 100644 --- a/glomap/estimators/bundle_adjustment.cc +++ b/glomap/estimators/bundle_adjustment.cc @@ -40,9 +40,9 @@ bool BundleAdjuster::Solve(const ViewGraph& view_graph, options_.solver_options.linear_solver_type = ceres::SPARSE_SCHUR; options_.solver_options.preconditioner_type = ceres::CLUSTER_TRIDIAGONAL; - options_.solver_options.minimizer_progress_to_stdout = options_.verbose; + options_.solver_options.minimizer_progress_to_stdout = VLOG_IS_ON(2); ceres::Solve(options_.solver_options, problem_.get(), &summary); - if (options_.verbose) + if (VLOG_IS_ON(2)) LOG(INFO) << summary.FullReport(); else LOG(INFO) << summary.BriefReport(); diff --git a/glomap/estimators/global_positioning.cc b/glomap/estimators/global_positioning.cc index 5590903..b18fa80 100644 --- a/glomap/estimators/global_positioning.cc +++ b/glomap/estimators/global_positioning.cc @@ -68,10 +68,10 @@ bool GlobalPositioner::Solve(const ViewGraph& view_graph, LOG(INFO) << "Solving the global positioner problem"; ceres::Solver::Summary summary; - options_.solver_options.minimizer_progress_to_stdout = options_.verbose; + options_.solver_options.minimizer_progress_to_stdout = VLOG_IS_ON(2); ceres::Solve(options_.solver_options, problem_.get(), &summary); - if (options_.verbose) { + if (VLOG_IS_ON(2)) { LOG(INFO) << summary.FullReport(); } else { LOG(INFO) << summary.BriefReport(); @@ -143,8 +143,7 @@ void GlobalPositioner::InitializeRandomPositions( image.cam_from_world.translation = image.Center(); } - if (options_.verbose) - LOG(INFO) << "Constrained positions: " << constrained_positions.size(); + VLOG(2) << "Constrained positions: " << constrained_positions.size(); } void GlobalPositioner::AddCameraToCameraConstraints( @@ -178,10 +177,9 @@ void GlobalPositioner::AddCameraToCameraConstraints( problem_->SetParameterLowerBound(&scale, 0, 1e-5); } - if (options_.verbose) - LOG(INFO) << problem_->NumResidualBlocks() - << " camera to camera constraints were added to the position " - "estimation problem."; + VLOG(2) << problem_->NumResidualBlocks() + << " camera to camera constraints were added to the position " + "estimation problem."; } void GlobalPositioner::AddPointToCameraConstraints( @@ -194,10 +192,9 @@ void GlobalPositioner::AddPointToCameraConstraints( // Find the tracks that are relevant to the current set of cameras const size_t num_pt_to_cam = tracks.size(); - if (options_.verbose) - LOG(INFO) << num_pt_to_cam - << " point to camera constriants were added to the position " - "estimation problem."; + VLOG(2) << num_pt_to_cam + << " point to camera constriants were added to the position " + "estimation problem."; if (num_pt_to_cam == 0) return; @@ -211,8 +208,7 @@ void GlobalPositioner::AddPointToCameraConstraints( static_cast(num_cam_to_cam) / static_cast(num_pt_to_cam); } - if (options_.verbose) - LOG(INFO) << "Point to camera weight scaled: " << weight_scale_pt; + VLOG(2) << "Point to camera weight scaled: " << weight_scale_pt; if (loss_function_ptcam_uncalibrated_ == nullptr) { loss_function_ptcam_uncalibrated_ = diff --git a/glomap/estimators/global_rotation_averaging.cc b/glomap/estimators/global_rotation_averaging.cc index 429a6bd..3f7d031 100644 --- a/glomap/estimators/global_rotation_averaging.cc +++ b/glomap/estimators/global_rotation_averaging.cc @@ -156,10 +156,6 @@ void RotationEstimator::SetupLinearSystem( } } - if (options_.verbose) - LOG(INFO) << "num_img: " << image_id_to_idx_.size() - << ", num_dof: " << num_dof; - rotation_estimated_.conservativeResize(num_dof); // Prepare the relative information @@ -204,8 +200,7 @@ void RotationEstimator::SetupLinearSystem( } } - if (options_.verbose) - LOG(INFO) << counter << " image pairs are gravity aligned" << std::endl; + VLOG(2) << counter << " image pairs are gravity aligned" << std::endl; std::vector> coeffs; coeffs.reserve(rel_temp_info_.size() * 6 + 3); @@ -290,11 +285,11 @@ bool RotationEstimator::SolveL1Regression( double curr_norm = 0; ComputeResiduals(view_graph, images); - if (options_.verbose) LOG(INFO) << "ComputeResiduals done "; + VLOG(2) << "ComputeResiduals done"; int iteration = 0; for (iteration = 0; iteration < options_.max_num_l1_iterations; iteration++) { - if (options_.verbose) LOG(INFO) << "L1 ADMM iteration: " << iteration; + VLOG(2) << "L1 ADMM iteration: " << iteration; last_norm = curr_norm; // use the current residual as b (Ax - b) @@ -307,7 +302,7 @@ bool RotationEstimator::SolveL1Regression( return false; } - if (options_.verbose) + if (VLOG_IS_ON(2)) LOG(INFO) << "residual:" << (sparse_matrix_ * tangent_space_step_ - tangent_space_residual_) @@ -332,7 +327,7 @@ bool RotationEstimator::SolveL1Regression( opt_l1_solver.max_num_iterations = std::min(opt_l1_solver.max_num_iterations * 2, 100); } - if (options_.verbose) LOG(INFO) << "L1 ADMM total iteration: " << iteration; + VLOG(2) << "L1 ADMM total iteration: " << iteration; return true; } @@ -347,8 +342,7 @@ bool RotationEstimator::SolveIRLS(const ViewGraph& view_graph, llt.analyzePattern(sparse_matrix_.transpose() * sparse_matrix_); const double sigma = DegToRad(options_.irls_loss_parameter_sigma); - if (options_.verbose) - LOG(INFO) << "sigma: " << options_.irls_loss_parameter_sigma; + VLOG(2) << "sigma: " << options_.irls_loss_parameter_sigma; Eigen::ArrayXd weights_irls(sparse_matrix_.rows()); Eigen::SparseMatrix at_weight; @@ -362,7 +356,7 @@ bool RotationEstimator::SolveIRLS(const ViewGraph& view_graph, int iteration = 0; for (iteration = 0; iteration < options_.max_num_irls_iterations; iteration++) { - if (options_.verbose) LOG(INFO) << "IRLS iteration: " << iteration; + VLOG(2) << "IRLS iteration: " << iteration; // Compute the weights for IRLS for (auto& [pair_id, pair_info] : rel_temp_info_) { @@ -416,7 +410,7 @@ bool RotationEstimator::SolveIRLS(const ViewGraph& view_graph, break; } } - if (options_.verbose) LOG(INFO) << "IRLS total iteration: " << iteration; + VLOG(2) << "IRLS total iteration: " << iteration; return true; } diff --git a/glomap/estimators/global_rotation_averaging.h b/glomap/estimators/global_rotation_averaging.h index 7b73d34..8899a1c 100644 --- a/glomap/estimators/global_rotation_averaging.h +++ b/glomap/estimators/global_rotation_averaging.h @@ -68,9 +68,6 @@ struct RotationEstimatorOptions { // Flag to use gravity for rotation averaging bool use_gravity = false; - - // Flag whether report the verbose information - bool verbose = false; }; // TODO: Implement the stratified camera rotation estimation diff --git a/glomap/estimators/optimization_base.h b/glomap/estimators/optimization_base.h index 6234ca2..2b43e06 100644 --- a/glomap/estimators/optimization_base.h +++ b/glomap/estimators/optimization_base.h @@ -9,9 +9,6 @@ namespace glomap { struct OptimizationBaseOptions { - // Logging control - bool verbose = false; - // The threshold for the loss function double thres_loss_function = 1e-1; @@ -24,7 +21,7 @@ struct OptimizationBaseOptions { OptimizationBaseOptions() { solver_options.num_threads = std::thread::hardware_concurrency(); solver_options.max_num_iterations = 100; - solver_options.minimizer_progress_to_stdout = verbose; + solver_options.minimizer_progress_to_stdout = false; solver_options.function_tolerance = 1e-5; } }; diff --git a/glomap/estimators/view_graph_calibration.cc b/glomap/estimators/view_graph_calibration.cc index 7991055..7d5b457 100644 --- a/glomap/estimators/view_graph_calibration.cc +++ b/glomap/estimators/view_graph_calibration.cc @@ -36,13 +36,10 @@ bool ViewGraphCalibrator::Solve(ViewGraph& view_graph, // Solve the problem ceres::Solver::Summary summary; - options_.solver_options.minimizer_progress_to_stdout = options_.verbose; + options_.solver_options.minimizer_progress_to_stdout = VLOG_IS_ON(2); ceres::Solve(options_.solver_options, problem_.get(), &summary); - // Print the summary only if verbose - if (options_.verbose) { - LOG(INFO) << summary.FullReport(); - } + VLOG(2) << summary.FullReport(); // Convert the results back to the camera CopyBackResults(cameras); @@ -132,10 +129,9 @@ void ViewGraphCalibrator::CopyBackResults( // if the estimated parameter is too crazy, reject it if (focals_[camera_id] / camera.Focal() > options_.thres_higher_ratio || focals_[camera_id] / camera.Focal() < options_.thres_lower_ratio) { - if (options_.verbose) - LOG(INFO) << "NOT ACCEPTED: Camera " << camera_id - << " focal: " << focals_[camera_id] - << " original focal: " << camera.Focal(); + VLOG(2) << "Ignoring degenerate camera camera " << camera_id + << " focal: " << focals_[camera_id] + << " original focal: " << camera.Focal(); counter++; continue; @@ -147,9 +143,6 @@ void ViewGraphCalibrator::CopyBackResults( // Update the focal length for (const size_t idx : camera.FocalLengthIdxs()) { camera.params[idx] = focals_[camera_id]; - if (options_.verbose) - LOG(INFO) << "Camera " << idx << " focal: " << focals_[camera_id] - << std::endl; } } LOG(INFO) << counter << " cameras are rejected in view graph calibration"; From 7ab1ced40e921d90a8a286884441efaf40575401 Mon Sep 17 00:00:00 2001 From: creeper <104205641+cre185@users.noreply.github.com> Date: Tue, 3 Sep 2024 18:09:26 +0800 Subject: [PATCH 08/29] fix bug caused by negative weight when calculating mst (#90) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix bug caused by negative weight when calculating mst * Apply suggestion Co-authored-by: Johannes Schönberger --------- Co-authored-by: Johannes Schönberger --- glomap/math/tree.cc | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/glomap/math/tree.cc b/glomap/math/tree.cc index 1a0ad4e..5f329e6 100644 --- a/glomap/math/tree.cc +++ b/glomap/math/tree.cc @@ -89,6 +89,16 @@ image_t MaximumSpanningTree(const ViewGraph& view_graph, image_id_to_idx[image_id] = image_id_to_idx.size(); } + double max_weight = 0; + for (const auto& [pair_id, image_pair] : view_graph.image_pairs) { + if (image_pair.is_valid == false) continue; + if (type == INLIER_RATIO) + max_weight = std::max(max_weight, image_pair.weight); + else + max_weight = + std::max(max_weight, static_cast(image_pair.inliers.size())); + } + // establish graph weighted_graph G(image_id_to_idx.size()); weight_map weights_boost = boost::get(boost::edge_weight, G); @@ -111,11 +121,11 @@ image_t MaximumSpanningTree(const ViewGraph& view_graph, // spanning tree e = boost::add_edge(idx1, idx2, G).first; if (type == INLIER_NUM) - weights_boost[e] = -image_pair.inliers.size(); + weights_boost[e] = max_weight - image_pair.inliers.size(); else if (type == INLIER_RATIO) - weights_boost[e] = -image_pair.weight; + weights_boost[e] = max_weight - image_pair.weight; else - weights_boost[e] = -image_pair.inliers.size(); + weights_boost[e] = max_weight - image_pair.inliers.size(); } std::vector @@ -142,4 +152,4 @@ image_t MaximumSpanningTree(const ViewGraph& view_graph, return idx_to_image_id[0]; } -}; // namespace glomap \ No newline at end of file +}; // namespace glomap From 410fc07abd059d9a09259ed4415c4e4f3cbf3480 Mon Sep 17 00:00:00 2001 From: lnex Date: Tue, 10 Sep 2024 00:19:17 +0800 Subject: [PATCH 09/29] fix: arg type for std::ceil and typo in glomap::EstimateRelativePoses (#94) --- glomap/estimators/relpose_estimation.cc | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/glomap/estimators/relpose_estimation.cc b/glomap/estimators/relpose_estimation.cc index c53f572..45f206b 100644 --- a/glomap/estimators/relpose_estimation.cc +++ b/glomap/estimators/relpose_estimation.cc @@ -20,14 +20,15 @@ void EstimateRelativePoses(ViewGraph& view_graph, const int64_t num_image_pairs = valid_pair_ids.size(); const int64_t kNumChunks = 10; - const int64_t inverval = std::ceil(num_image_pairs / kNumChunks); + const int64_t interval = + std::ceil(static_cast(num_image_pairs) / kNumChunks); LOG(INFO) << "Estimating relative pose for " << num_image_pairs << " pairs"; for (int64_t chunk_id = 0; chunk_id < kNumChunks; chunk_id++) { std::cout << "\r Estimating relative pose: " << chunk_id * kNumChunks << "%" << std::flush; - const int64_t start = chunk_id * inverval; + const int64_t start = chunk_id * interval; const int64_t end = - std::min((chunk_id + 1) * inverval, num_image_pairs); + std::min((chunk_id + 1) * interval, num_image_pairs); #pragma omp parallel for schedule(dynamic) private( \ points2D_1, points2D_2, inliers) From 11387c787305722ec8657af4e87c6fd8ec2feb11 Mon Sep 17 00:00:00 2001 From: lnex Date: Sat, 14 Sep 2024 23:12:41 +0800 Subject: [PATCH 10/29] fix: vector out of bounds in ViewGraph::KeepLargestConnectedComponents (#99) * fix: vector out of bounds in ViewGraph::KeepLargestConnectedComponents * fix: return GlobalMapper::Solve when no connected components --- glomap/controllers/global_mapper.cc | 14 ++++++++++++-- glomap/scene/view_graph.cc | 2 ++ 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/glomap/controllers/global_mapper.cc b/glomap/controllers/global_mapper.cc index af43612..22f14c8 100644 --- a/glomap/controllers/global_mapper.cc +++ b/glomap/controllers/global_mapper.cc @@ -63,7 +63,10 @@ bool GlobalMapper::Solve(const colmap::Database& database, RelPoseFilter::FilterInlierRatio( view_graph, options_.inlier_thresholds.min_inlier_ratio); - view_graph.KeepLargestConnectedComponents(images); + if (view_graph.KeepLargestConnectedComponents(images) == 0) { + LOG(ERROR) << "no connected components are found"; + return false; + } run_timer.PrintSeconds(); } @@ -83,7 +86,10 @@ bool GlobalMapper::Solve(const colmap::Database& database, RelPoseFilter::FilterRotations( view_graph, images, options_.inlier_thresholds.max_rotation_error); - view_graph.KeepLargestConnectedComponents(images); + if (view_graph.KeepLargestConnectedComponents(images) == 0) { + LOG(ERROR) << "no connected components are found"; + return false; + } // The second run is for final estimation if (!ra_engine.EstimateRotations(view_graph, images)) { @@ -92,6 +98,10 @@ bool GlobalMapper::Solve(const colmap::Database& database, RelPoseFilter::FilterRotations( view_graph, images, options_.inlier_thresholds.max_rotation_error); image_t num_img = view_graph.KeepLargestConnectedComponents(images); + if (num_img == 0) { + LOG(ERROR) << "no connected components are found"; + return false; + } LOG(INFO) << num_img << " / " << images.size() << " images are within the connected component." << std::endl; diff --git a/glomap/scene/view_graph.cc b/glomap/scene/view_graph.cc index d3a540f..0443d90 100644 --- a/glomap/scene/view_graph.cc +++ b/glomap/scene/view_graph.cc @@ -21,6 +21,8 @@ int ViewGraph::KeepLargestConnectedComponents( } } + if (max_img == 0) return 0; + std::unordered_set largest_component = connected_components[max_idx]; // Set all images to not registered From 55e40b11a68b8819e71fdb30773258e20a284451 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Johannes=20Sch=C3=B6nberger?= Date: Thu, 19 Sep 2024 13:41:12 +0200 Subject: [PATCH 11/29] Use colmap thread pool to simplify mac installation (#107) * Use colmap thread pool to simplify mac installation * d * d * d * d --- .github/workflows/mac.yml | 83 ++++++++ .github/workflows/ubuntu.yml | 2 +- .gitignore | 5 +- CMakeLists.txt | 1 - cmake/FindDependencies.cmake | 5 - docs/INSTALL_MAC.md | 193 ------------------- glomap/CMakeLists.txt | 4 - glomap/estimators/relpose_estimation.cc | 89 +++++---- glomap/processors/image_undistorter.cc | 32 +-- glomap/processors/view_graph_manipulation.cc | 80 ++++---- 10 files changed, 198 insertions(+), 296 deletions(-) create mode 100644 .github/workflows/mac.yml delete mode 100644 docs/INSTALL_MAC.md diff --git a/.github/workflows/mac.yml b/.github/workflows/mac.yml new file mode 100644 index 0000000..6cfdbeb --- /dev/null +++ b/.github/workflows/mac.yml @@ -0,0 +1,83 @@ +name: Mac + +on: + push: + branches: + - main + pull_request: + types: [ assigned, opened, synchronize, reopened ] + release: + types: [ published, edited ] + +jobs: + build: + name: ${{ matrix.config.os }} ${{ matrix.config.arch }} ${{ matrix.config.cmakeBuildType }} + runs-on: ${{ matrix.config.os }} + strategy: + matrix: + config: [ + { + os: macos-14, + arch: arm64, + cmakeBuildType: Release, + }, + ] + + env: + COMPILER_CACHE_VERSION: 1 + COMPILER_CACHE_DIR: ${{ github.workspace }}/compiler-cache + CCACHE_DIR: ${{ github.workspace }}/compiler-cache/ccache + CCACHE_BASEDIR: ${{ github.workspace }} + + steps: + - uses: actions/checkout@v4 + - uses: actions/cache@v4 + id: cache-builds + with: + key: v${{ env.COMPILER_CACHE_VERSION }}-${{ matrix.config.os }}-${{ matrix.config.arch }}-${{ matrix.config.cmakeBuildType }}-${{ github.run_id }}-${{ github.run_number }} + restore-keys: v${{ env.COMPILER_CACHE_VERSION }}-${{ matrix.config.os }}-${{ matrix.config.arch }}-${{ matrix.config.cmakeBuildType }} + path: ${{ env.COMPILER_CACHE_DIR }} + + - name: Setup Mac + run: | + brew install \ + cmake \ + ninja \ + boost \ + eigen \ + flann \ + freeimage \ + metis \ + glog \ + googletest \ + ceres-solver \ + qt5 \ + glew \ + cgal \ + sqlite3 \ + ccache + + - name: Configure and build + run: | + cmake --version + mkdir build + cd build + cmake .. \ + -GNinja \ + -DCMAKE_BUILD_TYPE=${{ matrix.config.cmakeBuildType }} \ + -DTESTS_ENABLED=ON \ + -DCMAKE_PREFIX_PATH="$(brew --prefix qt@5)" + ninja + + - name: Run tests + run: | + cd build + set +e + ctest --output-on-failure -E .+colmap_.* + + - name: Cleanup compiler cache + run: | + set -x + ccache --show-stats --verbose + ccache --evict-older-than 1d + ccache --show-stats --verbose diff --git a/.github/workflows/ubuntu.yml b/.github/workflows/ubuntu.yml index 455e8d8..a8246f9 100644 --- a/.github/workflows/ubuntu.yml +++ b/.github/workflows/ubuntu.yml @@ -185,4 +185,4 @@ jobs: # Delete cache older than 10 days. find "$CTCACHE_DIR"/*/ -mtime +10 -print0 | xargs -0 rm -rf echo "Size of ctcache after: $(du -sh $CTCACHE_DIR)" - echo "Number of ctcache files after: $(find $CTCACHE_DIR | wc -l)" \ No newline at end of file + echo "Number of ctcache files after: $(find $CTCACHE_DIR | wc -l)"'' diff --git a/.gitignore b/.gitignore index b4caee1..52abcd9 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,3 @@ -build -data +/build +/data +/.vscode diff --git a/CMakeLists.txt b/CMakeLists.txt index 56919a1..1d1c272 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -7,7 +7,6 @@ set(CMAKE_CXX_STANDARD_REQUIRED ON) set_property(GLOBAL PROPERTY GLOBAL_DEPENDS_NO_CYCLES ON) -option(OPENMP_ENABLED "Whether to enable OpenMP parallelization" ON) option(TESTS_ENABLED "Whether to build test binaries" OFF) option(ASAN_ENABLED "Whether to enable AddressSanitizer flags" OFF) option(CCACHE_ENABLED "Whether to enable compiler caching, if available" ON) diff --git a/cmake/FindDependencies.cmake b/cmake/FindDependencies.cmake index 2425a52..7bcf202 100644 --- a/cmake/FindDependencies.cmake +++ b/cmake/FindDependencies.cmake @@ -19,11 +19,6 @@ if(TESTS_ENABLED) find_package(GTest REQUIRED) endif() -if (OPENMP_ENABLED) - message(STATUS "Enabling OpenMP") - find_package(OpenMP REQUIRED) -endif() - include(FetchContent) FetchContent_Declare(PoseLib GIT_REPOSITORY https://github.com/PoseLib/PoseLib.git diff --git a/docs/INSTALL_MAC.md b/docs/INSTALL_MAC.md deleted file mode 100644 index 638f574..0000000 --- a/docs/INSTALL_MAC.md +++ /dev/null @@ -1,193 +0,0 @@ -We would like to thank [Asadali242](https://github.com/Asadali242) for providing the installation guide for MAC. -Leave your comments at [Issue #62](https://github.com/colmap/glomap/issues/62) if you encounter any problems. - -## Installing the COLMAP: - -*1. Open the Terminal and install the brew dependencies:* -``` -brew install \ -cmake \ -ninja \ -boost \ -eigen \ -flann \ -libomp \ #(Install Libomp as well) -freeimage \ -metis \ -glog \ -googletest \ -ceres-solver \ -qt@5 \ -glew \ -cgal \ -sqlite3 -``` - -*2. Clone the COLMAP repository:* -``` -git clone https://github.com/colmap/colmap.git -cd colmap -``` - -*3. Ensure Qt5 is in your PATH:* -``` -export PATH="/opt/homebrew/opt/qt@5/bin:$PATH" -``` - -*4. Create a build directory:* -``` -mkdir build -cd build -``` - -*5. After installing, link Qt5 to make sure it’s accessible:* -``` -brew link qt@5 --force -``` - -*6. Run CMake with the specific paths for ARM Mac(M1 and above):* -``` -cmake .. -GNinja \ - -DCMAKE_PREFIX_PATH="/opt/homebrew/opt/flann;/opt/homebrew/opt/metis;/opt/homebrew/opt/suite-sparse;/opt/homebrew/opt/qt@5;/opt/homebrew/opt/freeimage" -``` - -*7. Build and install COLMAP:* -``` -ninja -sudo ninja install -``` - -*8. Confirm COLMAP installation by running:* -``` -colmap -h -colmap gui -``` - -**This is the first part and will install Colmap on your device.** - - - -## Installing the GLOMAP: - -*1. Clone the GitHub repository:* -``` -git clone https://github.com/colmap/glomap -cd glomap -``` - -*2. Create a build directory:* -``` -mkdir build -cd build -``` - -*3. Export Environment Variables Again:* -``` -export PATH="/opt/homebrew/opt/qt@5/bin:$PATH" -export LDFLAGS="-L/opt/homebrew/opt/libomp/lib" -export CPPFLAGS="-I/opt/homebrew/opt/libomp/include" -export CMAKE_PREFIX_PATH="/opt/homebrew/opt/qt@5;/opt/homebrew/opt/libomp" -export PKG_CONFIG_PATH="/opt/homebrew/opt/qt@5/lib/pkgconfig" -``` - -*4. Run the CMake Command:* -``` -cmake -DCMAKE_PREFIX_PATH="/opt/homebrew/Cellar/qt@5/5.15.13_1;/opt/homebrew/opt/libomp" \ --DOpenMP_C_FLAGS="-Xclang -fopenmp" \ --DOpenMP_C_LIB_NAMES="libomp" \ --DOpenMP_CXX_FLAGS="-Xclang -fopenmp" \ --DOpenMP_CXX_LIB_NAMES="libomp" \ --DOpenMP_C_INCLUDE_DIRS="/opt/homebrew/opt/libomp/include" \ --DOpenMP_CXX_INCLUDE_DIRS="/opt/homebrew/opt/libomp/include" \ --DOpenMP_libomp_LIBRARY=/opt/homebrew/opt/libomp/lib/libomp.dylib \ --DOpenMP_INCLUDE_DIR=/opt/homebrew/opt/libomp/include \ -.. -GNinja -``` - -*5. Build the Project:* -``` -ninja -``` - -**NOTE: If at this point, there are build errors related to ‘cholmod.h’ or ‘omp.h’, clean the build and then re-run the make with the following commands:** -``` -cmake -DCMAKE_PREFIX_PATH="/opt/homebrew/Cellar/qt@5/5.15.13_1;/opt/homebrew/opt/libomp" \ --DOpenMP_C_FLAGS="-Xclang -fopenmp -I/opt/homebrew/opt/libomp/include" \ --DOpenMP_C_LIB_NAMES="libomp" \ --DOpenMP_CXX_FLAGS="-Xclang -fopenmp -I/opt/homebrew/opt/libomp/include" \ --DOpenMP_CXX_LIB_NAMES="libomp" \ --DOpenMP_libomp_LIBRARY=/opt/homebrew/opt/libomp/lib/libomp.dylib \ -.. -GNinja -``` - -**After the build is successful:** - -*6. Install the Built Project:* -``` -sudo ninja install -``` - -*7. Test the Installation:* -``` -glomap -h -``` - -**It should display something like:** -``` -GLOMAP -- Global Structure-from-Motion -Usage: -glomap mapper --database_path DATABASE --output_path -MODEL -glomap mapper_resume --input_path MODEL_INPUT --output_path MODEL_OUTPUT -Available commands: -help -mapper -mapper_resume -``` - - -## Testing with the end-to-end examples provided: - -*1. Open the end-to-end example database link provided:* -https://lpanaf.github.io/eccv24_glomap/ - -*2. Download one of the datasets provided and extract the zip file.* - -*3. Create a new directory named ‘data’ in the root directory ‘glomap’.* - -*4. Place the extracted dataset in the directory ‘data’.* - -*5. Now navigate back to the project directory ‘glomap’.* - -**NOTE: Following commands are assuming the dataset to be south-building:** - -*6. Extract Features with COLMAP:* -``` -colmap feature_extractor \ ---image_path ./data/south-building/images \ ---database_path ./data/south-building/database.db -``` - -*7. Match Features with COLMAP:* -``` -colmap exhaustive_matcher \ ---database_path ./data/south-building/database.db -``` - -*8. Run GLOMAP Mapper:* -``` -glomap mapper \ ---database_path ./data/south-building/database.db \ ---image_path ./data/south-building/images \ ---output_path ./output/south-building/sparse -``` - -**This should generate a new directory named ‘output’ in the project directory.** - -*9. Visualize the Results:* -``` -colmap gui \ ---database_path ./data/south-building/database.db \ ---image_path ./data/south-building/images \ ---import_path ./output/south-building/sparse/0 -``` diff --git a/glomap/CMakeLists.txt b/glomap/CMakeLists.txt index a5d8244..1325715 100644 --- a/glomap/CMakeLists.txt +++ b/glomap/CMakeLists.txt @@ -84,10 +84,6 @@ target_link_libraries( ) target_include_directories(glomap PUBLIC ..) -if(OPENMP_FOUND) - target_link_libraries(glomap PUBLIC OpenMP::OpenMP_CXX) -endif() - if(MSVC) target_compile_options(glomap PRIVATE /bigobj) else() diff --git a/glomap/estimators/relpose_estimation.cc b/glomap/estimators/relpose_estimation.cc index 45f206b..4676eb2 100644 --- a/glomap/estimators/relpose_estimation.cc +++ b/glomap/estimators/relpose_estimation.cc @@ -1,5 +1,7 @@ #include "glomap/estimators/relpose_estimation.h" +#include + #include namespace glomap { @@ -14,14 +16,13 @@ void EstimateRelativePoses(ViewGraph& view_graph, valid_pair_ids.push_back(image_pair_id); } - // Define outside loop to reuse memory and avoid reallocation. - std::vector points2D_1, points2D_2; - std::vector inliers; - const int64_t num_image_pairs = valid_pair_ids.size(); const int64_t kNumChunks = 10; const int64_t interval = std::ceil(static_cast(num_image_pairs) / kNumChunks); + + colmap::ThreadPool thread_pool(colmap::ThreadPool::kMaxNumThreads); + LOG(INFO) << "Estimating relative pose for " << num_image_pairs << " pairs"; for (int64_t chunk_id = 0; chunk_id < kNumChunks; chunk_id++) { std::cout << "\r Estimating relative pose: " << chunk_id * kNumChunks << "%" @@ -30,47 +31,55 @@ void EstimateRelativePoses(ViewGraph& view_graph, const int64_t end = std::min((chunk_id + 1) * interval, num_image_pairs); -#pragma omp parallel for schedule(dynamic) private( \ - points2D_1, points2D_2, inliers) for (int64_t pair_idx = start; pair_idx < end; pair_idx++) { - ImagePair& image_pair = view_graph.image_pairs[valid_pair_ids[pair_idx]]; - const Image& image1 = images[image_pair.image_id1]; - const Image& image2 = images[image_pair.image_id2]; - const Eigen::MatrixXi& matches = image_pair.matches; + thread_pool.AddTask([&, pair_idx]() { + // Define as thread-local to reuse memory allocation in different tasks. + thread_local std::vector points2D_1; + thread_local std::vector points2D_2; + thread_local std::vector inliers; + + ImagePair& image_pair = + view_graph.image_pairs[valid_pair_ids[pair_idx]]; + const Image& image1 = images[image_pair.image_id1]; + const Image& image2 = images[image_pair.image_id2]; + const Eigen::MatrixXi& matches = image_pair.matches; - // Collect the original 2D points - points2D_1.clear(); - points2D_2.clear(); - for (size_t idx = 0; idx < matches.rows(); idx++) { - points2D_1.push_back(image1.features[matches(idx, 0)]); - points2D_2.push_back(image2.features[matches(idx, 1)]); - } + // Collect the original 2D points + points2D_1.clear(); + points2D_2.clear(); + for (size_t idx = 0; idx < matches.rows(); idx++) { + points2D_1.push_back(image1.features[matches(idx, 0)]); + points2D_2.push_back(image2.features[matches(idx, 1)]); + } - inliers.clear(); - poselib::CameraPose pose_rel_calc; - try { - poselib::estimate_relative_pose( - points2D_1, - points2D_2, - ColmapCameraToPoseLibCamera(cameras[image1.camera_id]), - ColmapCameraToPoseLibCamera(cameras[image2.camera_id]), - options.ransac_options, - options.bundle_options, - &pose_rel_calc, - &inliers); - } catch (const std::exception& e) { - LOG(ERROR) << "Error in relative pose estimation: " << e.what(); - image_pair.is_valid = false; - continue; - } + inliers.clear(); + poselib::CameraPose pose_rel_calc; + try { + poselib::estimate_relative_pose( + points2D_1, + points2D_2, + ColmapCameraToPoseLibCamera(cameras[image1.camera_id]), + ColmapCameraToPoseLibCamera(cameras[image2.camera_id]), + options.ransac_options, + options.bundle_options, + &pose_rel_calc, + &inliers); + } catch (const std::exception& e) { + LOG(ERROR) << "Error in relative pose estimation: " << e.what(); + image_pair.is_valid = false; + return; + } - // Convert the relative pose to the glomap format - for (int i = 0; i < 4; i++) { - image_pair.cam2_from_cam1.rotation.coeffs()[i] = - pose_rel_calc.q[(i + 1) % 4]; - } - image_pair.cam2_from_cam1.translation = pose_rel_calc.t; + // Convert the relative pose to the glomap format + for (int i = 0; i < 4; i++) { + image_pair.cam2_from_cam1.rotation.coeffs()[i] = + pose_rel_calc.q[(i + 1) % 4]; + } + image_pair.cam2_from_cam1.translation = pose_rel_calc.t; + }); } + + thread_pool.Wait(); } std::cout << "\r Estimating relative pose: 100%" << std::endl; diff --git a/glomap/processors/image_undistorter.cc b/glomap/processors/image_undistorter.cc index c83402a..012465d 100644 --- a/glomap/processors/image_undistorter.cc +++ b/glomap/processors/image_undistorter.cc @@ -1,5 +1,7 @@ #include "glomap/processors/image_undistorter.h" +#include + namespace glomap { void UndistortImages(std::unordered_map& cameras, @@ -7,33 +9,35 @@ void UndistortImages(std::unordered_map& cameras, bool clean_points) { std::vector image_ids; for (auto& [image_id, image] : images) { - int num_points = image.features.size(); - + const int num_points = image.features.size(); if (image.features_undist.size() == num_points && !clean_points) continue; // already undistorted image_ids.push_back(image_id); } + colmap::ThreadPool thread_pool(colmap::ThreadPool::kMaxNumThreads); + LOG(INFO) << "Undistorting images.."; const int num_images = image_ids.size(); -#pragma omp parallel for for (int image_idx = 0; image_idx < num_images; image_idx++) { Image& image = images[image_ids[image_idx]]; - - int camera_id = image.camera_id; - Camera& camera = cameras[camera_id]; - int num_points = image.features.size(); - + const int num_points = image.features.size(); if (image.features_undist.size() == num_points && !clean_points) continue; // already undistorted - image.features_undist.clear(); - image.features_undist.reserve(num_points); - for (int i = 0; i < num_points; i++) { - image.features_undist.emplace_back( - camera.CamFromImg(image.features[i]).homogeneous().normalized()); - } + const Camera& camera = cameras[image.camera_id]; + + thread_pool.AddTask([&image, &camera, num_points]() { + image.features_undist.clear(); + image.features_undist.reserve(num_points); + for (int i = 0; i < num_points; i++) { + image.features_undist.emplace_back( + camera.CamFromImg(image.features[i]).homogeneous().normalized()); + } + }); } + + thread_pool.Wait(); LOG(INFO) << "Image undistortion done"; } diff --git a/glomap/processors/view_graph_manipulation.cc b/glomap/processors/view_graph_manipulation.cc index 9373fbb..38ec4dc 100644 --- a/glomap/processors/view_graph_manipulation.cc +++ b/glomap/processors/view_graph_manipulation.cc @@ -3,7 +3,10 @@ #include "glomap/math/two_view_geometry.h" #include "glomap/math/union_find.h" +#include + namespace glomap { + image_pair_t ViewGraphManipulater::SparsifyGraph( ViewGraph& view_graph, std::unordered_map& images, @@ -245,46 +248,51 @@ void ViewGraphManipulater::DecomposeRelPose( const int64_t num_image_pairs = image_pair_ids.size(); LOG(INFO) << "Decompose relative pose for " << num_image_pairs << " pairs"; -#pragma omp parallel for + colmap::ThreadPool thread_pool(colmap::ThreadPool::kMaxNumThreads); for (int64_t idx = 0; idx < num_image_pairs; idx++) { - ImagePair& image_pair = view_graph.image_pairs.at(image_pair_ids[idx]); - image_t image_id1 = image_pair.image_id1; - image_t image_id2 = image_pair.image_id2; + thread_pool.AddTask([&, idx]() { + ImagePair& image_pair = view_graph.image_pairs.at(image_pair_ids[idx]); + image_t image_id1 = image_pair.image_id1; + image_t image_id2 = image_pair.image_id2; - camera_t camera_id1 = images.at(image_id1).camera_id; - camera_t camera_id2 = images.at(image_id2).camera_id; - - // Use the two-view geometry to re-estimate the relative pose - colmap::TwoViewGeometry two_view_geometry; - two_view_geometry.E = image_pair.E; - two_view_geometry.F = image_pair.F; - two_view_geometry.H = image_pair.H; - two_view_geometry.config = image_pair.config; - - colmap::EstimateTwoViewGeometryPose(cameras[camera_id1], - images[image_id1].features, - cameras[camera_id2], - images[image_id2].features, - &two_view_geometry); - - // if it planar, then use the estimated relative pose - if (image_pair.config == colmap::TwoViewGeometry::PLANAR && - cameras[camera_id1].has_prior_focal_length && - cameras[camera_id2].has_prior_focal_length) { - image_pair.config = colmap::TwoViewGeometry::CALIBRATED; - continue; - } else if (!(cameras[camera_id1].has_prior_focal_length && - cameras[camera_id2].has_prior_focal_length)) - continue; + camera_t camera_id1 = images.at(image_id1).camera_id; + camera_t camera_id2 = images.at(image_id2).camera_id; + + // Use the two-view geometry to re-estimate the relative pose + colmap::TwoViewGeometry two_view_geometry; + two_view_geometry.E = image_pair.E; + two_view_geometry.F = image_pair.F; + two_view_geometry.H = image_pair.H; + two_view_geometry.config = image_pair.config; + + colmap::EstimateTwoViewGeometryPose(cameras[camera_id1], + images[image_id1].features, + cameras[camera_id2], + images[image_id2].features, + &two_view_geometry); + + // if it planar, then use the estimated relative pose + if (image_pair.config == colmap::TwoViewGeometry::PLANAR && + cameras[camera_id1].has_prior_focal_length && + cameras[camera_id2].has_prior_focal_length) { + image_pair.config = colmap::TwoViewGeometry::CALIBRATED; + return; + } else if (!(cameras[camera_id1].has_prior_focal_length && + cameras[camera_id2].has_prior_focal_length)) + return; + + image_pair.config = two_view_geometry.config; + image_pair.cam2_from_cam1 = two_view_geometry.cam2_from_cam1; + + if (image_pair.cam2_from_cam1.translation.norm() > EPS) { + image_pair.cam2_from_cam1.translation = + image_pair.cam2_from_cam1.translation.normalized(); + } + }); + } - image_pair.config = two_view_geometry.config; - image_pair.cam2_from_cam1 = two_view_geometry.cam2_from_cam1; + thread_pool.Wait(); - if (image_pair.cam2_from_cam1.translation.norm() > EPS) { - image_pair.cam2_from_cam1.translation = - image_pair.cam2_from_cam1.translation.normalized(); - } - } size_t counter = 0; for (size_t idx = 0; idx < image_pair_ids.size(); idx++) { ImagePair& image_pair = view_graph.image_pairs.at(image_pair_ids[idx]); From 98fa29a3bbaec5f0db40152321e4bf3a5cfc2166 Mon Sep 17 00:00:00 2001 From: Zhihao Zhan <92145772+zhan994@users.noreply.github.com> Date: Fri, 18 Oct 2024 19:21:01 +0800 Subject: [PATCH 12/29] fix calculation for fundmental_matrix (#125) --- glomap/math/two_view_geometry.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/glomap/math/two_view_geometry.cc b/glomap/math/two_view_geometry.cc index 3d9a5d1..b7d5f5d 100644 --- a/glomap/math/two_view_geometry.cc +++ b/glomap/math/two_view_geometry.cc @@ -51,7 +51,7 @@ void FundamentalFromMotionAndCameras(const Camera& camera1, Eigen::Matrix3d* F) { Eigen::Matrix3d E; EssentialFromMotion(pose, &E); - *F = camera1.GetK().transpose().inverse() * E * camera2.GetK().inverse(); + *F = camera2.GetK().transpose().inverse() * E * camera1.GetK().inverse(); } double SampsonError(const Eigen::Matrix3d& E, From 664aa7c8fb1d7074919df4213d26cf37be01dd30 Mon Sep 17 00:00:00 2001 From: Paul-Edouard Sarlin <15985472+sarlinpe@users.noreply.github.com> Date: Fri, 15 Nov 2024 17:41:35 +0100 Subject: [PATCH 13/29] Update to COLMAP HEAD (#132) * Update to COLMAP HEAD * Bump COLMAP tag * Install gmock in CI * Apply clang-tidy only to GLOMAP headers * Fix clang-tidy regexp header filter * Make PoseLib a system include --- .github/workflows/ubuntu.yml | 1 + cmake/FindDependencies.cmake | 3 ++- glomap/controllers/global_mapper.cc | 3 ++- glomap/controllers/track_retriangulation.cc | 16 +++++++++------- glomap/estimators/bundle_adjustment.cc | 2 +- glomap/exe/global_mapper.cc | 3 ++- glomap/io/colmap_converter.cc | 11 +++++++---- glomap/io/colmap_io.cc | 1 + 8 files changed, 25 insertions(+), 15 deletions(-) diff --git a/.github/workflows/ubuntu.yml b/.github/workflows/ubuntu.yml index a8246f9..5800015 100644 --- a/.github/workflows/ubuntu.yml +++ b/.github/workflows/ubuntu.yml @@ -107,6 +107,7 @@ jobs: libmetis-dev \ libgoogle-glog-dev \ libgtest-dev \ + libgmock-dev \ libsqlite3-dev \ libglew-dev \ qtbase5-dev \ diff --git a/cmake/FindDependencies.cmake b/cmake/FindDependencies.cmake index 7bcf202..5a89c7c 100644 --- a/cmake/FindDependencies.cmake +++ b/cmake/FindDependencies.cmake @@ -24,6 +24,7 @@ FetchContent_Declare(PoseLib GIT_REPOSITORY https://github.com/PoseLib/PoseLib.git GIT_TAG 0439b2d361125915b8821043fca9376e6cc575b9 EXCLUDE_FROM_ALL + SYSTEM ) message(STATUS "Configuring PoseLib...") if (FETCH_POSELIB) @@ -35,7 +36,7 @@ message(STATUS "Configuring PoseLib... done") FetchContent_Declare(COLMAP GIT_REPOSITORY https://github.com/colmap/colmap.git - GIT_TAG 66fd8e56a0d160d68af2f29e9ac6941d442d2322 + GIT_TAG 3254263266949413d7c669e44abeb4a7c2670a8b EXCLUDE_FROM_ALL ) message(STATUS "Configuring COLMAP...") diff --git a/glomap/controllers/global_mapper.cc b/glomap/controllers/global_mapper.cc index 22f14c8..c359419 100644 --- a/glomap/controllers/global_mapper.cc +++ b/glomap/controllers/global_mapper.cc @@ -7,6 +7,7 @@ #include "glomap/processors/track_filter.h" #include "glomap/processors/view_graph_manipulation.h" +#include #include namespace glomap { @@ -325,4 +326,4 @@ bool GlobalMapper::Solve(const colmap::Database& database, return true; } -} // namespace glomap \ No newline at end of file +} // namespace glomap diff --git a/glomap/controllers/track_retriangulation.cc b/glomap/controllers/track_retriangulation.cc index 18559c2..678f99a 100644 --- a/glomap/controllers/track_retriangulation.cc +++ b/glomap/controllers/track_retriangulation.cc @@ -2,10 +2,12 @@ #include "glomap/io/colmap_converter.h" -#include +#include #include #include +#include + namespace glomap { bool RetriangulateTracks(const TriangulatorOptions& options, @@ -40,7 +42,7 @@ bool RetriangulateTracks(const TriangulatorOptions& options, std::unordered_map(), *reconstruction_ptr); - colmap::IncrementalMapperOptions options_colmap; + colmap::IncrementalPipelineOptions options_colmap; options_colmap.triangulation.complete_max_reproj_error = options.tri_complete_max_reproj_error; options_colmap.triangulation.merge_max_reproj_error = @@ -57,13 +59,13 @@ bool RetriangulateTracks(const TriangulatorOptions& options, const auto tri_options = options_colmap.Triangulation(); const auto mapper_options = options_colmap.Mapper(); - const std::vector& reg_image_ids = reconstruction_ptr->RegImageIds(); + const std::set& reg_image_ids = reconstruction_ptr->RegImageIds(); - for (size_t i = 0; i < reg_image_ids.size(); ++i) { - std::cout << "\r Triangulating image " << i + 1 << " / " + size_t image_idx = 0; + for (const image_t image_id : reg_image_ids) { + std::cout << "\r Triangulating image " << image_idx++ + 1 << " / " << reg_image_ids.size() << std::flush; - const image_t image_id = reg_image_ids[i]; const auto& image = reconstruction_ptr->Image(image_id); int num_tris = mapper.TriangulateImage(tri_options, image_id); @@ -128,4 +130,4 @@ bool RetriangulateTracks(const TriangulatorOptions& options, return true; } -} // namespace glomap \ No newline at end of file +} // namespace glomap diff --git a/glomap/estimators/bundle_adjustment.cc b/glomap/estimators/bundle_adjustment.cc index 32204ba..f9aa51c 100644 --- a/glomap/estimators/bundle_adjustment.cc +++ b/glomap/estimators/bundle_adjustment.cc @@ -70,7 +70,7 @@ void BundleAdjuster::AddPointToCameraConstraints( Image& image = images[observation.first]; ceres::CostFunction* cost_function = - colmap::CameraCostFunction( + colmap::CreateCameraCostFunction( cameras[image.camera_id].model_id, image.features[observation.second]); diff --git a/glomap/exe/global_mapper.cc b/glomap/exe/global_mapper.cc index 7e4deac..fb1e512 100644 --- a/glomap/exe/global_mapper.cc +++ b/glomap/exe/global_mapper.cc @@ -4,6 +4,7 @@ #include "glomap/io/colmap_io.h" #include "glomap/types.h" +#include #include #include @@ -151,4 +152,4 @@ int RunMapperResume(int argc, char** argv) { return EXIT_SUCCESS; } -} // namespace glomap \ No newline at end of file +} // namespace glomap diff --git a/glomap/io/colmap_converter.cc b/glomap/io/colmap_converter.cc index eceaead..75ab3ba 100644 --- a/glomap/io/colmap_converter.cc +++ b/glomap/io/colmap_converter.cc @@ -9,9 +9,10 @@ void ConvertGlomapToColmapImage(const Image& image, bool keep_points) { image_colmap.SetImageId(image.image_id); image_colmap.SetCameraId(image.camera_id); - image_colmap.SetRegistered(image.is_registered); image_colmap.SetName(image.file_name); - image_colmap.CamFromWorld() = image.cam_from_world; + if (image.is_registered) { + image_colmap.SetCamFromWorld(image.cam_from_world); + } if (keep_points) { image_colmap.SetPoints2D(image.features); @@ -130,8 +131,10 @@ void ConvertColmapToGlomap(const colmap::Reconstruction& reconstruction, image_colmap.Name()))); Image& image = ite.first->second; - image.is_registered = image_colmap.IsRegistered(); - image.cam_from_world = static_cast(image_colmap.CamFromWorld()); + image.is_registered = image_colmap.HasPose(); + if (image_colmap.HasPose()) { + image.cam_from_world = static_cast(image_colmap.CamFromWorld()); + } image.features.clear(); image.features.reserve(image_colmap.NumPoints2D()); diff --git a/glomap/io/colmap_io.cc b/glomap/io/colmap_io.cc index 88dc4cc..5189e6d 100644 --- a/glomap/io/colmap_io.cc +++ b/glomap/io/colmap_io.cc @@ -1,5 +1,6 @@ #include "glomap/io/colmap_io.h" +#include #include namespace glomap { From d41b747407b6c5ce564e3936d0f2f2ea52f89dae Mon Sep 17 00:00:00 2001 From: Linfei Pan <36349740+lpanaf@users.noreply.github.com> Date: Tue, 19 Nov 2024 13:45:51 +0100 Subject: [PATCH 14/29] Add reconstruction normalization after each step (#136) * add reconstruction normalization after each step * seperate out the helper function for normalization * d * f --- glomap/CMakeLists.txt | 2 + glomap/controllers/global_mapper.cc | 12 +++ .../processors/reconstruction_normalizer.cc | 75 +++++++++++++++++++ glomap/processors/reconstruction_normalizer.h | 17 +++++ 4 files changed, 106 insertions(+) create mode 100644 glomap/processors/reconstruction_normalizer.cc create mode 100644 glomap/processors/reconstruction_normalizer.h diff --git a/glomap/CMakeLists.txt b/glomap/CMakeLists.txt index 1325715..7d60ee1 100644 --- a/glomap/CMakeLists.txt +++ b/glomap/CMakeLists.txt @@ -18,6 +18,7 @@ set(SOURCES math/two_view_geometry.cc processors/image_pair_inliers.cc processors/image_undistorter.cc + processors/reconstruction_normalizer.cc processors/reconstruction_pruning.cc processors/relpose_filter.cc processors/track_filter.cc @@ -49,6 +50,7 @@ set(HEADERS math/union_find.h processors/image_pair_inliers.h processors/image_undistorter.h + processors/reconstruction_normalizer.h processors/reconstruction_pruning.h processors/relpose_filter.h processors/track_filter.h diff --git a/glomap/controllers/global_mapper.cc b/glomap/controllers/global_mapper.cc index c359419..6de88d9 100644 --- a/glomap/controllers/global_mapper.cc +++ b/glomap/controllers/global_mapper.cc @@ -1,7 +1,9 @@ #include "global_mapper.h" +#include "glomap/io/colmap_converter.h" #include "glomap/processors/image_pair_inliers.h" #include "glomap/processors/image_undistorter.h" +#include "glomap/processors/reconstruction_normalizer.h" #include "glomap/processors/reconstruction_pruning.h" #include "glomap/processors/relpose_filter.h" #include "glomap/processors/track_filter.h" @@ -167,6 +169,10 @@ bool GlobalMapper::Solve(const colmap::Database& database, images, tracks, options_.inlier_thresholds.max_angle_error); + + // Normalize the structure + NormalizeReconstruction(cameras, images, tracks); + run_timer.PrintSeconds(); } @@ -209,6 +215,9 @@ bool GlobalMapper::Solve(const colmap::Database& database, if (ite != options_.num_iteration_bundle_adjustment - 1) run_timer.PrintSeconds(); + // Normalize the structure + NormalizeReconstruction(cameras, images, tracks); + // 6.3. Filter tracks based on the estimation // For the filtering, in each round, the criteria for outlier is // tightened. If only few tracks are changed, no need to start bundle @@ -292,6 +301,9 @@ bool GlobalMapper::Solve(const colmap::Database& database, run_timer.PrintSeconds(); } + // Normalize the structure + NormalizeReconstruction(cameras, images, tracks); + // Filter tracks based on the estimation UndistortImages(cameras, images, true); LOG(INFO) << "Filtering tracks by reprojection ..."; diff --git a/glomap/processors/reconstruction_normalizer.cc b/glomap/processors/reconstruction_normalizer.cc new file mode 100644 index 0000000..b2af04a --- /dev/null +++ b/glomap/processors/reconstruction_normalizer.cc @@ -0,0 +1,75 @@ +#include "reconstruction_normalizer.h" + +namespace glomap { + +colmap::Sim3d NormalizeReconstruction( + std::unordered_map& cameras, + std::unordered_map& images, + std::unordered_map& tracks, + bool fixed_scale, + double extent, + double p0, + double p1) { + // Coordinates of image centers or point locations. + std::vector coords_x; + std::vector coords_y; + std::vector coords_z; + + coords_x.reserve(images.size()); + coords_y.reserve(images.size()); + coords_z.reserve(images.size()); + for (const auto& [image_id, image] : images) { + if (!image.is_registered) continue; + const Eigen::Vector3d proj_center = image.Center(); + coords_x.push_back(static_cast(proj_center(0))); + coords_y.push_back(static_cast(proj_center(1))); + coords_z.push_back(static_cast(proj_center(2))); + } + + // Determine robust bounding box and mean. + std::sort(coords_x.begin(), coords_x.end()); + std::sort(coords_y.begin(), coords_y.end()); + std::sort(coords_z.begin(), coords_z.end()); + + const size_t P0 = static_cast( + (coords_x.size() > 3) ? p0 * (coords_x.size() - 1) : 0); + const size_t P1 = static_cast( + (coords_x.size() > 3) ? p1 * (coords_x.size() - 1) : coords_x.size() - 1); + + const Eigen::Vector3d bbox_min(coords_x[P0], coords_y[P0], coords_z[P0]); + const Eigen::Vector3d bbox_max(coords_x[P1], coords_y[P1], coords_z[P1]); + + Eigen::Vector3d mean_coord(0, 0, 0); + for (size_t i = P0; i <= P1; ++i) { + mean_coord(0) += coords_x[i]; + mean_coord(1) += coords_y[i]; + mean_coord(2) += coords_z[i]; + } + mean_coord /= P1 - P0 + 1; + + // Calculate scale and translation, such that + // translation is applied before scaling. + double scale = 1.; + if (!fixed_scale) { + const double old_extent = (bbox_max - bbox_min).norm(); + if (old_extent >= std::numeric_limits::epsilon()) { + scale = extent / old_extent; + } + } + colmap::Sim3d tform( + scale, Eigen::Quaterniond::Identity(), -scale * mean_coord); + + for (auto& [_, image] : images) { + if (image.is_registered) { + image.cam_from_world = TransformCameraWorld(tform, image.cam_from_world); + } + } + + for (auto& [_, track] : tracks) { + track.xyz = tform * track.xyz; + } + + return tform; +} + +} // namespace glomap \ No newline at end of file diff --git a/glomap/processors/reconstruction_normalizer.h b/glomap/processors/reconstruction_normalizer.h new file mode 100644 index 0000000..51d51fd --- /dev/null +++ b/glomap/processors/reconstruction_normalizer.h @@ -0,0 +1,17 @@ +#pragma once + +#include "glomap/scene/types_sfm.h" + +#include "colmap/geometry/pose.h" + +namespace glomap { + +colmap::Sim3d NormalizeReconstruction( + std::unordered_map& cameras, + std::unordered_map& images, + std::unordered_map& tracks, + bool fixed_scale = false, + double extent = 10., + double p0 = 0.1, + double p1 = 0.9); +} // namespace glomap From ae015fd87b9dcd5e4bc2dd44ce6d2239b828701d Mon Sep 17 00:00:00 2001 From: Jonas Konrad <79854538+JonasKonrad@users.noreply.github.com> Date: Wed, 27 Nov 2024 13:34:15 +0100 Subject: [PATCH 15/29] Moved glog version export out of MSVC scope. (#140) --- cmake/FindDependencies.cmake | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/cmake/FindDependencies.cmake b/cmake/FindDependencies.cmake index 5a89c7c..f4eba33 100644 --- a/cmake/FindDependencies.cmake +++ b/cmake/FindDependencies.cmake @@ -7,11 +7,12 @@ find_package(Boost REQUIRED) if(CMAKE_CXX_COMPILER_ID STREQUAL "MSVC") find_package(Glog REQUIRED) - if(DEFINED glog_VERSION_MAJOR) - # Older versions of glog don't export version variables. - add_definitions("-DGLOG_VERSION_MAJOR=${glog_VERSION_MAJOR}") - add_definitions("-DGLOG_VERSION_MINOR=${glog_VERSION_MINOR}") - endif() +endif() + +if(DEFINED glog_VERSION_MAJOR) + # Older versions of glog don't export version variables. + add_definitions("-DGLOG_VERSION_MAJOR=${glog_VERSION_MAJOR}") + add_definitions("-DGLOG_VERSION_MINOR=${glog_VERSION_MINOR}") endif() if(TESTS_ENABLED) From e7e8a41c16d5bb051aadf3fd39b5884977b490cb Mon Sep 17 00:00:00 2001 From: Linfei Pan <36349740+lpanaf@users.noreply.github.com> Date: Thu, 28 Nov 2024 15:00:54 +0100 Subject: [PATCH 16/29] update colmap version (#143) * update colmap version * f * d --- cmake/FindDependencies.cmake | 2 +- glomap/controllers/track_retriangulation.cc | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/cmake/FindDependencies.cmake b/cmake/FindDependencies.cmake index f4eba33..d7e5e58 100644 --- a/cmake/FindDependencies.cmake +++ b/cmake/FindDependencies.cmake @@ -37,7 +37,7 @@ message(STATUS "Configuring PoseLib... done") FetchContent_Declare(COLMAP GIT_REPOSITORY https://github.com/colmap/colmap.git - GIT_TAG 3254263266949413d7c669e44abeb4a7c2670a8b + GIT_TAG 78f1eefacae542d753c2e4f6a26771a0d976227d EXCLUDE_FROM_ALL ) message(STATUS "Configuring COLMAP...") diff --git a/glomap/controllers/track_retriangulation.cc b/glomap/controllers/track_retriangulation.cc index 678f99a..92d73e0 100644 --- a/glomap/controllers/track_retriangulation.cc +++ b/glomap/controllers/track_retriangulation.cc @@ -98,10 +98,10 @@ bool RetriangulateTracks(const TriangulatorOptions& options, const size_t num_observations = reconstruction_ptr->ComputeNumObservations(); - // PrintHeading1("Bundle adjustment"); - colmap::BundleAdjuster bundle_adjuster(ba_options, ba_config); - // THROW_CHECK(bundle_adjuster.Solve(reconstruction.get())); - if (!bundle_adjuster.Solve(reconstruction_ptr.get())) { + std::unique_ptr bundle_adjuster; + bundle_adjuster = + CreateDefaultBundleAdjuster(ba_options, ba_config, *reconstruction_ptr); + if (bundle_adjuster->Solve().termination_type == ceres::FAILURE) { return false; } From ebd50af044e7eb335e2598fff87ee506b0ff028e Mon Sep 17 00:00:00 2001 From: Linfei Pan <36349740+lpanaf@users.noreply.github.com> Date: Fri, 13 Dec 2024 15:10:26 +0100 Subject: [PATCH 17/29] fix the bug for threshold update (#148) * fix the bug that the update of threshold from command line is not reflected in the structure * f * update * refactor the loss_function * d --- glomap/estimators/bundle_adjustment.cc | 3 ++- glomap/estimators/bundle_adjustment.h | 6 +++++- glomap/estimators/global_positioning.cc | 14 +++++++------- glomap/estimators/global_positioning.h | 6 +++++- glomap/estimators/gravity_refinement.cc | 5 +++-- glomap/estimators/gravity_refinement.h | 7 +++++-- glomap/estimators/optimization_base.h | 3 --- glomap/estimators/view_graph_calibration.cc | 7 +++---- glomap/estimators/view_graph_calibration.h | 5 +++++ pybind11 | 1 + pyglomap/pybind11 | 1 + 11 files changed, 37 insertions(+), 21 deletions(-) create mode 160000 pybind11 create mode 160000 pyglomap/pybind11 diff --git a/glomap/estimators/bundle_adjustment.cc b/glomap/estimators/bundle_adjustment.cc index f9aa51c..ee2be7a 100644 --- a/glomap/estimators/bundle_adjustment.cc +++ b/glomap/estimators/bundle_adjustment.cc @@ -54,6 +54,7 @@ void BundleAdjuster::Reset() { ceres::Problem::Options problem_options; problem_options.loss_function_ownership = ceres::DO_NOT_TAKE_OWNERSHIP; problem_ = std::make_unique(problem_options); + loss_function_ = options_.CreateLossFunction(); } void BundleAdjuster::AddPointToCameraConstraints( @@ -77,7 +78,7 @@ void BundleAdjuster::AddPointToCameraConstraints( if (cost_function != nullptr) { problem_->AddResidualBlock( cost_function, - options_.loss_function.get(), + loss_function_.get(), image.cam_from_world.rotation.coeffs().data(), image.cam_from_world.translation.data(), tracks[track_id].xyz.data(), diff --git a/glomap/estimators/bundle_adjustment.h b/glomap/estimators/bundle_adjustment.h index 97419a5..76a81b9 100644 --- a/glomap/estimators/bundle_adjustment.h +++ b/glomap/estimators/bundle_adjustment.h @@ -21,9 +21,12 @@ struct BundleAdjusterOptions : public OptimizationBaseOptions { BundleAdjusterOptions() : OptimizationBaseOptions() { thres_loss_function = 1.; - loss_function = std::make_shared(thres_loss_function); solver_options.max_num_iterations = 200; } + + std::shared_ptr CreateLossFunction() { + return std::make_shared(thres_loss_function); + } }; class BundleAdjuster { @@ -65,6 +68,7 @@ class BundleAdjuster { BundleAdjusterOptions options_; std::unique_ptr problem_; + std::shared_ptr loss_function_; }; } // namespace glomap diff --git a/glomap/estimators/global_positioning.cc b/glomap/estimators/global_positioning.cc index b18fa80..8a9b064 100644 --- a/glomap/estimators/global_positioning.cc +++ b/glomap/estimators/global_positioning.cc @@ -87,6 +87,8 @@ void GlobalPositioner::SetupProblem( ceres::Problem::Options problem_options; problem_options.loss_function_ownership = ceres::DO_NOT_TAKE_OWNERSHIP; problem_ = std::make_unique(problem_options); + loss_function_ = options_.CreateLossFunction(); + // Allocate enough memory for the scales. One for each residual. // Due to possibly invalid image pairs or tracks, the actual number of // residuals may be smaller. @@ -169,7 +171,7 @@ void GlobalPositioner::AddCameraToCameraConstraints( BATAPairwiseDirectionError::Create(translation); problem_->AddResidualBlock( cost_function, - options_.loss_function.get(), + loss_function_.get(), images[image_id1].cam_from_world.translation.data(), images[image_id2].cam_from_world.translation.data(), &scale); @@ -212,19 +214,17 @@ void GlobalPositioner::AddPointToCameraConstraints( if (loss_function_ptcam_uncalibrated_ == nullptr) { loss_function_ptcam_uncalibrated_ = - std::make_shared(options_.loss_function.get(), + std::make_shared(loss_function_.get(), 0.5 * weight_scale_pt, ceres::DO_NOT_TAKE_OWNERSHIP); } if (options_.constraint_type == GlobalPositionerOptions::POINTS_AND_CAMERAS_BALANCED) { - loss_function_ptcam_calibrated_ = - std::make_shared(options_.loss_function.get(), - weight_scale_pt, - ceres::DO_NOT_TAKE_OWNERSHIP); + loss_function_ptcam_calibrated_ = std::make_shared( + loss_function_.get(), weight_scale_pt, ceres::DO_NOT_TAKE_OWNERSHIP); } else { - loss_function_ptcam_calibrated_ = options_.loss_function; + loss_function_ptcam_calibrated_ = loss_function_; } for (auto& [track_id, track] : tracks) { diff --git a/glomap/estimators/global_positioning.h b/glomap/estimators/global_positioning.h index 0eb1185..3926ff7 100644 --- a/glomap/estimators/global_positioning.h +++ b/glomap/estimators/global_positioning.h @@ -42,7 +42,10 @@ struct GlobalPositionerOptions : public OptimizationBaseOptions { GlobalPositionerOptions() : OptimizationBaseOptions() { thres_loss_function = 1e-1; - loss_function = std::make_shared(thres_loss_function); + } + + std::shared_ptr CreateLossFunction() { + return std::make_shared(thres_loss_function); } }; @@ -104,6 +107,7 @@ class GlobalPositioner { std::unique_ptr problem_; // Loss functions for reweighted terms. + std::shared_ptr loss_function_; std::shared_ptr loss_function_ptcam_uncalibrated_; std::shared_ptr loss_function_ptcam_calibrated_; diff --git a/glomap/estimators/gravity_refinement.cc b/glomap/estimators/gravity_refinement.cc index 6015d66..d1f96fa 100644 --- a/glomap/estimators/gravity_refinement.cc +++ b/glomap/estimators/gravity_refinement.cc @@ -23,6 +23,8 @@ void GravityRefiner::RefineGravity(const ViewGraph& view_graph, return; } + loss_function_ = options_.CreateLossFunction(); + int counter_progress = 0; // Iterate through the error prone images for (auto image_id : error_prone_images) { @@ -66,8 +68,7 @@ void GravityRefiner::RefineGravity(const ViewGraph& view_graph, ceres::CostFunction* coor_cost = GravError::CreateCost(gravities[counter]); - problem.AddResidualBlock( - coor_cost, options_.loss_function.get(), gravity.data()); + problem.AddResidualBlock(coor_cost, loss_function_.get(), gravity.data()); counter++; } diff --git a/glomap/estimators/gravity_refinement.h b/glomap/estimators/gravity_refinement.h index 581b434..b966715 100644 --- a/glomap/estimators/gravity_refinement.h +++ b/glomap/estimators/gravity_refinement.h @@ -17,8 +17,10 @@ struct GravityRefinerOptions : public OptimizationBaseOptions { // Only refine the gravity of the images with more than min_neighbors int min_num_neighbors = 7; - GravityRefinerOptions() : OptimizationBaseOptions() { - loss_function = std::make_shared( + GravityRefinerOptions() : OptimizationBaseOptions() {} + + std::shared_ptr CreateLossFunction() { + return std::make_shared( 1 - std::cos(DegToRad(max_gravity_error))); } }; @@ -35,6 +37,7 @@ class GravityRefiner { const std::unordered_map& images, std::unordered_set& error_prone_images); GravityRefinerOptions options_; + std::shared_ptr loss_function_; }; } // namespace glomap diff --git a/glomap/estimators/optimization_base.h b/glomap/estimators/optimization_base.h index 2b43e06..c6a67ab 100644 --- a/glomap/estimators/optimization_base.h +++ b/glomap/estimators/optimization_base.h @@ -12,9 +12,6 @@ struct OptimizationBaseOptions { // The threshold for the loss function double thres_loss_function = 1e-1; - // The loss function for the calibration - std::shared_ptr loss_function; - // The options for the solver ceres::Solver::Options solver_options; diff --git a/glomap/estimators/view_graph_calibration.cc b/glomap/estimators/view_graph_calibration.cc index 7d5b457..30c7d34 100644 --- a/glomap/estimators/view_graph_calibration.cc +++ b/glomap/estimators/view_graph_calibration.cc @@ -61,8 +61,7 @@ void ViewGraphCalibrator::Reset( ceres::Problem::Options problem_options; problem_options.loss_function_ownership = ceres::DO_NOT_TAKE_OWNERSHIP; problem_ = std::make_unique(problem_options); - options_.loss_function = - std::make_shared(options_.thres_loss_function); + loss_function_ = options_.CreateLossFunction(); } void ViewGraphCalibrator::AddImagePairsToProblem( @@ -90,14 +89,14 @@ void ViewGraphCalibrator::AddImagePair( problem_->AddResidualBlock( FetzerFocalLengthSameCameraCost::Create( image_pair.F, cameras.at(camera_id1).PrincipalPoint()), - options_.loss_function.get(), + loss_function_.get(), &(focals_[camera_id1])); } else { problem_->AddResidualBlock( FetzerFocalLengthCost::Create(image_pair.F, cameras.at(camera_id1).PrincipalPoint(), cameras.at(camera_id2).PrincipalPoint()), - options_.loss_function.get(), + loss_function_.get(), &(focals_[camera_id1]), &(focals_[camera_id2])); } diff --git a/glomap/estimators/view_graph_calibration.h b/glomap/estimators/view_graph_calibration.h index f7f5104..c1cebd6 100644 --- a/glomap/estimators/view_graph_calibration.h +++ b/glomap/estimators/view_graph_calibration.h @@ -21,6 +21,10 @@ struct ViewGraphCalibratorOptions : public OptimizationBaseOptions { ViewGraphCalibratorOptions() : OptimizationBaseOptions() { thres_loss_function = 1e-2; } + + std::shared_ptr CreateLossFunction() { + return std::make_shared(thres_loss_function); + } }; class ViewGraphCalibrator { @@ -61,6 +65,7 @@ class ViewGraphCalibrator { ViewGraphCalibratorOptions options_; std::unique_ptr problem_; std::unordered_map focals_; + std::shared_ptr loss_function_; }; } // namespace glomap diff --git a/pybind11 b/pybind11 new file mode 160000 index 0000000..3ebdc50 --- /dev/null +++ b/pybind11 @@ -0,0 +1 @@ +Subproject commit 3ebdc503d29c7f089b9a0bc1823add0dda76f40d diff --git a/pyglomap/pybind11 b/pyglomap/pybind11 new file mode 160000 index 0000000..3ebdc50 --- /dev/null +++ b/pyglomap/pybind11 @@ -0,0 +1 @@ +Subproject commit 3ebdc503d29c7f089b9a0bc1823add0dda76f40d From 2a3db5009ce95b1e5553a70a845a44fb24e05eff Mon Sep 17 00:00:00 2001 From: duan-she-li <1063135843@qq.com> Date: Fri, 13 Dec 2024 22:23:23 +0800 Subject: [PATCH 18/29] fix (#149) --- glomap/io/colmap_converter.cc | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/glomap/io/colmap_converter.cc b/glomap/io/colmap_converter.cc index 75ab3ba..eb4bf51 100644 --- a/glomap/io/colmap_converter.cc +++ b/glomap/io/colmap_converter.cc @@ -34,6 +34,7 @@ void ConvertGlomapToColmap(const std::unordered_map& cameras, } // Prepare the 2d-3d correspondences + size_t min_supports = 2; std::unordered_map> image_to_point3D; if (tracks.size() > 0 || include_image_points) { // Initialize every point to corresponds to invalid point @@ -47,7 +48,7 @@ void ConvertGlomapToColmap(const std::unordered_map& cameras, if (tracks.size() > 0) { for (auto& [track_id, track] : tracks) { - if (track.observations.size() < 3) { + if (track.observations.size() < min_supports) { continue; } for (auto& observation : track.observations) { @@ -80,7 +81,7 @@ void ConvertGlomapToColmap(const std::unordered_map& cameras, colmap_point.track.AddElement(colmap_track_el); } - if (colmap_point.track.Length() < 2) continue; + if (colmap_point.track.Length() < min_supports) continue; colmap_point.track.Compress(); reconstruction.AddPoint3D(track_id, std::move(colmap_point)); From db294c141e7c21fb781192b69f2c8ccec246f21d Mon Sep 17 00:00:00 2001 From: Linfei Pan <36349740+lpanaf@users.noreply.github.com> Date: Fri, 13 Dec 2024 19:19:24 +0100 Subject: [PATCH 19/29] fix colmap convention for the pair id calculation (#150) --- glomap/scene/types.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/glomap/scene/types.h b/glomap/scene/types.h index 3499a10..dc3aa03 100644 --- a/glomap/scene/types.h +++ b/glomap/scene/types.h @@ -1,6 +1,7 @@ #pragma once #include +#include #include #include @@ -33,7 +34,7 @@ typedef uint64_t track_t; using colmap::Rigid3d; -const image_t kMaxNumImages = std::numeric_limits::max(); +const image_t kMaxNumImages = colmap::Database::kMaxNumImages; const image_pair_t kInvalidImagePairId = -1; } // namespace glomap From 18a70eed0e36eae7c42e1c5ef0ae6a6f8cdc6943 Mon Sep 17 00:00:00 2001 From: Linfei Pan <36349740+lpanaf@users.noreply.github.com> Date: Tue, 17 Dec 2024 15:03:41 +0100 Subject: [PATCH 20/29] avoid the inconsistency between the exported 3d point and 3d point in images (#151) --- glomap/io/colmap_converter.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/glomap/io/colmap_converter.cc b/glomap/io/colmap_converter.cc index eb4bf51..a52455c 100644 --- a/glomap/io/colmap_converter.cc +++ b/glomap/io/colmap_converter.cc @@ -81,7 +81,7 @@ void ConvertGlomapToColmap(const std::unordered_map& cameras, colmap_point.track.AddElement(colmap_track_el); } - if (colmap_point.track.Length() < min_supports) continue; + if (track.observations.size() < min_supports) continue; colmap_point.track.Compress(); reconstruction.AddPoint3D(track_id, std::move(colmap_point)); From cbb0b5ece38b1c3639d33dcc6b9374862ab5a21e Mon Sep 17 00:00:00 2001 From: Linfei Pan <36349740+lpanaf@users.noreply.github.com> Date: Thu, 13 Mar 2025 13:56:12 +0100 Subject: [PATCH 21/29] add more options in CLI and add support to optimize principal point (#170) * add more options in CLI and add support to optimize principal point * d * by default, do not optimize pp --- glomap/controllers/option_manager.cc | 10 ++++++++++ glomap/estimators/bundle_adjustment.cc | 6 +++--- glomap/estimators/bundle_adjustment.h | 1 + 3 files changed, 14 insertions(+), 3 deletions(-) diff --git a/glomap/controllers/option_manager.cc b/glomap/controllers/option_manager.cc index d1e0304..9b8fb6a 100644 --- a/glomap/controllers/option_manager.cc +++ b/glomap/controllers/option_manager.cc @@ -205,6 +205,8 @@ void OptionManager::AddBundleAdjusterOptions() { &mapper->opt_ba.optimize_translation); AddAndRegisterDefaultOption("BundleAdjustment.optimize_intrinsics", &mapper->opt_ba.optimize_intrinsics); + AddAndRegisterDefaultOption("BundleAdjustment.optimize_principal_point", + &mapper->opt_ba.optimize_principal_point); AddAndRegisterDefaultOption("BundleAdjustment.optimize_points", &mapper->opt_ba.optimize_points); AddAndRegisterDefaultOption("BundleAdjustment.thres_loss_function", @@ -234,6 +236,14 @@ void OptionManager::AddInlierThresholdOptions() { return; } added_inliers_options_ = true; + AddAndRegisterDefaultOption("Thresholds.max_angle_error", + &mapper->inlier_thresholds.max_angle_error); + AddAndRegisterDefaultOption( + "Thresholds.max_reprojection_error", + &mapper->inlier_thresholds.max_reprojection_error); + AddAndRegisterDefaultOption( + "Thresholds.min_triangulation_angle", + &mapper->inlier_thresholds.min_triangulation_angle); AddAndRegisterDefaultOption("Thresholds.max_epipolar_error_E", &mapper->inlier_thresholds.max_epipolar_error_E); AddAndRegisterDefaultOption("Thresholds.max_epipolar_error_F", diff --git a/glomap/estimators/bundle_adjustment.cc b/glomap/estimators/bundle_adjustment.cc index ee2be7a..3bc83d0 100644 --- a/glomap/estimators/bundle_adjustment.cc +++ b/glomap/estimators/bundle_adjustment.cc @@ -161,7 +161,7 @@ void BundleAdjuster::ParameterizeVariables( images[center].cam_from_world.translation.data()); // Parameterize the camera parameters, or set them to be constant if desired - if (options_.optimize_intrinsics) { + if (options_.optimize_intrinsics && !options_.optimize_principal_point) { for (auto& [camera_id, camera] : cameras) { if (problem_->HasParameterBlock(camera.params.data())) { std::vector principal_point_idxs; @@ -174,8 +174,8 @@ void BundleAdjuster::ParameterizeVariables( camera.params.data()); } } - - } else { + } else if (!options_.optimize_intrinsics && + !options_.optimize_principal_point) { for (auto& [camera_id, camera] : cameras) { if (problem_->HasParameterBlock(camera.params.data())) { problem_->SetParameterBlockConstant(camera.params.data()); diff --git a/glomap/estimators/bundle_adjustment.h b/glomap/estimators/bundle_adjustment.h index 76a81b9..36a142f 100644 --- a/glomap/estimators/bundle_adjustment.h +++ b/glomap/estimators/bundle_adjustment.h @@ -14,6 +14,7 @@ struct BundleAdjusterOptions : public OptimizationBaseOptions { bool optimize_rotations = true; bool optimize_translation = true; bool optimize_intrinsics = true; + bool optimize_principal_point = false; bool optimize_points = true; // Constrain the minimum number of views per track From 07596cdfe12a6a5719322358e374548bb0bcf4e5 Mon Sep 17 00:00:00 2001 From: Linfei Pan <36349740+lpanaf@users.noreply.github.com> Date: Thu, 20 Mar 2025 16:01:22 +0100 Subject: [PATCH 22/29] Relpose estimation bug fix (#171) * fix the bug that relative pose estimation fails when camera model is not supported by poselib * allow the focal length to be different for unrecognized camera type --- glomap/estimators/relpose_estimation.cc | 52 ++++++++++++++++++++----- 1 file changed, 43 insertions(+), 9 deletions(-) diff --git a/glomap/estimators/relpose_estimation.cc b/glomap/estimators/relpose_estimation.cc index 4676eb2..8cd3b38 100644 --- a/glomap/estimators/relpose_estimation.cc +++ b/glomap/estimators/relpose_estimation.cc @@ -44,6 +44,13 @@ void EstimateRelativePoses(ViewGraph& view_graph, const Image& image2 = images[image_pair.image_id2]; const Eigen::MatrixXi& matches = image_pair.matches; + const Camera& camera1 = cameras[image1.camera_id]; + const Camera& camera2 = cameras[image2.camera_id]; + poselib::Camera camera_poselib1 = ColmapCameraToPoseLibCamera(camera1); + poselib::Camera camera_poselib2 = ColmapCameraToPoseLibCamera(camera2); + bool valid_camera_model = + (camera_poselib1.model_id >= 0 && camera_poselib2.model_id >= 0); + // Collect the original 2D points points2D_1.clear(); points2D_2.clear(); @@ -51,19 +58,46 @@ void EstimateRelativePoses(ViewGraph& view_graph, points2D_1.push_back(image1.features[matches(idx, 0)]); points2D_2.push_back(image2.features[matches(idx, 1)]); } + // If the camera model is not supported by poselib + if (!valid_camera_model) { + // Undistort points + // Note that here, we still rescale by the focal length (to avoid + // change the RANSAC threshold) + Eigen::Matrix2d K1_new = Eigen::Matrix2d::Zero(); + Eigen::Matrix2d K2_new = Eigen::Matrix2d::Zero(); + K1_new(0, 0) = camera1.FocalLengthX(); + K1_new(1, 1) = camera1.FocalLengthY(); + K2_new(0, 0) = camera2.FocalLengthX(); + K2_new(1, 1) = camera2.FocalLengthY(); + for (size_t idx = 0; idx < matches.rows(); idx++) { + points2D_1[idx] = K1_new * camera1.CamFromImg(points2D_1[idx]); + points2D_2[idx] = K2_new * camera2.CamFromImg(points2D_2[idx]); + } + // Reset the camera to be the pinhole camera with original focal + // length and zero principal point + camera_poselib1 = poselib::Camera( + "PINHOLE", + {camera1.FocalLengthX(), camera1.FocalLengthY(), 0., 0.}, + camera1.width, + camera1.height); + camera_poselib2 = poselib::Camera( + "PINHOLE", + {camera2.FocalLengthX(), camera2.FocalLengthY(), 0., 0.}, + camera2.width, + camera2.height); + } inliers.clear(); poselib::CameraPose pose_rel_calc; try { - poselib::estimate_relative_pose( - points2D_1, - points2D_2, - ColmapCameraToPoseLibCamera(cameras[image1.camera_id]), - ColmapCameraToPoseLibCamera(cameras[image2.camera_id]), - options.ransac_options, - options.bundle_options, - &pose_rel_calc, - &inliers); + poselib::estimate_relative_pose(points2D_1, + points2D_2, + camera_poselib1, + camera_poselib2, + options.ransac_options, + options.bundle_options, + &pose_rel_calc, + &inliers); } catch (const std::exception& e) { LOG(ERROR) << "Error in relative pose estimation: " << e.what(); image_pair.is_valid = false; From 8b278581ec2662b53097c435f250ddffdd00c76a Mon Sep 17 00:00:00 2001 From: Gareth Date: Thu, 20 Mar 2025 08:01:43 -0700 Subject: [PATCH 23/29] Use CMAKE_CURRENT_SOURCE_DIR in FindDependencies.cmake (#175) --- cmake/FindDependencies.cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/FindDependencies.cmake b/cmake/FindDependencies.cmake index d7e5e58..769ee7e 100644 --- a/cmake/FindDependencies.cmake +++ b/cmake/FindDependencies.cmake @@ -1,4 +1,4 @@ -set(CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake") +set(CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake") find_package(Eigen3 3.4 REQUIRED) find_package(SuiteSparse COMPONENTS CHOLMOD REQUIRED) From ce51d8b1b2899e77a38799b31470a20711796378 Mon Sep 17 00:00:00 2001 From: Charlie Date: Fri, 21 Mar 2025 00:14:24 +0900 Subject: [PATCH 24/29] added gpu support (#174) * added gpu support * add gpu to bundle adjustment * f * add gpu to global positioning and bundle adjustment * format * change back to original glomap naming * keep consistent between gp and ba * change to the workflow to test cuda as well * add option in cmake whether to use cuda * d * d * d * d * d * change version * keep the minimum_num to be consistent with COLMAP --------- Co-authored-by: Linfei Pan --- .github/workflows/ubuntu.yml | 3 +- .github/workflows/windows.yml | 35 +++++++++++-- CMakeLists.txt | 23 ++++++--- cmake/FindDependencies.cmake | 68 ++++++++++++++++++++++++- glomap/controllers/option_manager.cc | 8 +++ glomap/estimators/bundle_adjustment.cc | 54 ++++++++++++++++++++ glomap/estimators/bundle_adjustment.h | 4 ++ glomap/estimators/global_positioning.cc | 55 ++++++++++++++++++++ glomap/estimators/global_positioning.h | 4 ++ glomap/glomap.cc | 6 +++ vcpkg.json | 8 +++ 11 files changed, 254 insertions(+), 14 deletions(-) diff --git a/.github/workflows/ubuntu.yml b/.github/workflows/ubuntu.yml index 5800015..ebd5654 100644 --- a/.github/workflows/ubuntu.yml +++ b/.github/workflows/ubuntu.yml @@ -35,7 +35,7 @@ jobs: os: ubuntu-22.04, cmakeBuildType: Release, asanEnabled: true, - cudaEnabled: false, + cudaEnabled: true, checkCodeFormat: false, }, { @@ -163,6 +163,7 @@ jobs: -DCMAKE_BUILD_TYPE=${{ matrix.config.cmakeBuildType }} \ -DCMAKE_INSTALL_PREFIX=./install \ -DCMAKE_CUDA_ARCHITECTURES=50 \ + -DCUDA_ENABLED=${{ matrix.config.cudaEnabled }} \ -DTESTS_ENABLED=ON \ -DASAN_ENABLED=${{ matrix.config.asanEnabled }} ninja -k 10000 diff --git a/.github/workflows/windows.yml b/.github/workflows/windows.yml index 74e1fe5..1adc363 100644 --- a/.github/workflows/windows.yml +++ b/.github/workflows/windows.yml @@ -23,6 +23,13 @@ jobs: testsEnabled: true, exportPackage: false, }, + { + os: windows-2022, + cmakeBuildType: Release, + cudaEnabled: true, + testsEnabled: true, + exportPackage: true, + }, { os: windows-2022, cmakeBuildType: Release, @@ -70,6 +77,15 @@ jobs: .github/workflows/install-ccache.ps1 -Destination "${{ env.COMPILER_CACHE_DIR }}/bin" + - name: Install CUDA + uses: Jimver/cuda-toolkit@v0.2.18 + if: matrix.config.cudaEnabled + id: cuda-toolkit + with: + cuda: '12.6.2' + sub-packages: '["nvcc", "nvtx", "cudart", "curand", "curand_dev", "nvrtc_dev"]' + method: 'network' + - name: Install CMake and Ninja uses: lukka/get-cmake@latest @@ -96,10 +112,11 @@ jobs: -DCMAKE_MAKE_PROGRAM=ninja ` -DCMAKE_BUILD_TYPE=Release ` -DTESTS_ENABLED=ON ` - -DCUDA_ENABLED=OFF ` + -DCUDA_ENABLED=${{ matrix.config.cudaEnabled }} ` -DGUI_ENABLED=OFF ` -DCGAL_ENABLED=OFF ` -DCMAKE_CUDA_ARCHITECTURES=all-major ` + -DCUDAToolkit_ROOT="${{ steps.cuda-toolkit.outputs.CUDA_PATH }}" ` -DCMAKE_TOOLCHAIN_FILE="${{ github.workspace }}/vcpkg/scripts/buildsystems/vcpkg.cmake" ` -DVCPKG_TARGET_TRIPLET=x64-windows-release ` -DCMAKE_INSTALL_PREFIX=install @@ -123,15 +140,27 @@ jobs: ../vcpkg/vcpkg.exe install ` --triplet=x64-windows-release + $(if ($${{ matrix.config.cudaEnabled }}) { echo "--x-feature=cuda" }) ../vcpkg/vcpkg.exe export --raw --output-dir vcpkg_export --output glomap cp vcpkg_export/glomap/installed/x64-windows/bin/*.dll install/bin cp vcpkg_export/glomap/installed/x64-windows-release/bin/*.dll install/bin + if ($${{ matrix.config.cudaEnabled }}) { + cp "${{ steps.cuda-toolkit.outputs.CUDA_PATH }}/bin/cudart64_*.dll" install/bin + cp "${{ steps.cuda-toolkit.outputs.CUDA_PATH }}/bin/curand64_*.dll" install/bin + } + + - name: Upload package + uses: actions/upload-artifact@v4 + if: ${{ matrix.config.exportPackage && matrix.config.cudaEnabled }} + with: + name: glomap-x64-windows-cuda + path: build/install - name: Upload package uses: actions/upload-artifact@v4 - if: ${{ matrix.config.exportPackage }} + if: ${{ matrix.config.exportPackage && !matrix.config.cudaEnabled }} with: - name: glomap-x64-windows + name: glomap-x64-windows-nocuda path: build/install - name: Cleanup compiler cache diff --git a/CMakeLists.txt b/CMakeLists.txt index 1d1c272..1c0f9ab 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,21 +1,27 @@ cmake_minimum_required(VERSION 3.28) -project(glomap VERSION 1.0.0) - -set(CMAKE_CXX_STANDARD 17) -set(CMAKE_CXX_STANDARD_REQUIRED ON) - -set_property(GLOBAL PROPERTY GLOBAL_DEPENDS_NO_CYCLES ON) - +option(CUDA_ENABLED "Whether to enable CUDA, if available" ON) option(TESTS_ENABLED "Whether to build test binaries" OFF) option(ASAN_ENABLED "Whether to enable AddressSanitizer flags" OFF) option(CCACHE_ENABLED "Whether to enable compiler caching, if available" ON) option(FETCH_COLMAP "Whether to use COLMAP with FetchContent or with self-installed software" ON) option(FETCH_POSELIB "Whether to use PoseLib with FetchContent or with self-installed software" ON) +# Propagate options to vcpkg manifest. +if(CUDA_ENABLED) + list(APPEND VCPKG_MANIFEST_FEATURES "cuda") +endif() + +# Initialize the project. +project(glomap VERSION 1.1.0) + +set(CMAKE_CXX_STANDARD 17) +set(CMAKE_CXX_STANDARD_REQUIRED ON) + +set_property(GLOBAL PROPERTY GLOBAL_DEPENDS_NO_CYCLES ON) + include(cmake/FindDependencies.cmake) -# Propagate options to vcpkg manifest. if (TESTS_ENABLED) enable_testing() endif() @@ -39,6 +45,7 @@ else() message(STATUS "Disabling ccache support") endif() + if(CMAKE_CXX_COMPILER_ID STREQUAL "MSVC") # Some fixes for the Glog library. add_definitions("-DGLOG_USE_GLOG_EXPORT") diff --git a/cmake/FindDependencies.cmake b/cmake/FindDependencies.cmake index 769ee7e..a59323a 100644 --- a/cmake/FindDependencies.cmake +++ b/cmake/FindDependencies.cmake @@ -28,7 +28,7 @@ FetchContent_Declare(PoseLib SYSTEM ) message(STATUS "Configuring PoseLib...") -if (FETCH_POSELIB) +if (FETCH_POSELIB) FetchContent_MakeAvailable(PoseLib) else() find_package(PoseLib REQUIRED) @@ -42,9 +42,73 @@ FetchContent_Declare(COLMAP ) message(STATUS "Configuring COLMAP...") set(UNINSTALL_ENABLED OFF CACHE INTERNAL "") -if (FETCH_COLMAP) +if (FETCH_COLMAP) FetchContent_MakeAvailable(COLMAP) else() find_package(COLMAP REQUIRED) endif() message(STATUS "Configuring COLMAP... done") + +set(CUDA_MIN_VERSION "7.0") +if(CUDA_ENABLED) + if(CMAKE_VERSION VERSION_LESS 3.17) + find_package(CUDA QUIET) + if(CUDA_FOUND) + message(STATUS "Found CUDA version ${CUDA_VERSION} installed in " + "${CUDA_TOOLKIT_ROOT_DIR} via legacy CMake (<3.17) module. " + "Using the legacy CMake module means that any installation of " + "COLMAP will require that the CUDA libraries are " + "available under LD_LIBRARY_PATH.") + message(STATUS "Found CUDA ") + message(STATUS " Includes : ${CUDA_INCLUDE_DIRS}") + message(STATUS " Libraries : ${CUDA_LIBRARIES}") + + enable_language(CUDA) + + macro(declare_imported_cuda_target module) + add_library(CUDA::${module} INTERFACE IMPORTED) + target_include_directories( + CUDA::${module} INTERFACE ${CUDA_INCLUDE_DIRS}) + target_link_libraries( + CUDA::${module} INTERFACE ${CUDA_${module}_LIBRARY} ${ARGN}) + endmacro() + + declare_imported_cuda_target(cudart ${CUDA_LIBRARIES}) + declare_imported_cuda_target(curand ${CUDA_LIBRARIES}) + + set(CUDAToolkit_VERSION "${CUDA_VERSION_STRING}") + set(CUDAToolkit_BIN_DIR "${CUDA_TOOLKIT_ROOT_DIR}/bin") + else() + message(STATUS "CUDA not found") + endif() + else() + find_package(CUDAToolkit QUIET) + if(CUDAToolkit_FOUND) + set(CUDA_FOUND ON) + enable_language(CUDA) + else() + message(STATUS "CUDA not found") + endif() + endif() +endif() + +if(CUDA_ENABLED AND CUDA_FOUND) + if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES) + set(CMAKE_CUDA_ARCHITECTURES "native") + endif() + + add_definitions("-DGLOMAP_CUDA_ENABLED") + + # Do not show warnings if the architectures are deprecated. + set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} -Wno-deprecated-gpu-targets") + # Explicitly set PIC flags for CUDA targets. + if(NOT IS_MSVC) + set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} --compiler-options -fPIC") + endif() + + message(STATUS "Enabling CUDA support (version: ${CUDAToolkit_VERSION}, " + "archs: ${CMAKE_CUDA_ARCHITECTURES})") +else() + set(CUDA_ENABLED OFF) + message(STATUS "Disabling CUDA support") +endif() diff --git a/glomap/controllers/option_manager.cc b/glomap/controllers/option_manager.cc index 9b8fb6a..d8041a6 100644 --- a/glomap/controllers/option_manager.cc +++ b/glomap/controllers/option_manager.cc @@ -180,6 +180,10 @@ void OptionManager::AddGlobalPositionerOptions() { return; } added_global_positioning_options_ = true; + AddAndRegisterDefaultOption("GlobalPositioning.use_gpu", + &mapper->opt_gp.use_gpu); + AddAndRegisterDefaultOption("GlobalPositioning.gpu_index", + &mapper->opt_gp.gpu_index); AddAndRegisterDefaultOption("GlobalPositioning.optimize_positions", &mapper->opt_gp.optimize_positions); AddAndRegisterDefaultOption("GlobalPositioning.optimize_points", @@ -199,6 +203,10 @@ void OptionManager::AddBundleAdjusterOptions() { return; } added_bundle_adjustment_options_ = true; + AddAndRegisterDefaultOption("BundleAdjustment.use_gpu", + &mapper->opt_ba.use_gpu); + AddAndRegisterDefaultOption("BundleAdjustment.gpu_index", + &mapper->opt_ba.gpu_index); AddAndRegisterDefaultOption("BundleAdjustment.optimize_rotations", &mapper->opt_ba.optimize_rotations); AddAndRegisterDefaultOption("BundleAdjustment.optimize_translation", diff --git a/glomap/estimators/bundle_adjustment.cc b/glomap/estimators/bundle_adjustment.cc index 3bc83d0..57f85b1 100644 --- a/glomap/estimators/bundle_adjustment.cc +++ b/glomap/estimators/bundle_adjustment.cc @@ -3,6 +3,8 @@ #include #include #include +#include +#include namespace glomap { @@ -36,6 +38,58 @@ bool BundleAdjuster::Solve(const ViewGraph& view_graph, // Set the solver options. ceres::Solver::Summary summary; + int num_images = images.size(); +#ifdef GLOMAP_CUDA_ENABLED + bool cuda_solver_enabled = false; + +#if (CERES_VERSION_MAJOR >= 3 || \ + (CERES_VERSION_MAJOR == 2 && CERES_VERSION_MINOR >= 2)) && \ + !defined(CERES_NO_CUDA) + if (options_.use_gpu && num_images >= options_.min_num_images_gpu_solver) { + cuda_solver_enabled = true; + options_.solver_options.dense_linear_algebra_library_type = ceres::CUDA; + } +#else + if (options_.use_gpu) { + LOG_FIRST_N(WARNING, 1) + << "Requested to use GPU for bundle adjustment, but Ceres was " + "compiled without CUDA support. Falling back to CPU-based dense " + "solvers."; + } +#endif + +#if (CERES_VERSION_MAJOR >= 3 || \ + (CERES_VERSION_MAJOR == 2 && CERES_VERSION_MINOR >= 3)) && \ + !defined(CERES_NO_CUDSS) + if (options_.use_gpu && num_images >= options_.min_num_images_gpu_solver) { + cuda_solver_enabled = true; + options_.solver_options.sparse_linear_algebra_library_type = + ceres::CUDA_SPARSE; + } +#else + if (options_.use_gpu) { + LOG_FIRST_N(WARNING, 1) + << "Requested to use GPU for bundle adjustment, but Ceres was " + "compiled without cuDSS support. Falling back to CPU-based sparse " + "solvers."; + } +#endif + + if (cuda_solver_enabled) { + const std::vector gpu_indices = + colmap::CSVToVector(options_.gpu_index); + THROW_CHECK_GT(gpu_indices.size(), 0); + colmap::SetBestCudaDevice(gpu_indices[0]); + } +#else + if (options_.use_gpu) { + LOG_FIRST_N(WARNING, 1) + << "Requested to use GPU for bundle adjustment, but COLMAP was " + "compiled without CUDA support. Falling back to CPU-based " + "solvers."; + } +#endif // GLOMAP_CUDA_ENABLED + // Do not use the iterative solver, as it does not seem to be helpful options_.solver_options.linear_solver_type = ceres::SPARSE_SCHUR; options_.solver_options.preconditioner_type = ceres::CLUSTER_TRIDIAGONAL; diff --git a/glomap/estimators/bundle_adjustment.h b/glomap/estimators/bundle_adjustment.h index 36a142f..b78347c 100644 --- a/glomap/estimators/bundle_adjustment.h +++ b/glomap/estimators/bundle_adjustment.h @@ -17,6 +17,10 @@ struct BundleAdjusterOptions : public OptimizationBaseOptions { bool optimize_principal_point = false; bool optimize_points = true; + bool use_gpu = true; + std::string gpu_index = "-1"; + int min_num_images_gpu_solver = 50; + // Constrain the minimum number of views per track int min_num_view_per_track = 3; diff --git a/glomap/estimators/global_positioning.cc b/glomap/estimators/global_positioning.cc index 8a9b064..ebe1b8d 100644 --- a/glomap/estimators/global_positioning.cc +++ b/glomap/estimators/global_positioning.cc @@ -2,6 +2,9 @@ #include "glomap/estimators/cost_function.h" +#include +#include + namespace glomap { namespace { @@ -361,6 +364,58 @@ void GlobalPositioner::ParameterizeVariables( } } + int num_images = images.size(); +#ifdef GLOMAP_CUDA_ENABLED + bool cuda_solver_enabled = false; + +#if (CERES_VERSION_MAJOR >= 3 || \ + (CERES_VERSION_MAJOR == 2 && CERES_VERSION_MINOR >= 2)) && \ + !defined(CERES_NO_CUDA) + if (options_.use_gpu && num_images >= options_.min_num_images_gpu_solver) { + cuda_solver_enabled = true; + options_.solver_options.dense_linear_algebra_library_type = ceres::CUDA; + } +#else + if (options_.use_gpu) { + LOG_FIRST_N(WARNING, 1) + << "Requested to use GPU for bundle adjustment, but Ceres was " + "compiled without CUDA support. Falling back to CPU-based dense " + "solvers."; + } +#endif + +#if (CERES_VERSION_MAJOR >= 3 || \ + (CERES_VERSION_MAJOR == 2 && CERES_VERSION_MINOR >= 3)) && \ + !defined(CERES_NO_CUDSS) + if (options_.use_gpu && num_images >= options_.min_num_images_gpu_solver) { + cuda_solver_enabled = true; + options_.solver_options.sparse_linear_algebra_library_type = + ceres::CUDA_SPARSE; + } +#else + if (options_.use_gpu) { + LOG_FIRST_N(WARNING, 1) + << "Requested to use GPU for bundle adjustment, but Ceres was " + "compiled without cuDSS support. Falling back to CPU-based sparse " + "solvers."; + } +#endif + + if (cuda_solver_enabled) { + const std::vector gpu_indices = + colmap::CSVToVector(options_.gpu_index); + THROW_CHECK_GT(gpu_indices.size(), 0); + colmap::SetBestCudaDevice(gpu_indices[0]); + } +#else + if (options_.use_gpu) { + LOG_FIRST_N(WARNING, 1) + << "Requested to use GPU for bundle adjustment, but COLMAP was " + "compiled without CUDA support. Falling back to CPU-based " + "solvers."; + } +#endif // GLOMAP_CUDA_ENABLED + // Set up the options for the solver // Do not use iterative solvers, for its suboptimal performance. if (tracks.size() > 0) { diff --git a/glomap/estimators/global_positioning.h b/glomap/estimators/global_positioning.h index 3926ff7..f318e8f 100644 --- a/glomap/estimators/global_positioning.h +++ b/glomap/estimators/global_positioning.h @@ -29,6 +29,10 @@ struct GlobalPositionerOptions : public OptimizationBaseOptions { bool optimize_points = true; bool optimize_scales = true; + bool use_gpu = true; + std::string gpu_index = "-1"; + int min_num_images_gpu_solver = 50; + // Constrain the minimum number of views per track int min_num_view_per_track = 3; diff --git a/glomap/glomap.cc b/glomap/glomap.cc index aa300a1..19c7725 100644 --- a/glomap/glomap.cc +++ b/glomap/glomap.cc @@ -12,6 +12,12 @@ int ShowHelp( std::cout << "GLOMAP -- Global Structure-from-Motion" << std::endl << std::endl; +#ifdef GLOMAP_CUDA_ENABLED + std::cout << "This version was compiled with CUDA!" << std::endl << std::endl; +#else + std::cout << "This version was NOT compiled CUDA!" << std::endl << std::endl; +#endif + std::cout << "Usage:" << std::endl; std::cout << " glomap mapper --database_path DATABASE --output_path MODEL" << std::endl; diff --git a/vcpkg.json b/vcpkg.json index 4ec5778..a1b5010 100644 --- a/vcpkg.json +++ b/vcpkg.json @@ -16,6 +16,7 @@ "name": "ceres", "features": [ "lapack", + "schur", "suitesparse" ] }, @@ -42,5 +43,12 @@ "suitesparse" ], "features": { + "cuda": { + "description": "Build with CUDA.", + "dependencies": [ + "glew", + "cuda" + ] + } } } From e44ce358fa6c1e1344984b8cedfe511354ed3d79 Mon Sep 17 00:00:00 2001 From: Linfei Pan <36349740+lpanaf@users.noreply.github.com> Date: Thu, 20 Mar 2025 16:23:17 +0100 Subject: [PATCH 25/29] Add the rotation averager to GLOMAP (#113) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Fix conversion of colmap pose prior * d * restore the pose prior after rotation averaging * f * f * f * gravity refinement tested * add the stratified rotation averager and CLI * f * d * d * add timer * add options for gravity refinement * merge gravity_io to pose_io * add readme for the rotation averager * f * Update glomap/math/gravity.cc Co-authored-by: Johannes Schönberger * f * Update glomap/controllers/rotation_averager.cc Co-authored-by: Johannes Schönberger * Update glomap/controllers/rotation_averager.cc Co-authored-by: Johannes Schönberger * change rotation averager from class to struct * d * d * add weighting and corresponding IO. Tested * change the writing of relative pose to be sorted * unit test for rotation averager * inject noise to avoid local minima for error-free case * f * add more points in the unit test --------- Co-authored-by: Johannes Schönberger --- README.md | 1 + docs/rotation_averager.md | 62 +++++ glomap/CMakeLists.txt | 12 +- glomap/controllers/option_manager.cc | 15 ++ glomap/controllers/option_manager.h | 4 + glomap/controllers/rotation_averager.cc | 61 +++++ glomap/controllers/rotation_averager.h | 15 ++ glomap/controllers/rotation_averager_test.cc | 236 ++++++++++++++++++ .../estimators/global_rotation_averaging.cc | 45 +++- glomap/estimators/global_rotation_averaging.h | 5 +- glomap/estimators/gravity_refinement.cc | 5 + glomap/estimators/view_graph_calibration.cc | 2 +- glomap/exe/rotation_averager.cc | 105 ++++++++ glomap/exe/rotation_averager.h | 10 + glomap/glomap.cc | 4 +- glomap/io/colmap_converter.cc | 21 +- glomap/io/gravity_io.cc | 45 ---- glomap/io/gravity_io.h | 14 -- glomap/io/pose_io.cc | 212 ++++++++++++++++ glomap/io/pose_io.h | 37 +++ glomap/math/gravity.cc | 66 +++++ glomap/math/gravity.h | 5 + glomap/scene/image.h | 3 +- glomap/scene/image_pair.h | 2 +- 24 files changed, 904 insertions(+), 83 deletions(-) create mode 100644 docs/rotation_averager.md create mode 100644 glomap/controllers/rotation_averager.cc create mode 100644 glomap/controllers/rotation_averager.h create mode 100644 glomap/controllers/rotation_averager_test.cc create mode 100644 glomap/exe/rotation_averager.cc create mode 100644 glomap/exe/rotation_averager.h delete mode 100644 glomap/io/gravity_io.cc delete mode 100644 glomap/io/gravity_io.h create mode 100644 glomap/io/pose_io.cc create mode 100644 glomap/io/pose_io.h diff --git a/README.md b/README.md index e5f4455..5c388af 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,7 @@ If you use this project for your research, please cite year={2024}, } ``` +To use the seperate rotation averaging module, refer to [this README](docs/rotation_averager.md). ## Getting Started diff --git a/docs/rotation_averager.md b/docs/rotation_averager.md new file mode 100644 index 0000000..5c4e994 --- /dev/null +++ b/docs/rotation_averager.md @@ -0,0 +1,62 @@ +# Gravity-aligned Rotation Averaging with Circular Regression + +[Project page](https://lpanaf.github.io/eccv24_ra1dof/) | [Paper](https://www.ecva.net/papers/eccv_2024/papers_ECCV/papers/05651.pdf) | [Supp.](https://lpanaf.github.io/assets/pdf/eccv24_ra1dof_sm.pdf) +--- + +## About + +This project aims at solving the rotation averaging problem with gravity prior. +To achieve this, circular regression is leveraged. + +If you use this project for your research, please cite +``` +@inproceedings{pan2024ra1dof, + author={Pan, Linfei and Pollefeys, Marc and Barath, Daniel}, + title={{Gravity-aligned Rotation Averaging with Circular Regression}}, + booktitle={European Conference on Computer Vision (ECCV)}, + year={2024}, +} +``` + +## Getting Started +Install GLOMAP as instrcucted in [README](../README.md). +Then, call the rotation averager (3 degree-of-freedom) via +``` +glomap rotation_averager --relpose_path RELPOSE_PATH --output_path OUTPUT_PATH +``` + +If gravity directions are available, call the rotation averager (1 degree-of-freedom) via +``` +glomap rotation_averager \ + --relpose_path RELPOSE_PATH \ + --output_path OUTPUT_PATH \ + --gravity_path GRAVTIY PATH +``` +It is recommended to set `--use_stratified=1` if only a subset of images have gravity direction. +If gravity measurements are subject to i.i.d. noise, they can be refined by setting `--refine_gravity=1`. + + +## File Formats +### Relative Pose +The relative pose file is expected to be of the following format +``` +IMAGE_NAME_1 IMAGE_NAME_2 QW QX QY QZ TX TY TZ +``` +Only images contained in at least one relative pose will be included in the following procedure. +The relative pose should be 2R1 x1 + 2t1 = x2. + +### Gravity Direction +The gravity direction file is expected to be of the following format +``` +IMAGE_NAME GX GY GZ +``` +The gravity direction $g$ should $[0, 1, 0]$ if the image is parallel to the ground plane, and the estimated rotation would have the property that $R_i \cdot [0, 1, 0]^\top = g$. +If is acceptable if only a subset of all images have gravity direciton. +If the specified image name does not match any known image name from relative pose, it is ignored. + +### Output +The estimated global rotation will be in the following format +``` +IMAGE_NAME QW QX QY QZ +``` +Any images that are not within the largest connected component of the view-graph formed by the relative pose will be ignored. diff --git a/glomap/CMakeLists.txt b/glomap/CMakeLists.txt index 7d60ee1..e191049 100644 --- a/glomap/CMakeLists.txt +++ b/glomap/CMakeLists.txt @@ -1,6 +1,7 @@ set(SOURCES controllers/global_mapper.cc controllers/option_manager.cc + controllers/rotation_averager.cc controllers/track_establishment.cc controllers/track_retriangulation.cc estimators/bundle_adjustment.cc @@ -11,7 +12,7 @@ set(SOURCES estimators/view_graph_calibration.cc io/colmap_converter.cc io/colmap_io.cc - io/gravity_io.cc + io/pose_io.cc math/gravity.cc math/rigid3d.cc math/tree.cc @@ -29,6 +30,7 @@ set(SOURCES set(HEADERS controllers/global_mapper.h controllers/option_manager.h + controllers/rotation_averager.h controllers/track_establishment.h controllers/track_retriangulation.h estimators/bundle_adjustment.h @@ -41,7 +43,7 @@ set(HEADERS estimators/view_graph_calibration.h io/colmap_converter.h io/colmap_io.h - io/gravity_io.h + io/pose_io.h math/gravity.h math/l1_solver.h math/rigid3d.h @@ -101,7 +103,10 @@ endif() add_executable(glomap_main glomap.cc exe/global_mapper.h - exe/global_mapper.cc) + exe/global_mapper.cc + exe/rotation_averager.h + exe/rotation_averager.cc +) target_link_libraries(glomap_main glomap) set_target_properties(glomap_main PROPERTIES OUTPUT_NAME glomap) @@ -111,6 +116,7 @@ install(TARGETS glomap_main DESTINATION bin) if(TESTS_ENABLED) add_executable(glomap_test controllers/global_mapper_test.cc + controllers/rotation_averager_test.cc ) target_link_libraries( glomap_test diff --git a/glomap/controllers/option_manager.cc b/glomap/controllers/option_manager.cc index d8041a6..3a7c8ff 100644 --- a/glomap/controllers/option_manager.cc +++ b/glomap/controllers/option_manager.cc @@ -1,6 +1,7 @@ #include "option_manager.h" #include "glomap/controllers/global_mapper.h" +#include "glomap/estimators/gravity_refinement.h" #include #include @@ -14,6 +15,7 @@ OptionManager::OptionManager(bool add_project_options) { image_path = std::make_shared(); mapper = std::make_shared(); + gravity_refiner = std::make_shared(); Reset(); desc_->add_options()("help,h", ""); @@ -266,6 +268,18 @@ void OptionManager::AddInlierThresholdOptions() { &mapper->inlier_thresholds.max_rotation_error); } +void OptionManager::AddGravityRefinerOptions() { + if (added_gravity_refiner_options_) { + return; + } + added_gravity_refiner_options_ = true; + AddAndRegisterDefaultOption("GravityRefiner.max_outlier_ratio", + &gravity_refiner->max_outlier_ratio); + AddAndRegisterDefaultOption("GravityRefiner.max_gravity_error", + &gravity_refiner->max_gravity_error); + AddAndRegisterDefaultOption("GravityRefiner.min_num_neighbors", + &gravity_refiner->min_num_neighbors); +} void OptionManager::Reset() { const bool kResetPaths = true; ResetOptions(kResetPaths); @@ -294,6 +308,7 @@ void OptionManager::ResetOptions(const bool reset_paths) { *image_path = ""; } *mapper = GlobalMapperOptions(); + *gravity_refiner = GravityRefinerOptions(); } void OptionManager::Parse(const int argc, char** argv) { diff --git a/glomap/controllers/option_manager.h b/glomap/controllers/option_manager.h index 0926387..030d38e 100644 --- a/glomap/controllers/option_manager.h +++ b/glomap/controllers/option_manager.h @@ -18,6 +18,7 @@ struct GlobalPositionerOptions; struct BundleAdjusterOptions; struct TriangulatorOptions; struct InlierThresholdOptions; +struct GravityRefinerOptions; class OptionManager { public: @@ -37,6 +38,7 @@ class OptionManager { void AddBundleAdjusterOptions(); void AddTriangulatorOptions(); void AddInlierThresholdOptions(); + void AddGravityRefinerOptions(); template void AddRequiredOption(const std::string& name, @@ -56,6 +58,7 @@ class OptionManager { std::shared_ptr image_path; std::shared_ptr mapper; + std::shared_ptr gravity_refiner; private: template @@ -88,6 +91,7 @@ class OptionManager { bool added_bundle_adjustment_options_ = false; bool added_triangulation_options_ = false; bool added_inliers_options_ = false; + bool added_gravity_refiner_options_ = false; }; template diff --git a/glomap/controllers/rotation_averager.cc b/glomap/controllers/rotation_averager.cc new file mode 100644 index 0000000..09039e8 --- /dev/null +++ b/glomap/controllers/rotation_averager.cc @@ -0,0 +1,61 @@ +#include "glomap/controllers/rotation_averager.h" + +namespace glomap { + +bool SolveRotationAveraging(ViewGraph& view_graph, + std::unordered_map& images, + const RotationAveragerOptions& options) { + view_graph.KeepLargestConnectedComponents(images); + + bool solve_1dof_system = options.use_gravity && options.use_stratified; + + ViewGraph view_graph_grav; + image_pair_t total_pairs = 0; + image_pair_t grav_pairs = 0; + if (solve_1dof_system) { + // Prepare two sets: ones all with gravity, and one does not have gravity. + // Solve them separately first, then solve them in a single system + for (const auto& [pair_id, image_pair] : view_graph.image_pairs) { + if (!image_pair.is_valid) continue; + + image_t image_id1 = image_pair.image_id1; + image_t image_id2 = image_pair.image_id2; + + Image& image1 = images[image_id1]; + Image& image2 = images[image_id2]; + + if (!image1.is_registered || !image2.is_registered) continue; + + total_pairs++; + + if (image1.gravity_info.has_gravity && image2.gravity_info.has_gravity) { + view_graph_grav.image_pairs.emplace( + pair_id, + ImagePair(image_id1, image_id2, image_pair.cam2_from_cam1)); + grav_pairs++; + } + } + } + + // If there is no image pairs with gravity or most image pairs are with + // gravity, then just run the 3dof version + bool status = (grav_pairs == 0) || (grav_pairs > total_pairs * 0.95); + solve_1dof_system = solve_1dof_system && (!status); + + if (solve_1dof_system) { + // Run the 1dof optimization + LOG(INFO) << "Solving subset 1DoF rotation averaging problem in the mixed " + "prior system"; + int num_img_grv = view_graph_grav.KeepLargestConnectedComponents(images); + RotationEstimator rotation_estimator_grav(options); + if (!rotation_estimator_grav.EstimateRotations(view_graph_grav, images)) { + return false; + } + view_graph.KeepLargestConnectedComponents(images); + } + + RotationEstimator rotation_estimator(options); + return rotation_estimator.EstimateRotations(view_graph, images); +} + +} // namespace glomap \ No newline at end of file diff --git a/glomap/controllers/rotation_averager.h b/glomap/controllers/rotation_averager.h new file mode 100644 index 0000000..cdf7389 --- /dev/null +++ b/glomap/controllers/rotation_averager.h @@ -0,0 +1,15 @@ +#pragma once + +#include "glomap/estimators/global_rotation_averaging.h" + +namespace glomap { + +struct RotationAveragerOptions : public RotationEstimatorOptions { + bool use_stratified = true; +}; + +bool SolveRotationAveraging(ViewGraph& view_graph, + std::unordered_map& images, + const RotationAveragerOptions& options); + +} // namespace glomap \ No newline at end of file diff --git a/glomap/controllers/rotation_averager_test.cc b/glomap/controllers/rotation_averager_test.cc new file mode 100644 index 0000000..dbab164 --- /dev/null +++ b/glomap/controllers/rotation_averager_test.cc @@ -0,0 +1,236 @@ +#include "glomap/controllers/rotation_averager.h" + +#include "glomap/controllers/global_mapper.h" +#include "glomap/estimators/gravity_refinement.h" +#include "glomap/io/colmap_io.h" +#include "glomap/math/rigid3d.h" +#include "glomap/types.h" + +#include +#include +#include + +#include + +namespace glomap { +namespace { + +void CreateRandomRotation(const double stddev, Eigen::Quaterniond& q) { + std::random_device rd{}; + std::mt19937 gen{rd()}; + + // Construct a random axis + double theta = double(rand()) / RAND_MAX * 2 * M_PI; + double phi = double(rand()) / RAND_MAX * M_PI; + Eigen::Vector3d axis(std::cos(theta) * std::sin(phi), + std::sin(theta) * std::sin(phi), + std::cos(phi)); + + // Construct a random angle + std::normal_distribution d{0, stddev}; + double angle = d(gen); + q = Eigen::AngleAxisd(angle, axis); +} + +void PrepareGravity(const colmap::Reconstruction& gt, + std::unordered_map& images, + double stddev_gravity = 0.0, + double outlier_ratio = 0.0) { + for (auto& image_id : gt.RegImageIds()) { + Eigen::Vector3d gravity = + gt.Image(image_id).CamFromWorld().rotation * Eigen::Vector3d(0, 1, 0); + + if (stddev_gravity > 0.0) { + Eigen::Quaterniond q; + CreateRandomRotation(DegToRad(stddev_gravity), q); + gravity = q * gravity; + } + + if (outlier_ratio > 0.0 && double(rand()) / RAND_MAX < outlier_ratio) { + Eigen::Quaterniond q; + CreateRandomRotation(1., q); + gravity = + Rigid3dToAngleAxis(Rigid3d(q, Eigen::Vector3d::Zero())).normalized(); + } + images[image_id].gravity_info.SetGravity(gravity); + } +} + +GlobalMapperOptions CreateMapperTestOptions() { + GlobalMapperOptions options; + options.skip_view_graph_calibration = false; + options.skip_relative_pose_estimation = false; + options.skip_rotation_averaging = true; + options.skip_track_establishment = true; + options.skip_global_positioning = true; + options.skip_bundle_adjustment = true; + options.skip_retriangulation = true; + + return options; +} + +RotationAveragerOptions CreateRATestOptions(bool use_gravity = false) { + RotationAveragerOptions options; + options.skip_initialization = true; + options.use_gravity = use_gravity; + return options; +} + +void ExpectEqualRotations(const colmap::Reconstruction& gt, + const colmap::Reconstruction& computed, + const double max_rotation_error_deg) { + const std::set reg_image_ids_set = gt.RegImageIds(); + std::vector reg_image_ids(reg_image_ids_set.begin(), + reg_image_ids_set.end()); + for (size_t i = 0; i < reg_image_ids.size(); i++) { + const image_t image_id1 = reg_image_ids[i]; + for (size_t j = 0; j < reg_image_ids.size(); j++) { + if (i == j) continue; + const image_t image_id2 = reg_image_ids[j]; + + const Rigid3d cam2_from_cam1 = + computed.Image(image_id2).CamFromWorld() * + colmap::Inverse(computed.Image(image_id1).CamFromWorld()); + + const Rigid3d cam2_from_cam1_gt = + gt.Image(image_id2).CamFromWorld() * + colmap::Inverse(gt.Image(image_id1).CamFromWorld()); + + double rotation_error_deg = CalcAngle(cam2_from_cam1_gt, cam2_from_cam1); + EXPECT_LT(rotation_error_deg, max_rotation_error_deg); + } + } +} + +void ExpectEqualGravity( + const colmap::Reconstruction& gt, + const std::unordered_map& images_computed, + const double max_gravity_error_deg) { + for (const auto& image_id : gt.RegImageIds()) { + const Eigen::Vector3d gravity_gt = + gt.Image(image_id).CamFromWorld().rotation * Eigen::Vector3d(0, 1, 0); + const Eigen::Vector3d gravity_computed = + images_computed.at(image_id).gravity_info.GetGravity(); + + double gravity_error_deg = CalcAngle(gravity_gt, gravity_computed); + EXPECT_LT(gravity_error_deg, max_gravity_error_deg); + } +} + +TEST(RotationEstimator, WithoutNoise) { + const std::string database_path = colmap::CreateTestDir() + "/database.db"; + + colmap::Database database(database_path); + colmap::Reconstruction gt_reconstruction; + colmap::SyntheticDatasetOptions synthetic_dataset_options; + synthetic_dataset_options.num_cameras = 2; + synthetic_dataset_options.num_images = 9; + synthetic_dataset_options.num_points3D = 50; + synthetic_dataset_options.point2D_stddev = 0; + colmap::SynthesizeDataset( + synthetic_dataset_options, >_reconstruction, &database); + + ViewGraph view_graph; + std::unordered_map cameras; + std::unordered_map images; + std::unordered_map tracks; + + ConvertDatabaseToGlomap(database, view_graph, cameras, images); + + // PrepareRelativeRotations(view_graph, images); + PrepareGravity(gt_reconstruction, images); + + GlobalMapper global_mapper(CreateMapperTestOptions()); + global_mapper.Solve(database, view_graph, cameras, images, tracks); + + // Version with Gravity + for (bool use_gravity : {true, false}) { + SolveRotationAveraging( + view_graph, images, CreateRATestOptions(use_gravity)); + + colmap::Reconstruction reconstruction; + ConvertGlomapToColmap(cameras, images, tracks, reconstruction); + ExpectEqualRotations( + gt_reconstruction, reconstruction, /*max_rotation_error_deg=*/1e-2); + } +} + +TEST(RotationEstimator, WithNoiseAndOutliers) { + const std::string database_path = colmap::CreateTestDir() + "/database.db"; + + // FLAGS_v = 1; + colmap::Database database(database_path); + colmap::Reconstruction gt_reconstruction; + colmap::SyntheticDatasetOptions synthetic_dataset_options; + synthetic_dataset_options.num_cameras = 2; + synthetic_dataset_options.num_images = 7; + synthetic_dataset_options.num_points3D = 100; + synthetic_dataset_options.point2D_stddev = 1; + synthetic_dataset_options.inlier_match_ratio = 0.6; + colmap::SynthesizeDataset( + synthetic_dataset_options, >_reconstruction, &database); + + ViewGraph view_graph; + std::unordered_map cameras; + std::unordered_map images; + std::unordered_map tracks; + + ConvertDatabaseToGlomap(database, view_graph, cameras, images); + + PrepareGravity(gt_reconstruction, images, /*stddev_gravity=*/3e-1); + + GlobalMapper global_mapper(CreateMapperTestOptions()); + global_mapper.Solve(database, view_graph, cameras, images, tracks); + + for (bool use_gravity : {true, false}) { + SolveRotationAveraging( + view_graph, images, CreateRATestOptions(use_gravity)); + + colmap::Reconstruction reconstruction; + ConvertGlomapToColmap(cameras, images, tracks, reconstruction); + if (use_gravity) + ExpectEqualRotations( + gt_reconstruction, reconstruction, /*max_rotation_error_deg=*/1.); + else + ExpectEqualRotations( + gt_reconstruction, reconstruction, /*max_rotation_error_deg=*/2.); + } +} + +TEST(RotationEstimator, RefineGravity) { + const std::string database_path = colmap::CreateTestDir() + "/database.db"; + + // FLAGS_v = 2; + colmap::Database database(database_path); + colmap::Reconstruction gt_reconstruction; + colmap::SyntheticDatasetOptions synthetic_dataset_options; + synthetic_dataset_options.num_cameras = 2; + synthetic_dataset_options.num_images = 100; + synthetic_dataset_options.num_points3D = 200; + synthetic_dataset_options.point2D_stddev = 0; + colmap::SynthesizeDataset( + synthetic_dataset_options, >_reconstruction, &database); + + ViewGraph view_graph; + std::unordered_map cameras; + std::unordered_map images; + std::unordered_map tracks; + + ConvertDatabaseToGlomap(database, view_graph, cameras, images); + + PrepareGravity( + gt_reconstruction, images, /*stddev_gravity=*/0., /*outlier_ratio=*/0.3); + + GlobalMapper global_mapper(CreateMapperTestOptions()); + global_mapper.Solve(database, view_graph, cameras, images, tracks); + + GravityRefinerOptions opt_grav_refine; + GravityRefiner grav_refiner(opt_grav_refine); + grav_refiner.RefineGravity(view_graph, images); + + // Check whether the gravity does not have error after refinement + ExpectEqualGravity(gt_reconstruction, images, /*max_gravity_error_deg=*/1e-2); +} + +} // namespace +} // namespace glomap diff --git a/glomap/estimators/global_rotation_averaging.cc b/glomap/estimators/global_rotation_averaging.cc index 3f7d031..e7122fe 100644 --- a/glomap/estimators/global_rotation_averaging.cc +++ b/glomap/estimators/global_rotation_averaging.cc @@ -16,6 +16,15 @@ double RelAngleError(double angle_12, double angle_1, double angle_2) { while (est < -EIGEN_PI) est += TWO_PI; + // Inject random noise if the angle is too close to the boundary to break the + // possible balance at the local minima + if (est > EIGEN_PI - 0.01 || est < -EIGEN_PI + 0.01) { + if (est < 0) + est += (rand() % 1000) / 1000.0 * 0.01; + else + est -= (rand() % 1000) / 1000.0 * 0.01; + } + return est; } } // namespace @@ -56,6 +65,9 @@ bool RotationEstimator::EstimateRotations( image.cam_from_world.rotation = Eigen::Quaterniond(AngleAxisToRotation( rotation_estimated_.segment(image_id_to_idx_[image_id], 3))); } + // Restore the prior position (t = -Rc = R * R_ori * t_ori = R * t_ori) + image.cam_from_world.translation = + (image.cam_from_world.rotation * image.cam_from_world.translation); } return true; @@ -207,6 +219,8 @@ void RotationEstimator::SetupLinearSystem( // Establish linear systems size_t curr_pos = 0; + std::vector weights; + weights.reserve(3 * view_graph.image_pairs.size()); for (const auto& [pair_id, image_pair] : view_graph.image_pairs) { if (!image_pair.is_valid) continue; @@ -221,6 +235,10 @@ void RotationEstimator::SetupLinearSystem( if (rel_temp_info_[pair_id].has_gravity) { coeffs.emplace_back(Eigen::Triplet(curr_pos, vector_idx1, -1)); coeffs.emplace_back(Eigen::Triplet(curr_pos, vector_idx2, 1)); + if (image_pair.weight >= 0) + weights.emplace_back(image_pair.weight); + else + weights.emplace_back(1); curr_pos++; } else { // If it is not gravity aligned, then we need to consider 3 dof @@ -245,6 +263,12 @@ void RotationEstimator::SetupLinearSystem( } else coeffs.emplace_back( Eigen::Triplet(curr_pos + 1, vector_idx2, 1)); + for (int i = 0; i < 3; i++) { + if (image_pair.weight >= 0) + weights.emplace_back(image_pair.weight); + else + weights.emplace_back(1); + } curr_pos += 3; } @@ -257,11 +281,13 @@ void RotationEstimator::SetupLinearSystem( images[fixed_camera_id_].gravity_info.has_gravity) { coeffs.emplace_back(Eigen::Triplet( curr_pos, image_id_to_idx_[fixed_camera_id_], 1)); + weights.emplace_back(1); curr_pos++; } else { for (int i = 0; i < 3; i++) { coeffs.emplace_back(Eigen::Triplet( curr_pos + i, image_id_to_idx_[fixed_camera_id_] + i, 1)); + weights.emplace_back(1); } curr_pos += 3; } @@ -269,6 +295,14 @@ void RotationEstimator::SetupLinearSystem( sparse_matrix_.resize(curr_pos, num_dof); sparse_matrix_.setFromTriplets(coeffs.begin(), coeffs.end()); + // Set up the weight matrix for the linear system + if (!options_.use_weight) { + weights_ = Eigen::ArrayXd::Ones(curr_pos); + } else { + weights_ = Eigen::ArrayXd(weights.size()); + for (size_t i = 0; i < weights.size(); i++) weights_[i] = weights[i]; + } + // Initialize x and b tangent_space_step_.resize(num_dof); tangent_space_residual_.resize(curr_pos); @@ -279,8 +313,8 @@ bool RotationEstimator::SolveL1Regression( L1SolverOptions opt_l1_solver; opt_l1_solver.max_num_iterations = 10; - L1Solver> l1_solver(opt_l1_solver, - sparse_matrix_); + L1Solver> l1_solver( + opt_l1_solver, weights_.matrix().asDiagonal() * sparse_matrix_); double last_norm = 0; double curr_norm = 0; @@ -295,7 +329,8 @@ bool RotationEstimator::SolveL1Regression( // use the current residual as b (Ax - b) tangent_space_step_.setZero(); - l1_solver.Solve(tangent_space_residual_, &tangent_space_step_); + l1_solver.Solve(weights_.matrix().asDiagonal() * tangent_space_residual_, + &tangent_space_step_); if (tangent_space_step_.array().isNaN().any()) { LOG(ERROR) << "nan error"; iteration++; @@ -393,7 +428,9 @@ bool RotationEstimator::SolveIRLS(const ViewGraph& view_graph, } // Update the factorization for the weighted values. - at_weight = sparse_matrix_.transpose() * weights_irls.matrix().asDiagonal(); + at_weight = sparse_matrix_.transpose() * + weights_irls.matrix().asDiagonal() * + weights_.matrix().asDiagonal(); llt.factorize(at_weight * sparse_matrix_); diff --git a/glomap/estimators/global_rotation_averaging.h b/glomap/estimators/global_rotation_averaging.h index 8899a1c..599c7c5 100644 --- a/glomap/estimators/global_rotation_averaging.h +++ b/glomap/estimators/global_rotation_averaging.h @@ -63,7 +63,7 @@ struct RotationEstimatorOptions { // Flg to use maximum spanning tree for initialization bool skip_initialization = false; - // TODO: Implement the weighted version for rotation averaging + // Flag to use weighting for rotation averaging bool use_weight = false; // Flag to use gravity for rotation averaging @@ -145,6 +145,9 @@ class RotationEstimator { // The fixed camera rotation (if with initialization, it would not be identity // matrix) Eigen::Vector3d fixed_camera_rotation_; + + // The weights for the edges + Eigen::ArrayXd weights_; }; } // namespace glomap diff --git a/glomap/estimators/gravity_refinement.cc b/glomap/estimators/gravity_refinement.cc index d1f96fa..0f679ba 100644 --- a/glomap/estimators/gravity_refinement.cc +++ b/glomap/estimators/gravity_refinement.cc @@ -12,6 +12,10 @@ void GravityRefiner::RefineGravity(const ViewGraph& view_graph, view_graph.image_pairs; const std::unordered_map>& adjacency_list = view_graph.GetAdjacencyList(); + if (adjacency_list.empty()) { + LOG(INFO) << "Adjacency list not established" << std::endl; + return; + } // Identify the images that are error prone int counter_rect = 0; @@ -75,6 +79,7 @@ void GravityRefiner::RefineGravity(const ViewGraph& view_graph, if (gravities.size() < options_.min_num_neighbors) continue; // Then, run refinment + gravity = AverageGravity(gravities); colmap::SetSphereManifold<3>(&problem, gravity.data()); ceres::Solver::Summary summary_solver; ceres::Solve(options_.solver_options, &problem, &summary_solver); diff --git a/glomap/estimators/view_graph_calibration.cc b/glomap/estimators/view_graph_calibration.cc index 30c7d34..a66b5a2 100644 --- a/glomap/estimators/view_graph_calibration.cc +++ b/glomap/estimators/view_graph_calibration.cc @@ -179,7 +179,7 @@ size_t ViewGraphCalibrator::FilterImagePairs(ViewGraph& view_graph) const { } LOG(INFO) << "invalid / total number of two view geometry: " - << invalid_counter << " / " << counter; + << invalid_counter << " / " << (counter / 2); return invalid_counter; } diff --git a/glomap/exe/rotation_averager.cc b/glomap/exe/rotation_averager.cc new file mode 100644 index 0000000..98460ab --- /dev/null +++ b/glomap/exe/rotation_averager.cc @@ -0,0 +1,105 @@ + +#include "glomap/controllers/rotation_averager.h" + +#include "glomap/controllers/option_manager.h" +#include "glomap/estimators/gravity_refinement.h" +#include "glomap/io/colmap_io.h" +#include "glomap/io/pose_io.h" +#include "glomap/types.h" + +#include +#include + +namespace glomap { +// ------------------------------------- +// Running Global Rotation Averager +// ------------------------------------- +int RunRotationAverager(int argc, char** argv) { + std::string relpose_path; + std::string output_path; + std::string gravity_path = ""; + std::string weight_path = ""; + + bool use_stratified = true; + bool refine_gravity = false; + bool use_weight = false; + + OptionManager options; + options.AddRequiredOption("relpose_path", &relpose_path); + options.AddRequiredOption("output_path", &output_path); + options.AddDefaultOption("gravity_path", &gravity_path); + options.AddDefaultOption("weight_path", &weight_path); + options.AddDefaultOption("use_stratified", &use_stratified); + options.AddDefaultOption("refine_gravity", &refine_gravity); + options.AddDefaultOption("use_weight", &use_weight); + options.AddGravityRefinerOptions(); + options.Parse(argc, argv); + + if (!colmap::ExistsFile(relpose_path)) { + LOG(ERROR) << "`relpose_path` is not a file"; + return EXIT_FAILURE; + } + + if (gravity_path != "" && !colmap::ExistsFile(gravity_path)) { + LOG(ERROR) << "`gravity_path` is not a file"; + return EXIT_FAILURE; + } + + if (weight_path != "" && !colmap::ExistsFile(weight_path)) { + LOG(ERROR) << "`weight_path` is not a file"; + return EXIT_FAILURE; + } + + if (use_weight && weight_path == "") { + LOG(ERROR) << "Weight path is required when use_weight is set to true"; + return EXIT_FAILURE; + } + + RotationAveragerOptions rotation_averager_options; + rotation_averager_options.skip_initialization = true; + rotation_averager_options.use_gravity = true; + + rotation_averager_options.use_stratified = use_stratified; + rotation_averager_options.use_weight = use_weight; + + // Load the database + ViewGraph view_graph; + std::unordered_map images; + + ReadRelPose(relpose_path, images, view_graph); + + if (gravity_path != "") { + ReadGravity(gravity_path, images); + } + + if (use_weight) { + ReadRelWeight(weight_path, images, view_graph); + } + + int num_img = view_graph.KeepLargestConnectedComponents(images); + LOG(INFO) << num_img << " / " << images.size() + << " are within the largest connected component"; + + if (refine_gravity && gravity_path != "") { + GravityRefiner grav_refiner(*options.gravity_refiner); + grav_refiner.RefineGravity(view_graph, images); + } + + colmap::Timer run_timer; + run_timer.Start(); + if (!SolveRotationAveraging(view_graph, images, rotation_averager_options)) { + LOG(ERROR) << "Failed to solve global rotation averaging"; + return EXIT_FAILURE; + } + run_timer.Pause(); + LOG(INFO) << "Global rotation averaging done in " + << run_timer.ElapsedSeconds() << " seconds"; + + // Write out the estimated rotation + WriteGlobalRotation(output_path, images); + LOG(INFO) << "Global rotation averaging done" << std::endl; + + return EXIT_SUCCESS; +} + +} // namespace glomap \ No newline at end of file diff --git a/glomap/exe/rotation_averager.h b/glomap/exe/rotation_averager.h new file mode 100644 index 0000000..0786b81 --- /dev/null +++ b/glomap/exe/rotation_averager.h @@ -0,0 +1,10 @@ +#pragma once + +#include "glomap/estimators/global_rotation_averaging.h" + +namespace glomap { + +// Use default values for most of the settings from database +int RunRotationAverager(int argc, char** argv); + +} // namespace glomap \ No newline at end of file diff --git a/glomap/glomap.cc b/glomap/glomap.cc index 19c7725..f9afc56 100644 --- a/glomap/glomap.cc +++ b/glomap/glomap.cc @@ -1,4 +1,5 @@ #include "glomap/exe/global_mapper.h" +#include "glomap/exe/rotation_averager.h" #include @@ -44,6 +45,7 @@ int main(int argc, char** argv) { std::vector> commands; commands.emplace_back("mapper", &glomap::RunMapper); commands.emplace_back("mapper_resume", &glomap::RunMapperResume); + commands.emplace_back("rotation_averager", &glomap::RunRotationAverager); if (argc == 1) { return ShowHelp(commands); @@ -62,7 +64,7 @@ int main(int argc, char** argv) { } if (matched_command_func == nullptr) { std::cout << "Command " << command << " not recognized. " - << "To list the available commands, run `colmap help`." + << "To list the available commands, run `glomap help`." << std::endl; return EXIT_FAILURE; } else { diff --git a/glomap/io/colmap_converter.cc b/glomap/io/colmap_converter.cc index a52455c..abfe1ff 100644 --- a/glomap/io/colmap_converter.cc +++ b/glomap/io/colmap_converter.cc @@ -188,14 +188,15 @@ void ConvertDatabaseToGlomap(const colmap::Database& database, << images_colmap.size() << std::flush; counter++; - image_t image_id = image.ImageId(); + const image_t image_id = image.ImageId(); if (image_id == colmap::kInvalidImageId) continue; auto ite = images.insert(std::make_pair( image_id, Image(image_id, image.CameraId(), image.Name()))); const colmap::PosePrior prior = database.ReadPosePrior(image_id); if (prior.IsValid()) { - ite.first->second.cam_from_world = Rigid3d( - colmap::Rigid3d(Eigen::Quaterniond::Identity(), prior.position)); + const colmap::Rigid3d world_from_cam_prior(Eigen::Quaterniond::Identity(), + prior.position); + ite.first->second.cam_from_world = Rigid3d(Inverse(world_from_cam_prior)); } else { ite.first->second.cam_from_world = Rigid3d(); } @@ -204,20 +205,18 @@ void ConvertDatabaseToGlomap(const colmap::Database& database, // Read keypoints for (auto& [image_id, image] : images) { - colmap::FeatureKeypoints keypoints = database.ReadKeypoints(image_id); - - image.features.reserve(keypoints.size()); - for (int i = 0; i < keypoints.size(); i++) { - image.features.emplace_back( - Eigen::Vector2d(keypoints[i].x, keypoints[i].y)); + const colmap::FeatureKeypoints keypoints = database.ReadKeypoints(image_id); + const int num_keypoints = keypoints.size(); + image.features.resize(num_keypoints); + for (int i = 0; i < num_keypoints; i++) { + image.features[i] = Eigen::Vector2d(keypoints[i].x, keypoints[i].y); } } // Add the cameras std::vector cameras_colmap = database.ReadAllCameras(); for (auto& camera : cameras_colmap) { - camera_t camera_id = camera.camera_id; - cameras[camera_id] = camera; + cameras[camera.camera_id] = camera; } // Add the matches diff --git a/glomap/io/gravity_io.cc b/glomap/io/gravity_io.cc deleted file mode 100644 index 7e974db..0000000 --- a/glomap/io/gravity_io.cc +++ /dev/null @@ -1,45 +0,0 @@ -#include "gravity_io.h" - -#include - -namespace glomap { -void ReadGravity(const std::string& gravity_path, - std::unordered_map& images) { - std::unordered_map name_idx; - for (const auto& [image_id, image] : images) { - name_idx[image.file_name] = image_id; - } - - std::ifstream file(gravity_path); - - // Read in the file list - std::string line, item; - Eigen::Vector3d gravity; - int counter = 0; - while (std::getline(file, line)) { - std::stringstream line_stream(line); - - // file_name - std::string name; - std::getline(line_stream, name, ' '); - - // Gravity - for (double i = 0; i < 3; i++) { - std::getline(line_stream, item, ' '); - gravity[i] = std::stod(item); - } - - // Check whether the image present - auto ite = name_idx.find(name); - if (ite != name_idx.end()) { - counter++; - images[ite->second].gravity_info.SetGravity(gravity); - // Make sure the initialization is aligned with the gravity - images[ite->second].cam_from_world.rotation = - images[ite->second].gravity_info.GetRAlign().transpose(); - } - } - LOG(INFO) << counter << " images are loaded with gravity" << std::endl; -} - -} // namespace glomap \ No newline at end of file diff --git a/glomap/io/gravity_io.h b/glomap/io/gravity_io.h deleted file mode 100644 index 4dabda2..0000000 --- a/glomap/io/gravity_io.h +++ /dev/null @@ -1,14 +0,0 @@ -#pragma once - -#include "glomap/scene/image.h" - -#include - -namespace glomap { -// Require the gravity in the format: image_name, gravity (3 numbers) -// Gravity should be the direction of [0,1,0] in the image frame -// image.cam_from_world * [0,1,0]^T = g -void ReadGravity(const std::string& gravity_path, - std::unordered_map& images); - -} // namespace glomap \ No newline at end of file diff --git a/glomap/io/pose_io.cc b/glomap/io/pose_io.cc new file mode 100644 index 0000000..eeda2e6 --- /dev/null +++ b/glomap/io/pose_io.cc @@ -0,0 +1,212 @@ +#include "pose_io.h" + +#include +#include +#include + +namespace glomap { +void ReadRelPose(const std::string& file_path, + std::unordered_map& images, + ViewGraph& view_graph) { + std::unordered_map name_idx; + image_t max_image_id = 0; + for (const auto& [image_id, image] : images) { + name_idx[image.file_name] = image_id; + + max_image_id = std::max(max_image_id, image_id); + } + + std::ifstream file(file_path); + + // Read in data + std::string line; + std::string item; + + size_t counter = 0; + + // Required data structures + // IMAGE_NAME_1 IMAGE_NAME_2 QW QX QY QZ TX TY TZ + while (std::getline(file, line)) { + std::stringstream line_stream(line); + + std::string file1, file2; + std::getline(line_stream, item, ' '); + file1 = item; + std::getline(line_stream, item, ' '); + file2 = item; + + if (name_idx.find(file1) == name_idx.end()) { + max_image_id += 1; + images.insert( + std::make_pair(max_image_id, Image(max_image_id, -1, file1))); + name_idx[file1] = max_image_id; + } + if (name_idx.find(file2) == name_idx.end()) { + max_image_id += 1; + images.insert( + std::make_pair(max_image_id, Image(max_image_id, -1, file2))); + name_idx[file2] = max_image_id; + } + + image_t index1 = name_idx[file1]; + image_t index2 = name_idx[file2]; + + image_pair_t pair_id = ImagePair::ImagePairToPairId(index1, index2); + + // rotation + Rigid3d pose_rel; + for (int i = 0; i < 4; i++) { + std::getline(line_stream, item, ' '); + pose_rel.rotation.coeffs()[(i + 3) % 4] = std::stod(item); + } + + for (int i = 0; i < 3; i++) { + std::getline(line_stream, item, ' '); + pose_rel.translation[i] = std::stod(item); + } + + view_graph.image_pairs.insert( + std::make_pair(pair_id, ImagePair(index1, index2, pose_rel))); + counter++; + } + LOG(INFO) << counter << " relpose are loaded" << std::endl; +} + +void ReadRelWeight(const std::string& file_path, + const std::unordered_map& images, + ViewGraph& view_graph) { + std::unordered_map name_idx; + for (const auto& [image_id, image] : images) { + name_idx[image.file_name] = image_id; + } + + std::ifstream file(file_path); + + // Read in data + std::string line; + std::string item; + + size_t counter = 0; + + // Required data structures + // IMAGE_NAME_1 IMAGE_NAME_2 QW QX QY QZ TX TY TZ + while (std::getline(file, line)) { + std::stringstream line_stream(line); + + std::string file1, file2; + std::getline(line_stream, item, ' '); + file1 = item; + std::getline(line_stream, item, ' '); + file2 = item; + + if (name_idx.find(file1) == name_idx.end() || + name_idx.find(file2) == name_idx.end()) + continue; + + image_t index1 = name_idx[file1]; + image_t index2 = name_idx[file2]; + + image_pair_t pair_id = ImagePair::ImagePairToPairId(index1, index2); + + if (view_graph.image_pairs.find(pair_id) == view_graph.image_pairs.end()) + continue; + + std::getline(line_stream, item, ' '); + view_graph.image_pairs[pair_id].weight = std::stod(item); + counter++; + } + LOG(INFO) << counter << " weights are used are loaded" << std::endl; +} + +void ReadGravity(const std::string& gravity_path, + std::unordered_map& images) { + std::unordered_map name_idx; + for (const auto& [image_id, image] : images) { + name_idx[image.file_name] = image_id; + } + + std::ifstream file(gravity_path); + + // Read in the file list + std::string line, item; + Eigen::Vector3d gravity; + int counter = 0; + while (std::getline(file, line)) { + std::stringstream line_stream(line); + + // file_name + std::string name; + std::getline(line_stream, name, ' '); + + // Gravity + for (double i = 0; i < 3; i++) { + std::getline(line_stream, item, ' '); + gravity[i] = std::stod(item); + } + + // Check whether the image present + auto ite = name_idx.find(name); + if (ite != name_idx.end()) { + counter++; + images[ite->second].gravity_info.SetGravity(gravity); + // Make sure the initialization is aligned with the gravity + images[ite->second].cam_from_world.rotation = + images[ite->second].gravity_info.GetRAlign().transpose(); + } + } + LOG(INFO) << counter << " images are loaded with gravity" << std::endl; +} + +void WriteGlobalRotation(const std::string& file_path, + const std::unordered_map& images) { + std::ofstream file(file_path); + std::set existing_images; + for (const auto& [image_id, image] : images) { + if (image.is_registered) { + existing_images.insert(image_id); + } + } + for (const auto& image_id : existing_images) { + const auto image = images.at(image_id); + if (!image.is_registered) continue; + file << image.file_name; + for (int i = 0; i < 4; i++) { + file << " " << image.cam_from_world.rotation.coeffs()[(i + 3) % 4]; + } + file << "\n"; + } +} + +void WriteRelPose(const std::string& file_path, + const std::unordered_map& images, + const ViewGraph& view_graph) { + std::ofstream file(file_path); + + // Sort the image pairs by image name + std::map name_pair; + for (const auto& [pair_id, image_pair] : view_graph.image_pairs) { + if (image_pair.is_valid) { + const auto image1 = images.at(image_pair.image_id1); + const auto image2 = images.at(image_pair.image_id2); + name_pair[image1.file_name + " " + image2.file_name] = pair_id; + } + } + + // Write the image pairs + for (const auto& [name, pair_id] : name_pair) { + const auto image_pair = view_graph.image_pairs.at(pair_id); + if (!image_pair.is_valid) continue; + file << images.at(image_pair.image_id1).file_name << " " + << images.at(image_pair.image_id2).file_name; + for (int i = 0; i < 4; i++) { + file << " " << image_pair.cam2_from_cam1.rotation.coeffs()[(i + 3) % 4]; + } + for (int i = 0; i < 3; i++) { + file << " " << image_pair.cam2_from_cam1.translation[i]; + } + file << "\n"; + } + + LOG(INFO) << name_pair.size() << " relpose are written" << std::endl; +} +} // namespace glomap \ No newline at end of file diff --git a/glomap/io/pose_io.h b/glomap/io/pose_io.h new file mode 100644 index 0000000..e98112f --- /dev/null +++ b/glomap/io/pose_io.h @@ -0,0 +1,37 @@ +#pragma once + +#include "glomap/scene/types_sfm.h" + +#include + +namespace glomap { +// Required data structures +// IMAGE_NAME_1 IMAGE_NAME_2 QW QX QY QZ TX TY TZ +void ReadRelPose(const std::string& file_path, + std::unordered_map& images, + ViewGraph& view_graph); + +// Required data structures +// IMAGE_NAME_1 IMAGE_NAME_2 weight +void ReadRelWeight(const std::string& file_path, + const std::unordered_map& images, + ViewGraph& view_graph); + +// Require the gravity in the format: +// IMAGE_NAME GX GY GZ +// Gravity should be the direction of [0,1,0] in the image frame +// image.cam_from_world * [0,1,0]^T = g +void ReadGravity(const std::string& gravity_path, + std::unordered_map& images); + +// Output would be of the format: +// IMAGE_NAME QW QX QY QZ +void WriteGlobalRotation(const std::string& file_path, + const std::unordered_map& images); + +// Output would be of the format: +// IMAGE_NAME_1 IMAGE_NAME_2 QW QX QY QZ TX TY TZ +void WriteRelPose(const std::string& file_path, + const std::unordered_map& images, + const ViewGraph& view_graph); +} // namespace glomap \ No newline at end of file diff --git a/glomap/math/gravity.cc b/glomap/math/gravity.cc index 485a4cd..30b83c6 100644 --- a/glomap/math/gravity.cc +++ b/glomap/math/gravity.cc @@ -31,4 +31,70 @@ Eigen::Matrix3d AngleToRotUp(double angle) { Eigen::Vector3d aa(0, angle, 0); return AngleAxisToRotation(aa); } + +// Code adapted from +// https://gist.github.com/PeteBlackerThe3rd/f73e9d569e29f23e8bd828d7886636a0 +Eigen::Vector3d AverageGravity(const std::vector& gravities) { + if (gravities.size() == 0) { + std::cerr + << "Error trying to calculate the average gravities of an empty set!\n"; + return Eigen::Vector3d::Zero(); + } + + // first build a 3x3 matrix which is the elementwise sum of the product of + // each quaternion with itself + Eigen::Matrix3d A = Eigen::Matrix3d::Zero(); + + for (int g = 0; g < gravities.size(); ++g) + A += gravities[g] * gravities[g].transpose(); + + // normalise with the number of gravities + A /= gravities.size(); + + // Compute the SVD of this 3x3 matrix + Eigen::JacobiSVD svd( + A, Eigen::ComputeThinU | Eigen::ComputeThinV); + + Eigen::VectorXd singular_values = svd.singularValues(); + Eigen::MatrixXd U = svd.matrixU(); + + // find the eigen vector corresponding to the largest eigen value + int largest_eigen_value_index = -1; + float largest_eigen_value; + bool first = true; + + for (int i = 0; i < singular_values.rows(); ++i) { + if (first) { + largest_eigen_value = singular_values(i); + largest_eigen_value_index = i; + first = false; + } else if (singular_values(i) > largest_eigen_value) { + largest_eigen_value = singular_values(i); + largest_eigen_value_index = i; + } + } + + Eigen::Vector3d average; + average(0) = U(0, largest_eigen_value_index); + average(1) = U(1, largest_eigen_value_index); + average(2) = U(2, largest_eigen_value_index); + + int negative_counter = 0; + for (int g = 0; g < gravities.size(); ++g) { + if (gravities[g].dot(average) < 0) negative_counter++; + } + if (negative_counter > gravities.size() / 2) { + average = -average; + } + + return average; +} + +double CalcAngle(const Eigen::Vector3d& gravity1, + const Eigen::Vector3d& gravity2) { + double cos_r = gravity1.dot(gravity2) / (gravity1.norm() * gravity2.norm()); + cos_r = std::min(std::max(cos_r, -1.), 1.); + + return std::acos(cos_r) * 180 / EIGEN_PI; +} } // namespace glomap \ No newline at end of file diff --git a/glomap/math/gravity.h b/glomap/math/gravity.h index 789ddd1..bf7a040 100644 --- a/glomap/math/gravity.h +++ b/glomap/math/gravity.h @@ -14,4 +14,9 @@ double RotUpToAngle(const Eigen::Matrix3d& R_up); // Get the upright rotation matrix from a rotation angle Eigen::Matrix3d AngleToRotUp(double angle); +// Estimate the average gravity direction from a set of gravity directions +Eigen::Vector3d AverageGravity(const std::vector& gravities); + +double CalcAngle(const Eigen::Vector3d& gravity1, + const Eigen::Vector3d& gravity2); } // namespace glomap diff --git a/glomap/scene/image.h b/glomap/scene/image.h index 1497efa..9dd94f0 100644 --- a/glomap/scene/image.h +++ b/glomap/scene/image.h @@ -14,7 +14,7 @@ struct GravityInfo { const Eigen::Matrix3d& GetRAlign() const { return R_align; } inline void SetGravity(const Eigen::Vector3d& g); - inline Eigen::Vector3d GetGravity(); + inline Eigen::Vector3d GetGravity() const { return gravity; }; private: // Direction of the gravity @@ -66,5 +66,4 @@ void GravityInfo::SetGravity(const Eigen::Vector3d& g) { has_gravity = true; } -Eigen::Vector3d GravityInfo::GetGravity() { return gravity; } } // namespace glomap diff --git a/glomap/scene/image_pair.h b/glomap/scene/image_pair.h index c913cab..fba6534 100644 --- a/glomap/scene/image_pair.h +++ b/glomap/scene/image_pair.h @@ -27,7 +27,7 @@ struct ImagePair { bool is_valid = true; // weight is the initial inlier rate - double weight = 0; + double weight = -1; // one of `ConfigurationType`. int config = colmap::TwoViewGeometry::UNDEFINED; From 262e122da0bedf2f48ec4735f84c93127862b2a1 Mon Sep 17 00:00:00 2001 From: Linfei Pan <36349740+lpanaf@users.noreply.github.com> Date: Thu, 20 Mar 2025 17:07:56 +0100 Subject: [PATCH 26/29] loose threshold for the test (#177) --- glomap/controllers/rotation_averager_test.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/glomap/controllers/rotation_averager_test.cc b/glomap/controllers/rotation_averager_test.cc index dbab164..4da6b98 100644 --- a/glomap/controllers/rotation_averager_test.cc +++ b/glomap/controllers/rotation_averager_test.cc @@ -190,7 +190,7 @@ TEST(RotationEstimator, WithNoiseAndOutliers) { ConvertGlomapToColmap(cameras, images, tracks, reconstruction); if (use_gravity) ExpectEqualRotations( - gt_reconstruction, reconstruction, /*max_rotation_error_deg=*/1.); + gt_reconstruction, reconstruction, /*max_rotation_error_deg=*/1.5); else ExpectEqualRotations( gt_reconstruction, reconstruction, /*max_rotation_error_deg=*/2.); From 484667b721e6b37a403fc975d5a0ca9842de48f8 Mon Sep 17 00:00:00 2001 From: Jhacson Meza Date: Wed, 7 May 2025 20:30:00 +0200 Subject: [PATCH 27/29] Check if value is set to avoid out-of-bound indexing (#178) --- glomap/estimators/bundle_adjustment.cc | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/glomap/estimators/bundle_adjustment.cc b/glomap/estimators/bundle_adjustment.cc index 57f85b1..0e5ee5d 100644 --- a/glomap/estimators/bundle_adjustment.cc +++ b/glomap/estimators/bundle_adjustment.cc @@ -208,11 +208,13 @@ void BundleAdjuster::ParameterizeVariables( } } - // Set the first camera to be fixed to remove the gauge ambiguity. - problem_->SetParameterBlockConstant( - images[center].cam_from_world.rotation.coeffs().data()); - problem_->SetParameterBlockConstant( - images[center].cam_from_world.translation.data()); + if (counter > 0) { + // Set the first camera to be fixed to remove the gauge ambiguity. + problem_->SetParameterBlockConstant( + images[center].cam_from_world.rotation.coeffs().data()); + problem_->SetParameterBlockConstant( + images[center].cam_from_world.translation.data()); + } // Parameterize the camera parameters, or set them to be constant if desired if (options_.optimize_intrinsics && !options_.optimize_principal_point) { From e02a801bd73f8b05f5b7654c5ab4cfd3714c45b0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Johannes=20Sch=C3=B6nberger?= Date: Mon, 12 May 2025 17:07:35 +0200 Subject: [PATCH 28/29] Update vcpkg and CI pipeline with latest changes in colmap (#193) * Update vcpkg to latest version * d * d * d --- .github/workflows/install-ccache.ps1 | 6 +-- .github/workflows/ubuntu.yml | 32 +++++++++---- .github/workflows/windows.yml | 53 +++++++++++++--------- cmake/FindDependencies.cmake | 5 +- scripts/format/{clang_format.sh => c++.sh} | 26 ++--------- 5 files changed, 66 insertions(+), 56 deletions(-) rename scripts/format/{clang_format.sh => c++.sh} (58%) diff --git a/.github/workflows/install-ccache.ps1 b/.github/workflows/install-ccache.ps1 index f56a953..c9d1503 100644 --- a/.github/workflows/install-ccache.ps1 +++ b/.github/workflows/install-ccache.ps1 @@ -4,10 +4,10 @@ param ( [string] $Destination ) -$version = "4.8" -$folder = "ccache-$version-windows-x86_64" +$version = "4.10.2" +$folder="ccache-$version-windows-x86_64" $url = "https://github.com/ccache/ccache/releases/download/v$version/$folder.zip" -$expectedSha256 = "A2B3BAB4BB8318FFC5B3E4074DC25636258BC7E4B51261F7D9BEF8127FDA8309" +$expectedSha256 = "6252F081876A9A9F700FAE13A5AEC5D0D486B28261D7F1F72AC11C7AD9DF4DA9" $ErrorActionPreference = "Stop" diff --git a/.github/workflows/ubuntu.yml b/.github/workflows/ubuntu.yml index ebd5654..f3c2afc 100644 --- a/.github/workflows/ubuntu.yml +++ b/.github/workflows/ubuntu.yml @@ -16,7 +16,13 @@ jobs: strategy: matrix: config: [ - + { + os: ubuntu-24.04, + cmakeBuildType: RelWithDebInfo, + asanEnabled: false, + cudaEnabled: false, + checkCodeFormat: true, + }, { os: ubuntu-22.04, cmakeBuildType: Release, @@ -35,7 +41,7 @@ jobs: os: ubuntu-22.04, cmakeBuildType: Release, asanEnabled: true, - cudaEnabled: true, + cudaEnabled: false, checkCodeFormat: false, }, { @@ -62,14 +68,15 @@ jobs: key: v${{ env.COMPILER_CACHE_VERSION }}-${{ matrix.config.os }}-${{ matrix.config.cmakeBuildType }}-${{ matrix.config.asanEnabled }}--${{ matrix.config.cudaEnabled }}-${{ github.run_id }}-${{ github.run_number }} restore-keys: v${{ env.COMPILER_CACHE_VERSION }}-${{ matrix.config.os }}-${{ matrix.config.cmakeBuildType }}-${{ matrix.config.asanEnabled }}--${{ matrix.config.cudaEnabled }} path: ${{ env.COMPILER_CACHE_DIR }} - - name: Install compiler cache run: | mkdir -p "$CCACHE_DIR" "$CTCACHE_DIR" echo "$COMPILER_CACHE_DIR/bin" >> $GITHUB_PATH + if [ -f "$COMPILER_CACHE_DIR/bin/ccache" ]; then exit 0 fi + set -x wget https://github.com/ccache/ccache/releases/download/v4.8.2/ccache-4.8.2-linux-x86_64.tar.xz echo "0b33f39766fe9db67f40418aed6a5b3d7b2f4f7fab025a8213264b77a2d0e1b1 ccache-4.8.2-linux-x86_64.tar.xz" | sha256sum --check @@ -81,14 +88,16 @@ jobs: echo "108b087f156a9fe7da0c796de1ef73f5855d2a33a27983769ea39061359a40fc ${ctcache_commit_id}.zip" | sha256sum --check unzip "${ctcache_commit_id}.zip" mv ctcache-${ctcache_commit_id}/clang-tidy* "$COMPILER_CACHE_DIR/bin" + - name: Check code format if: matrix.config.checkCodeFormat run: | set +x -euo pipefail - sudo apt-get update && sudo apt-get install -y clang-format-14 black - ./scripts/format/clang_format.sh + python -m pip install clang-format==19.1.0 + ./scripts/format/c++.sh git diff --name-only git diff --exit-code || (echo "Code formatting failed" && exit 1) + - name: Setup Ubuntu run: | sudo apt-get update && sudo apt-get install -y \ @@ -96,11 +105,9 @@ jobs: cmake \ ninja-build \ libboost-program-options-dev \ - libboost-filesystem-dev \ libboost-graph-dev \ libboost-system-dev \ libeigen3-dev \ - libsuitesparse-dev \ libceres-dev \ libflann-dev \ libfreeimage-dev \ @@ -116,7 +123,9 @@ jobs: libcgal-qt5-dev \ libgl1-mesa-dri \ libunwind-dev \ + libcurl4-openssl-dev \ xvfb + if [ "${{ matrix.config.cudaEnabled }}" == "true" ]; then if [ "${{ matrix.config.os }}" == "ubuntu-20.04" ]; then sudo apt-get install -y \ @@ -134,16 +143,19 @@ jobs: echo "CUDAHOSTCXX=/usr/bin/g++-10" >> $GITHUB_ENV fi fi + if [ "${{ matrix.config.asanEnabled }}" == "true" ]; then sudo apt-get install -y clang-15 libomp-15-dev echo "CC=/usr/bin/clang-15" >> $GITHUB_ENV echo "CXX=/usr/bin/clang++-15" >> $GITHUB_ENV fi + if [ "${{ matrix.config.cmakeBuildType }}" == "ClangTidy" ]; then sudo apt-get install -y clang-15 clang-tidy-15 libomp-15-dev echo "CC=/usr/bin/clang-15" >> $GITHUB_ENV echo "CXX=/usr/bin/clang++-15" >> $GITHUB_ENV fi + - name: Upgrade CMake run: | CMAKE_VERSION=3.28.6 @@ -152,6 +164,7 @@ jobs: tar -xzf ${CMAKE_DIR}.tar.gz sudo cp -r ${CMAKE_DIR}/* /usr/local/ rm -rf ${CMAKE_DIR}* + - name: Configure and build run: | set -x @@ -167,6 +180,7 @@ jobs: -DTESTS_ENABLED=ON \ -DASAN_ENABLED=${{ matrix.config.asanEnabled }} ninja -k 10000 + - name: Run tests if: ${{ matrix.config.cmakeBuildType != 'ClangTidy' }} run: | @@ -176,15 +190,17 @@ jobs: sleep 3 cd build ctest --output-on-failure -E .+colmap_.* + - name: Cleanup compiler cache run: | set -x ccache --show-stats --verbose ccache --evict-older-than 1d ccache --show-stats --verbose + echo "Size of ctcache before: $(du -sh $CTCACHE_DIR)" echo "Number of ctcache files before: $(find $CTCACHE_DIR | wc -l)" # Delete cache older than 10 days. find "$CTCACHE_DIR"/*/ -mtime +10 -print0 | xargs -0 rm -rf echo "Size of ctcache after: $(du -sh $CTCACHE_DIR)" - echo "Number of ctcache files after: $(find $CTCACHE_DIR | wc -l)"'' + echo "Number of ctcache files after: $(find $CTCACHE_DIR | wc -l)" diff --git a/.github/workflows/windows.yml b/.github/workflows/windows.yml index 1adc363..2cfde89 100644 --- a/.github/workflows/windows.yml +++ b/.github/workflows/windows.yml @@ -16,13 +16,6 @@ jobs: strategy: matrix: config: [ - { - os: windows-2019, - cmakeBuildType: Release, - cudaEnabled: false, - testsEnabled: true, - exportPackage: false, - }, { os: windows-2022, cmakeBuildType: Release, @@ -44,18 +37,35 @@ jobs: COMPILER_CACHE_DIR: ${{ github.workspace }}/compiler-cache CCACHE_DIR: ${{ github.workspace }}/compiler-cache/ccache CCACHE_BASEDIR: ${{ github.workspace }} - VCPKG_COMMIT_ID: e01906b2ba7e645a76ee021a19de616edc98d29f - VCPKG_BINARY_SOURCES: "clear;x-gha,readwrite" + VCPKG_COMMIT_ID: bc3512a509f9d29b37346a7e7e929f9a26e66c7e + GLOG_v: 1 + GLOG_logtostderr: 1 steps: - uses: actions/checkout@v4 - - - name: Export GitHub Actions cache env - uses: actions/github-script@v7 - with: - script: | - core.exportVariable('ACTIONS_CACHE_URL', process.env.ACTIONS_CACHE_URL || ''); - core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || ''); + + # We define the vcpkg binary sources using separate variables for read and + # write operations: + # * Read sources are defined as inline. These can be read by anyone and, + # in particular, pull requests from forks. Unfortunately, we cannot + # define these as action environment variables. See: + # https://github.com/orgs/community/discussions/44322 + # * Write sources are defined as action secret variables. These cannot be + # read by pull requests from forks but only from pull requests from + # within the target repository (i.e., created by a repository owner). + # This protects us from malicious actors accessing our secrets and + # gaining write access to our binary cache. For more information, see: + # https://securitylab.github.com/resources/github-actions-preventing-pwn-requests/ + - name: Setup vcpkg binary cache + shell: pwsh + run: | + # !!!PLEASE!!! be nice and don't use this cache for your own purposes. This is only meant for CI purposes in this repository. + $VCPKG_BINARY_SOURCES = "clear;x-azblob,https://colmap.blob.core.windows.net/github-actions-cache,sp=r&st=2024-12-10T17:29:32Z&se=2030-12-31T01:29:32Z&spr=https&sv=2022-11-02&sr=c&sig=bWydkilTMjRn3LHKTxLgdWrFpV4h%2Finzoe9QCOcPpYQ%3D,read" + if ("${{ secrets.VCPKG_BINARY_CACHE_AZBLOB_URL }}") { + # The secrets are only accessible in runs triggered from within the target repository and not forks. + $VCPKG_BINARY_SOURCES += ";x-azblob,${{ secrets.VCPKG_BINARY_CACHE_AZBLOB_URL }},${{ secrets.VCPKG_BINARY_CACHE_AZBLOB_SAS }},write" + } + echo "VCPKG_BINARY_SOURCES=${VCPKG_BINARY_SOURCES}" >> "${env:GITHUB_ENV}" - name: Compiler cache uses: actions/cache@v4 @@ -70,11 +80,9 @@ jobs: run: | New-Item -ItemType Directory -Force -Path "${{ env.CCACHE_DIR }}" echo "${{ env.COMPILER_CACHE_DIR }}/bin" | Out-File -Encoding utf8 -Append -FilePath $env:GITHUB_PATH - if (Test-Path -PathType Leaf "${{ env.COMPILER_CACHE_DIR }}/bin/ccache.exe") { exit } - .github/workflows/install-ccache.ps1 -Destination "${{ env.COMPILER_CACHE_DIR }}/bin" - name: Install CUDA @@ -86,9 +94,6 @@ jobs: sub-packages: '["nvcc", "nvtx", "cudart", "curand", "curand_dev", "nvrtc_dev"]' method: 'network' - - name: Install CMake and Ninja - uses: lukka/get-cmake@latest - - name: Setup vcpkg shell: pwsh run: | @@ -99,6 +104,12 @@ jobs: git reset --hard ${{ env.VCPKG_COMMIT_ID }} ./bootstrap-vcpkg.bat + - name: Install CMake and Ninja + uses: lukka/get-cmake@latest + with: + cmakeVersion: "3.31.0" + ninjaVersion: "1.12.1" + - name: Configure and build shell: pwsh run: | diff --git a/cmake/FindDependencies.cmake b/cmake/FindDependencies.cmake index a59323a..caa2275 100644 --- a/cmake/FindDependencies.cmake +++ b/cmake/FindDependencies.cmake @@ -1,7 +1,10 @@ set(CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake") find_package(Eigen3 3.4 REQUIRED) -find_package(SuiteSparse COMPONENTS CHOLMOD REQUIRED) +find_package(CHOLMOD QUIET) +if(NOT TARGET SuiteSparse::CHOLMOD) + find_package(SuiteSparse COMPONENTS CHOLMOD REQUIRED) +endif() find_package(Ceres REQUIRED COMPONENTS SuiteSparse) find_package(Boost REQUIRED) diff --git a/scripts/format/clang_format.sh b/scripts/format/c++.sh similarity index 58% rename from scripts/format/clang_format.sh rename to scripts/format/c++.sh index 2971018..07f09f3 100755 --- a/scripts/format/clang_format.sh +++ b/scripts/format/c++.sh @@ -2,28 +2,9 @@ # This script applies clang-format to the whole repository. -# Find clang-format -tools=' - clang-format -' - -clang_format='' -for tool in ${tools}; do - if type -p "${tool}" > /dev/null; then - clang_format=$tool - break - fi -done - -if [ -z "$clang_format" ]; then - echo "Could not locate clang-format" - exit 1 -fi -echo "Found clang-format: $(which ${clang_format})" - # Check version -version_string=$($clang_format --version | sed -E 's/^.*(\d+\.\d+\.\d+-.*).*$/\1/') -expected_version_string='14.0.0' +version_string=$(clang-format --version | sed -E 's/^.*(\d+\.\d+\.\d+-.*).*$/\1/') +expected_version_string='19.1.0' if [[ "$version_string" =~ "$expected_version_string" ]]; then echo "clang-format version '$version_string' matches '$expected_version_string'" else @@ -40,5 +21,4 @@ all_files=$( \ num_files=$(echo $all_files | wc -w) echo "Formatting ${num_files} files" -# Run clang-format -${clang_format} -i $all_files +clang-format -i $all_files From 01060b4be509902ea9aaefaae982a2cb941ae5c3 Mon Sep 17 00:00:00 2001 From: Linfei Pan <36349740+lpanaf@users.noreply.github.com> Date: Mon, 12 May 2025 18:09:00 +0200 Subject: [PATCH 29/29] improved readme (#194) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * improved readme * Update docs/rotation_averager.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update docs/getting_started.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update docs/rotation_averager.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update docs/getting_started.md Co-authored-by: Johannes Schönberger * Update docs/getting_started.md Co-authored-by: Johannes Schönberger --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Johannes Schönberger --- README.md | 2 +- docs/getting_started.md | 6 ++++++ docs/rotation_averager.md | 11 +++++++++-- 3 files changed, 16 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 5c388af..d6250b5 100644 --- a/README.md +++ b/README.md @@ -41,7 +41,7 @@ glomap mapper --database_path DATABASE_PATH --output_path OUTPUT_PATH --image_pa ``` For more details on the command line interface, one can type `glomap -h` or `glomap mapper -h` for help. -We also provide a guide on improving the obtained reconstruction, which can be found [here](docs/getting_started.md) +We also provide a guide on improving the obtained reconstruction, which can be found [here](docs/getting_started.md). Note: - GLOMAP depends on two external libraries - [COLMAP](https://github.com/colmap/colmap) and [PoseLib](https://github.com/PoseLib/PoseLib). diff --git a/docs/getting_started.md b/docs/getting_started.md index 6ae0651..a6399f5 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -44,3 +44,9 @@ retriangulation should already been performed. The number of global positioning and bundle adjustment iterations can be limited using the `--GlobalPositioning.max_num_iterations` and `--BundleAdjustment.max_num_iterations` options. + +#### Enable GPU-based solver + +If Ceres 2.3 or above is installed and cuDSS is installed, GLOMAP supports GPU +accelerated optimization. The process can be largely sped up with flags +`--GlobalPositioning.use_gpu 1 --BundleAdjustment.use_gpu`. diff --git a/docs/rotation_averager.md b/docs/rotation_averager.md index 5c4e994..1fc342d 100644 --- a/docs/rotation_averager.md +++ b/docs/rotation_averager.md @@ -50,8 +50,15 @@ The gravity direction file is expected to be of the following format ``` IMAGE_NAME GX GY GZ ``` -The gravity direction $g$ should $[0, 1, 0]$ if the image is parallel to the ground plane, and the estimated rotation would have the property that $R_i \cdot [0, 1, 0]^\top = g$. -If is acceptable if only a subset of all images have gravity direciton. +The gravity direction $g$ should $[0, 1, 0]$ if the image is orthogonal to the ground plane, and the estimated rotation would have the property that $R_i \cdot [0, 1, 0]^\top = g$. +More explicitly, suppose we can transpose a 3D point from the world coordinate to the image coordinate by RX + t = x. Here: +- `R` is a 3x3 rotation matrix that aligns the world coordinate system with the image coordinate system. +- `X` is a 3D point in the world coordinate system. +- `t` is a 3x1 translation vector that shifts the world coordinate system to the image coordinate system. +- `x` is the corresponding 3D point in the image coordinate system. +The gravity direction should be the second column of the rotation matrix `R`. + +It is acceptable if only a subset of all images have gravity direction. If the specified image name does not match any known image name from relative pose, it is ignored. ### Output