Customized python interface for bundle adjustment #2509

B1ueber2y · 2024-04-11T20:03:13Z

This draft PR includes the following updates:

refactorization of the BundleAdjuster with better modular design (with equivalent logic)
Python pipeline (https://github.com/B1ueber2y/colmap/blob/features/custom_ba/pycolmap/custom_bundle_adjustment.py) for "adjust_local_bundle", "adjust_global_bundle", "iterative_local_refinement", "iterative_global_refinement" with equivalent logic as C++. The bundle adjustment interface in the custom_incremental_mapping is also updated into the new Python implementation
(planned feature, not working yet). Exposing the ceres problem from BundleAdjuster to be extensible in Python with pyceres

This PR is marked as draft because the attempt on exposing the ceres problem is not working until now. Specifically, the commented section here (https://github.com/B1ueber2y/colmap/blob/features/custom_ba/pycolmap/custom_bundle_adjustment.py#L14-L27) raises segfault at L24, where pyceres.solve cannot work on a C++ constructed ceres problem. (I hope this is not a silly bug)

Since i m on vacation from tomorrow to Tuesday night and do not want to potentially block relevant development, I submit this PR and sincerely ask for help on this particular issue. let me cc potentially relevant experts on pybind11 and pyceres in case you have time to help :) @sarlinpe @Phil26AT @ahojnnes @Zador-Pataki

Thanks in advance!

…struction.points3D performance

B1ueber2y · 2024-04-11T20:05:36Z

Also to add to this, the new example python code on bundle adjustment needs pyceres as a dependency once the issue is fixed. We need to specify this somewhere.

Also, Python code needs to be formatted with black after this is fixed.

B1ueber2y · 2024-04-20T14:57:05Z

As the issue of extending C++ constructed ceres problem has not been resolved until now, I finished another alternative attempt by reimplementing a python class of bundle adjuster (PyBundleAdjuster) here: https://github.com/B1ueber2y/colmap/blob/features/custom_ba/pycolmap/custom_bundle_adjustment.py#L12

The example interface is here: https://github.com/B1ueber2y/colmap/blob/features/custom_ba/pycolmap/custom_bundle_adjustment.py#L193-L195, with C++ and Python alternatives for constructing and solving the bundle adjustment, corresponding to estimators/bundle_adjustment.h in COLMAP.

From my test, the bundle adjustment results from this is completely equivalent to the C++ counterpart, while having the ceres problem exposed with pyceres. The fact that the problem is constructed in Python leads to slower performance (12s to 25s for the example test)

With this feature, it will be pretty easy to:

add losses on top of the COLMAP BA from different sparse features, sensors and geometric modeling
migrate into different python-based optimization framework other than ceres (by substituting the problem construction and solving) and potentially integrating learning-based deep losses.

If agreed, we can put it into the pycolmap package, such that one can use for example pycolmap.PyBundleAdjuster for gettting the ceres problem in pyceres form from COLMAP.

Minor: The opaque binding of ListPoint2D is deprecated since #2443, leading to slow copying behavior for image.points2D. Thus, I added a "Point2D" getter and use "image.point2D(point2D_id)" for now before this is fixed.

cc: @sarlinpe @ahojnnes

B1ueber2y · 2024-04-24T16:11:09Z

The issue is solved. It was due to a lifetime issue for the loss function, where there needs to be a stand-alone line in Python to hold the lifetime of the loss function: https://github.com/B1ueber2y/colmap/blob/features/custom_ba/pycolmap/custom_bundle_adjustment.py#L196.

Now the binding of bundle adjuster pycolmap.BundleAdjuster can work perfectly as here: https://github.com/B1ueber2y/colmap/blob/features/custom_ba/pycolmap/custom_bundle_adjustment.py#L193-L203, with similar performance. The Python class PyBundleAdjuster is commented as an alternative for the reference.

sarlinpe · 2024-04-24T16:27:08Z

src/colmap/estimators/bundle_adjustment.h

@@ -170,14 +172,22 @@ class BundleAdjuster {

  bool Solve(Reconstruction* reconstruction);

-  // Get the Ceres solver summary for the last call to `Solve`.
+  // Set up the problem
+  void SetUpProblem(Reconstruction* reconstruction,


Would it make sense to pass the reconstruction to the constructor such that it doesn't need to be provided as argument most of its member functions? I can't see any use case where we need to use a different reconstruction between function calls. Maybe for future PRs @ahojnnes

B1ueber2y · 2024-04-24T16:27:10Z

However, just realizing that although the issue can be solved with the fix, with the current design one still needs to be careful that the lifetime of the loss function is held somewhere in Python, which is extremely dangerous. Let me add a loss function attribute in the COLMAP bundle adjustment options from the C++ end, to hold the life of the loss.

B1ueber2y · 2024-04-24T16:57:53Z

However, just realizing that although the issue can be solved with the fix, with the current design one still needs to be careful that the lifetime of the loss function is held somewhere in Python, which is extremely dangerous. Let me add a loss function attribute in the COLMAP bundle adjustment options from the C++ end, to hold the life of the loss.

In the end, this was done with a cleaner way by using the keep_alive feature in pybind11.

src/colmap/estimators/bundle_adjustment.cc

src/pycolmap/estimators/bundle_adjustment.cc

sarlinpe · 2024-04-28T15:46:25Z

src/pycolmap/pipeline/sfm.cc

@@ -296,6 +296,7 @@ void BindSfM(py::module& m) {
  auto PyBundleAdjustmentOptions =


Shouldn't we move the binding of BundleAdjustmentOptions to estimators/bundle_adjustment.cc since it is defined in the corresponding COLMAP file?

I think we should. Also we should move the binding of IncrementalPipelineOptions to the corresponding file in sfm folder as well. I didnt update the structure pipeline/sfm.cc for both commits but I also believe that moving would be more reasonable.

Ok can do in the next PR then

src/pycolmap/pipeline/sfm.cc

src/pycolmap/sfm/incremental_mapper.cc

pycolmap/custom_bundle_adjustment.py

src/colmap/sfm/incremental_mapper.h

src/colmap/estimators/bundle_adjustment.h

src/pycolmap/pipeline/sfm.cc

henry123-boy · 2025-05-13T16:43:30Z

Thank you very much for making Ceres and COLMAP user - friendly with Python. I'm trying to add new residuals by defining a custom cost function in Python on top of pycolmap.create_default_bundle_adjuster, but it runs much slower than the original. Could you please give me some guidance on how to modify the relevant parts in the COLMAP C++ source code and rebuild pycolmap? Any advice would be greatly appreciated.

B1ueber2y added 16 commits April 8, 2024 20:53

remove unused teardown for raw bundle adjustment. add some getters.

9bc9e20

remove unused line.

9e72fb0

restructure bundle adjustment.

ad5dccb

add return value (ceres::Problem) for SetUpProblem.

3f5ec5c

fix format.

e702175

bind bundle adjustment draft.

31227c2

update. fix local ba.

0361bb7

revert.

09b5b1f

update pybind to v2.12. fix py::bind_map.

9c4b850

move the make_opaque for Point3DMap to reconstruction.cc to fix recon…

e54a158

…struction.points3D performance

add reconstruction bindings for accessing members directly by id.

26f61b6

fix format.

b6dde46

minor.

8daee6b

useless fix.

962a776

push debug code.

9d80081

fix format.

78ccc9b

B1ueber2y marked this pull request as draft April 11, 2024 20:03

B1ueber2y added 10 commits April 18, 2024 08:59

Merge branch 'dev' into features/custom_ba

dab7fb6

Merge branch 'fix/reconstruction_points3D' into features/custom_ba

5a128a1

merge with main.

c4793d9

update.

7f168a6

Merge branch 'dev' into features/custom_ba

209d100

update.

07c676a

fix formatting.

49020ca

update.

8740cde

add minor comment.

e4505d6

minor

2276cb7

B1ueber2y marked this pull request as ready for review April 20, 2024 14:57

B1ueber2y added 2 commits April 24, 2024 18:06

minor.

58fc7c2

fix formatting.

4f0b199

B1ueber2y marked this pull request as ready for review April 24, 2024 16:11

sarlinpe reviewed Apr 24, 2024

View reviewed changes

B1ueber2y marked this pull request as draft April 24, 2024 16:27

B1ueber2y added 2 commits April 24, 2024 18:56

add keep_alive to hold the life of loss.

a4012a3

fix formatting.

4f020ce

B1ueber2y marked this pull request as ready for review April 24, 2024 16:57

Merge branch 'main' into features/custom_ba

21debe0

sarlinpe reviewed Apr 28, 2024

View reviewed changes

B1ueber2y added 3 commits April 28, 2024 21:17

some updates on syntax.

3e85da7

fix formatting.

ee810cb

make problem_ a shared pointer rather than unique.

67f61f5

sarlinpe reviewed Apr 28, 2024

View reviewed changes

src/colmap/sfm/incremental_mapper.h Outdated Show resolved Hide resolved

src/colmap/estimators/bundle_adjustment.h Outdated Show resolved Hide resolved

B1ueber2y added 5 commits April 28, 2024 21:53

update.

2762e59

change const ref to value for shared_ptr return type

3133c41

remove the binding of ceres options from pycolmap.

edfb040

add minor comments for pyceres.

9378633

minor. add pyceres comments to the head of BA python file

e71d297

sarlinpe reviewed Apr 30, 2024

View reviewed changes

src/pycolmap/pipeline/sfm.cc Outdated Show resolved Hide resolved

minor + add ceres version string.

5226714

sarlinpe approved these changes May 1, 2024

View reviewed changes

sarlinpe merged commit 6fb0283 into colmap:main May 1, 2024

B1ueber2y deleted the features/custom_ba branch May 1, 2024 20:00

B1ueber2y mentioned this pull request May 2, 2024

Potential integration into pycolmap Parskatt/micro-bundle-adjustment#2

Open

sarlinpe mentioned this pull request Dec 3, 2024

How to set pycolmap BA parameter/s #2978

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Customized python interface for bundle adjustment #2509

Customized python interface for bundle adjustment #2509

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

		@@ -296,6 +296,7 @@ void BindSfM(py::module& m) {
		auto PyBundleAdjustmentOptions =

Customized python interface for bundle adjustment #2509

Customized python interface for bundle adjustment #2509

Conversation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!