GV-Bench: Benchmarking Local Feature Matching for Geometric Verification of Long-term Loop Closure Detection
Jingwen Yu, Hanjing Ye, Jianhao Jiao, Ping Tan, Hong Zhang
2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
arXiv, IEEEXplore, Project Page, Blog Post (in Chinese), Video (Bilibili)
In short, GV-Bench provides a benchmark for evaluating different local feature matching methods on geometric verification (GV), which is crucial for vision-based localization and mapping systems (e.g., Visual SLAM, Structure-from-Motion, Visual Localization).
- ⭐ Add support for Image-matching-models, thanks for their excellent work! 😄
- 🎉 Chinese intro on wechat offical account.
- ⭐ Paper is release on arxiv.
- 🎉 The paper is accepted by IROS 2024!
- 🚀 Releasing the visualization of image matching results.
- 🚀 Releasing the benchmark!
Please follow the installation of image-matching-models
# clone the repo
git clone --recursive https://github.com/jarvisyjw/GV-Bench.git
cd third_party/image-matching-models
git submodule init
git submodule update
conda create -n gvbench python=3.11
python -m pip install -U prettytable
# Then follow the installation of image-matching-models- Get the GV-Bench sequences from here.
- Unzip and organize the dataset folder like following:
|-- gt
| |-- day.txt
| |-- night.txt
| |-- nordland.txt
| |-- season.txt
| |-- uacampus.txt
| |-- weather.txt
|-- images
| |-- day0
| |-- day1
| |-- night0
| |...Now, we support using image-matching-models directly, which enables many more state-of-the-art matching methods for Geometric Verification (GV). The example usage:
Show all supported geometric verification models
python main.py --helppython main.py config/day.yamlIn the configs, please specify the data directory, sequence info (which is provided with the repo in dataset/gt folder), image_size and the matcher you want to use.
The evaluation supports runing multiple matchers in a single run. However, our sequences provides a rather large number of pairs, so the evaluation might takes time.
If you want to replicate the paper's result of IROS2024, please refer to this.
- We recommend contributing or building your image matcher as the standard of image-matching-models.
- Example usage of bench loader and evaluator
# bench sequence gvbench_seq = ImagePairDataset(config.data, transform=None) # load images labels = gvbench_seq.label # load labels MODEL = YOUR_MODEL(max_num_keypoints=2048) # your image matching model # if your method is two-stage image matching # i.e. Step 1: Keypoints Extraction # Step 2: Keypoints matching # We recommend you set a max_num_keypoints to 2048 # to be consistent with the image-matching-models default setup. scores = [] for data in gvbench_seq: img0 = load_image(data['img0'], resize) img1 = load_image(data['img1'], resize) inliers = MODEL(img0, img1) scores.append(inliers) # normalize scores_norm = (scores - np.min(scores)) / (np.max(scores)- np.min(scores)) mAP, MaxR = eval(scores, labels)
- This work thanks hloc, image-matching-models for their amazing works.
- Contact:
jingwen.yu@connect.ust.hk
@inproceedings{yu2024gv,
title={GV-Bench: Benchmarking Local Feature Matching for Geometric Verification of Long-term Loop Closure Detection},
author={Yu, Jingwen and Ye, Hanjing and Jiao, Jianhao and Tan, Ping and Zhang, Hong},
booktitle={2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
pages={7922--7928},
year={2024},
organization={IEEE}
}