HyperFree: A Channel-adaptive and Tuning-free Foundation Model for Hyperspectral Remote Sensing Imagery (CVPR2025)
Jingtao Li∗, Yingyi Liu∗, Xinyu Wang†, Yunning Peng, Chen Sun, Shaoyu Wang, Zhendong Sun, Tian Ke, Xiao Jiang, Tangwei Lu, Anran Zhao, Yanfei Zhong†
∗ Equal contribution, † Corresponding author
Paper | Code | Hyper-Seg Engine | Website | WeChat
Update | Outline | Hyper-Seg Data Engine | Pretrained Checkpoint | Tuning-free Usage | Acknowledgement
2025.02.27
- HyperFree is accepted by CVPR2025! (paper)
- We propose the first tuning-free hyperspectral foundation model, which can process any hyperspectral image in different tasks with promptable or zero-shot manner.
- A weight dictionary that spans the full spectrum, enabling dynamic generation of the embedding layer for varied band numbers according to input wavelengths.
- We propose to map both prompts and masks into feature space to identify multiple semantic-aware masks for one prompt, where different interaction workflows are designed for each downstream tasks.
- We built the Hyper-Seg data engine to train the HyperFree model and tested it on 11 datasets from 5 tasks in tuning-free manner, 14 datasets from 8 tasks in tuning manner as an extensive experiment.
Overview of HyperFree.
|
|
|
- We built a data engine called Hyper-Seg to generate segmented masks automatically for spectral images and expand the data scale for promptable training. Below is the engine workflow and we finally obtained 41900 high-resolution image pairs with size of 512×512×224.
- The dataset is available at here.
HyperFree is mainly tested with ViT-b version and the corresponding checkpoint is available at Hugging Face. Download it and put in the Ckpt folder.
HyperFree can complete five tasks including multi-class classification, one-class classification, target detection, anomaly detection, and change detection in tuning-free manner. We have provided both sample data (Data folder) and corresponding scripts (Fine-tuning-free-manner Folder).
Tips: In practice, we find that preprocessing operations such as removing noise bands and contrast enhancement can significantly improve the processing performance
- Hyperspectral multi-class classification. For each new image, change the below hyper-paramaters for promptable classification.
data_path = "./../../Data/hyperspectral_classification/WHU-Hi-LongKou.tif"
wavelengths = [429.410004, 439.230011, 449.059998,......]
GSD = 0.456 # Ground sampling distance (m/pixel)
num_classes = 3 # At least one prompt for each class
few_shots[0][120, 324] = 1
few_shots[1][258, 70] = 1
few_shots[2][159, 18] = 1- Hyperspectral one-class classification. For each new image, change the below hyper-paramaters for promptable classification.
parser.add_argument('-ds', '--data_path', type=str,
help='Dataset')
parser.add_argument('-sl', '--wavelengths', type=str,
help='Central wavelength of sensor')
parser.add_argument('-g', '--GSD', type=float, default='0.043', help='Ground sampling distance (m/pixel)')
parser.add_argument('-p', '--prompt_point', nargs='+', type=int, default=[600, 90],
help='The index of prompt_point')- Hyperspectral target detection. For each new image, change the below hyper-paramaters for promptable segmentation.
img_pth = './../../Data/hyperspectral_target_detection/Stone.mat'
wavelengths = [429.410004, 439.230011, 449.059998,......]
GSDS = 0.07 # Ground sampling distance (m/pixel)
target_spectrum = './../../Data/hyperspectral_target_detection/target_spectrum_stone.txt' # Storing the target spectraum- Hyperspectral anomaly detection. For each new image, change the below hyper-paramaters for zero-shot detection.
path = './../../Data/hyperspectral_anomaly_detection/abu-beach-2.mat'
wavelengths = [429.410004, 439.230011, 449.059998,......]
GSDS = 7.5 # Ground sampling distance (m/pixel)
area_ratio_threshold = 0.0009 # Decide how small the targets are anomalies- Hyperspectral change detection. For each new image, change the below hyper-paramaters for zero-shot detection. (mask_path is optional)
img1_paths = ['./../../Data/hyperspectral_change_detection/Hermiston/val/time1/img1_1.tif', ......] # Images at first time-step
img2_paths = ['./../../Data/hyperspectral_change_detection/Hermiston/val/time2/img2_1.tif', ......] # Images at second time-step
wavelengths = [429.410004, 439.230011, 449.059998,......]
GSD = 30 # Ground sampling distance (m/pixel)
ratio_threshold = 0.76 # a float, pixels with the change score higher than ratio_threshold quantile are considered as changes@inproceedings{li2025hyperfree,
title={HyperFree: A Channel-adaptive and Tuning-free Foundation Model for Hyperspectral Remote Sensing Imagery},
author={Li Jingtao and Liu yingyi and Wang Xinyu and Peng Yunning and Sun Chen and Wang Shaoyu and Sun Zhendong and Ke Tian and Jiang Xiao and Lu Tangwei and Zhao anran and Zhong Yanfei},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2025}
}
This project is based on SAM. Thank them for bringing prompt engineering from NLP into the visual field!