Ziyang Xu1, Huangxuan Zhao2, Ziwei Cui1, Wenyu Liu1, Chuansheng Zheng2, Xinggang Wang1 📧
(📧) corresponding author.
1 Institute of AI, School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan, China. 2 Department of Radiology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China.
-orange){kind=icon}
-blue){kind=icon}
-
July 10, 2024: We released our code. If you found it helpful, please give us a star 🌟 and cite our paper! 🤗 -
July 4, 2024: MoSt-DSA is accepted to ECAI2024! 🍻 The preprint paper 📔 can be found on arxiv.
Artificial intelligence has become a crucial tool for medical image analysis. As an advanced cerebral angiography technique, Digital Subtraction Angiography (DSA) poses a challenge where the radiation dose to humans is proportional to the image count. By reducing images and using AI interpolation instead, the radiation can be cut significantly. However, DSA images present more complex motion and structural features than natural scenes, making interpolation more challenging. We propose MoSt-DSA, the first work that uses deep learning for DSA frame interpolation. Unlike natural scene Video Frame Interpolation (VFI) methods that extract unclear or coarse-grained features, we devise a general module that models motion and structural context interactions between frames in an efficient full convolution manner by adjusting optimal context range and transforming contexts into linear functions. Benefiting from this, MoSt-DSA is also the first method that directly achieves any number of interpolations at any time steps with just one forward pass during both training and testing. We conduct extensive comparisons with 7 representative VFI models for interpolating 1 to 3 frames, MoSt-DSA demonstrates robust results across 470 DSA image sequences (each typically 152 images), with average SSIM over 0.93, average PSNR over 38 (standard deviations of less than 0.030 and 3.6, respectively), comprehensively achieving state-of-the-art performance in accuracy, speed, visual effect, and memory usage.
-
MoSt-DSA is the first work that uses deep learning for DSA frame interpolation, and also the first method that directly achieves any number of interpolations at any time steps with just one forward pass during both training and testing.
-
MoSt-DSA demonstrates robust results across 470 DSA image sequences (each typically 152 images), with average SSIM over 0.93, average PSNR over 38 (standard deviations of less than 0.030 and 3.6, respectively), comprehensively achieving SOTA performance in accuracy, speed, visual effect, and memory usage.
-
MoSt-DSA can significantly reduce the DSA radiation dose received by doctors and patients when applied clinically, lowering it by 50%, 67%, and 75% when interpolating 1 to 3 frames, respectively.
- python 3.8
- cudatoolkit 11.2.1
- cudnn 8.1.0.77
- See 'MoSt-DSA_env.txt' for Python libraries required
conda create -n MoSt_DSA python=3.8
conda activate MoSt_DSA
conda install cudatoolkit=11.2.1 cudnn=8.1.0.77
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html
# cd /xx/xx/MoSt-DSA
pip install -r MoSt-DSA_env.txtDownload the checkpoints of single-frame interpolation, two-frame interpolation, three-frame interpolation, and put all pkl files into ../MoSt-DSA/weights/checkpoints. You can use these checkpoints as pre-training weights or for inference.
Run the following commands to generate single/multi-frame interpolation:
- Single-frame interpolation
python Simple_Interpolator.py \
--model_path ./weights/checkpoints/Inf1.pkl \
--frame1 ./demo_images/DSA_a.png \
--frame2 ./demo_images/DSA_b.png \
--inter_frames 1- Two-frame interpolation
python Simple_Interpolator.py \
--model_path ./weights/checkpoints/Inf2.pkl \
--frame1 ./demo_images/DSA_a.png \
--frame2 ./demo_images/DSA_b.png \
--inter_frames 2- Three-frame interpolation
python Simple_Interpolator.py \
--model_path ./weights/checkpoints/Inf3.pkl \
--frame1 ./demo_images/DSA_a.png \
--frame2 ./demo_images/DSA_b.png \
--inter_frames 3You should extract consecutive frames of data and then organize your dataset as follows:
- For single-frame interpolation
├── <Your_datasets_root>
│ ├── Data_part_1
│ ├── Sequence_1
│ ├── frame_1.png
│ ├── frame_2_gt.png
│ ├── frame_3.png
│ ├── Sequence_2
│ ├── frame_1.png
│ ├── frame_2_gt.png
│ ├── frame_3.png
│ ├── ...
│ ├── Data_part_2
│ ├── Sequence_1
│ ├── frame_1.png
│ ├── frame_2_gt.png
│ ├── frame_3.png
│ ├── Sequence_2
│ ├── frame_1.png
│ ├── frame_2_gt.png
│ ├── frame_3.png
│ ├── ...
│ | ...
- For multi-frame interpolation(for example, three-frame)
├── <Your_datasets_root>
│ ├── Data_part_1
│ ├── Sequence_1
│ ├── frame_1.png
│ ├── frame_2_gt.png
│ ├── frame_3_gt.png
│ ├── frame_4_gt.png
│ ├── frame_5.png
│ ├── Sequence_2
│ ├── frame_1.png
│ ├── frame_2_gt.png
│ ├── frame_3_gt.png
│ ├── frame_4_gt.png
│ ├── frame_5.png
│ ├── ...
│ ├── Data_part_2
│ ├── Sequence_1
│ ├── frame_1.png
│ ├── frame_2_gt.png
│ ├── frame_3_gt.png
│ ├── frame_4_gt.png
│ ├── frame_5.png
│ ├── Sequence_2
│ ├── frame_1.png
│ ├── frame_2_gt.png
│ ├── frame_3_gt.png
│ ├── frame_4_gt.png
│ ├── frame_5.png
│ ├── ...
│ | ...
Also, you should list the paths of all the sequences for train-set and test-set by making TrainList.txt and TestList.txt, as follows:
- Make TrainList.txt for train-set. Each line in txt corresponds to a sequence path, like:
Data_part_1/Sequence_1
Data_part_1/Sequence_2
Data_part_1/Sequence_3
Data_part_1/Sequence_4
Data_part_2/Sequence_1
Data_part_2/Sequence_2
Data_part_2/Sequence_3
Data_part_2/Sequence_4
...
- Make TestList.txt for test-set. Each line in txt corresponds to a sequence path, like:
Data_part_301/Sequence_1
Data_part_301/Sequence_2
Data_part_301/Sequence_3
Data_part_301/Sequence_4
Data_part_302/Sequence_1
Data_part_302/Sequence_2
Data_part_302/Sequen
8437
ce_3
Data_part_302/Sequence_4
...
For calculating style loss and perceptual loss, download weight and put it into ../MoSt-DSA/weights/vgg_weight.
Taking the training single-frame interpolation model as an example. You can also modify the "--inter_frames" value to specify other number of interpolation frames. If you try to do so, please make sure that "--data_path" and "--txt_path" correspond to the "--inter_frames".
CUDA_VISIBLE_DEVICES=0 \
python -m torch.distributed.launch --nproc_per_node=1 --master_port 37501 Train_Test_Pipe.py \
--world_size 1 \
--batch_size 8 \
--data_path /xx/xx/Your_datasets_root \
--txt_path /xx/xx/txts_root/that/has/TrainList/and/TestList/txt \
--inter_frames 1 \
--model_save_name xxx_model \
--note name_of_log_folder \
--note_print xxx_experiment \
--max_epochs 50CUDA_VISIBLE_DEVICES=0,1,2,3 \
python -m torch.distributed.launch --nproc_per_node=4 --master_port 37502 Train_Test_Pipe.py \
--world_size 4 \
--batch_size 8 \
--data_path /xx/xx/Your_datasets_root \
--txt_path /xx/xx/txts_root/that/has/TrainList/and/TestList/txt \
--inter_frames 1 \
--model_save_name xxx_model \
--note name_of_log_folder \
--note_print xxx_experiment \
--max_epochs 50Taking the evaluating single-frame interpolation model as an example. You can also modify the "--inter_frames" value to specify other number of interpolation frames. If you try to do so, please make sure that "--data_path" and "--txt_path" correspond to the "--inter_frames".
CUDA_VISIBLE_DEVICES=0 \
python -m torch.distributed.launch --nproc_per_node=1 --master_port 37503 Train_Test_Pipe.py \
--world_size 1 \
--data_path /xx/xx/Your_datasets_root \
--txt_path /xx/xx/txts_root/that/has/TrainList/and/TestList/txt \
--inter_frames 1 \
--note Eval-Model-xxx \
--note_print My-Eval-Inf2 \
--pretrain_weight /xxx/xxx/xxx/modelxxx.pkl \
--only_eval TrueThis project is based on EMA-VFI, Lambda Networks, FILM, Perceptual Loss, Style Loss. Thanks for their wonderful works.
If you find MoSt-DSA is useful in your research or applications, please feel free to give us a star 🌟 and cite our paper:
@InProceedings{Xu2024MoSt-DSA,
title={MoSt-DSA: Modeling Motion and Structural Interactions for Direct Multi-Frame Interpolation in DSA Images},
author={Ziyang Xu and Huangxuan Zhao and Ziwei Cui and Wenyu Liu and Chuansheng Zheng and Xinggang Wang},
booktitle={European Conference on Artificial Intelligence},
pages={537-544},
year={2024},
url={https://ebooks.iospress.nl/pdf/doi/10.3233/FAIA240531},
}or
@misc{Xu2024MoSt-DSA,
title={MoSt-DSA: Modeling Motion and Structural Interactions for Direct Multi-Frame Interpolation in DSA Images},
author={Ziyang Xu and Huangxuan Zhao and Ziwei Cui and Wenyu Liu and Chuansheng Zheng and Xinggang Wang},
year={2024},
eprint={2407.07078},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2407.07078},
}