Our Contributions:
-
Comprehensive CSLR Workflow Implementation
Successfully executed the complete workflow, including feature extraction, ISLR training, and online CSLR inference. -
Training Process Optimization
Optimized the training pipeline. If you are experiencing long training times, you may refer to this repository for solutions. In our tests, we achieved approximately a 10x speedup. On 8× Nvidia A100 GPUs, training for 100 epochs on the csl-daily dataset (using the same hyperparameters as the original paper) now completes in 10.79 days. -
Keypoint Extraction Enhancement
Enhanced the efficiency of keypoint extraction and resolved out-of-memory (OOM) issues during this stage, resulting in at least a 10% increase in processing speed. -
Extended Dataset Support
Conducted training experiments on the ce_csl dataset and also provided a comprehensive preprocessing pipeline for custom datasets.
For further details, please refer to the README
The original repository README is provided below.
This repo contains the official implementations of the following papers on sign language processing:
-
[EMNLP 2024] Towards Online Continuous Sign Language Recognition and Translation [Paper] [Code]
-
[ECCV 2024] A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars [Paper] [Code]
-
[CVPR 2023] Natural Language-Assisted Sign Language Recognition [Paper] [Code]
-
[CVPR 2023] CiCo: Domain-Aware Sign Language Retrieval via Cross-Lingual Contrastive Learning [Paper] [Code]
-
[NeurIPS 2022] Two-Stream Network for Sign Language Recognition and Translation [Paper] [Code]
-
[CVPR 2022] A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation [Paper] [Code]
Please cite our works if you find this repo is helpful.
@inproceedings{zuo2024towards,
title={Towards Online Continuous Sign Language Recognition and Translation},
author={Zuo, Ronglai and Wei, Fangyun and Mak, Brian},
booktitle={EMNLP},
year={2024}
}
@inproceedings{zuo2024simple,
title={A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars},
author={Zuo, Ronglai and Wei, Fangyun and Chen, Zenggui and Mak, Brian and Yang, Jiaolong and Tong, Xin},
booktitle={ECCV},
year={2024}
}
@inproceedings{zuo2023natural,
title={Natural Language-Assisted Sign Language Recognition},
author={Zuo, Ronglai and Wei, Fangyun and Mak, Brian},
booktitle={CVPR},
year={2023}
}
@inproceedings{cheng2023cico,
title={CiCo: Domain-Aware Sign Language Retrieval via Cross-Lingual Contrastive Learning},
author={Cheng, Yiting and Wei, Fangyun and Jianmin, Bao and Chen, Dong and Zhang, Wen Qiang},
booktitle={CVPR},
year={2023}
}
@article{chen2022two,
title={Two-Stream Network for Sign Language Recognition and Translation},
author={Chen, Yutong and Zuo, Ronglai and Wei, Fangyun and Wu, Yu and Liu, Shujie and Mak, Brian},
journal={NeurIPS},
year={2022}
}
@inproceedings{chen2022simple,
title={A simple multi-modality transfer learning baseline for sign language translation},
author={Chen, Yutong and Wei, Fangyun and Sun, Xiao and Wu, Zhirong and Lin, Stephen},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={5120--5130},
year={2022}
}
@inproceedings{wei2023improving,
title={Improving Continuous Sign Language Recognition with Cross-Lingual Signs},
author={Wei, Fangyun and Chen, Yutong},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
pages={23612--23621},
year={2023}
}