Zerui Chen1 Shizhe Chen1 Etienne Arlaud1 Ivan Laptev2 Cordelia Schmid1
1WILLOW, INRIA Paris, France
2MBZUAI
This is the implementation of ViViDex under the MuJoCo simulator, a novel system for learning dexterous manipulation skills from human videos:
git clone --recursive https://github.com/zerchen/vividex_mujoco.git
conda create -n rl python=3.10
conda activate rl
conda install pytorch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 pytorch-cuda=12.1 -c pytorch -c nvidia
pip install -r requirements.txt
conda install pytorch-scatter -c pyg
cd tools
# Train the state-based relocation policy
python train.py env.name=seq_name
# Train the state-based pouring policy
python train.py env=dexycb_pour
# Train the state-based placing-inside policy
python train.py env=dexycb_placeAvailable seq_name can be found at: hand_imitation/env/models/assets/objects/trajectories/ycb. You can also download trained checkpoints here and check their config files for a reference. Please note that change 36 to 42 for L197-198 in hand_imitation/env/models/control.py when try to load my trained checkpoints for pour and place inside tasks. When state-based policies are trained, rollout these policies and train the visual policy using tools/dist_bc_train.py.
Parts of the code are based on DexMV and TCDM. We thank the authors for sharing their excellent work!
If you find ViViDex useful for your research, please consider citing our paper:
@inproceedings{chen2025vividex,
title={{ViViDex}: Learning Vision-based Dexterous Manipulation from Human Videos},
author={Chen, Zerui and Chen, Shizhe and Arlaud, Etienne and Laptev, Ivan and Schmid, Cordelia},
booktitle={ICRA},
year={2025}
}