Use with: mocap_importer
Wrapping code repositories of various motion capture papers & researches, to provide a unified interface through CLI. Simplify their installation and running.
Only tested on Linux. Not stable yet.
sincerelly thanks to gvhmr/wilor/wilor-mini developers and others that help each other
Caution
Currently broken! Code refactoring.... TODO code refactor in v0.2.1:
- MCP standard support: json-rpc
- use
pixianduvinstead ofmamba- simplify by override toml: prefix-dev/pixi#3890
- test case and CI, to make the installation process hardly failed.
- wilor continuous predict. pixi config set --local run-post-link-scripts insecure
Warning
There's a huge code refactoring in the up-comming release v0.2.1. The run part in v0.2.0 should be ok, see issue#2 即将发布的 v0.2.1 版本中将进行大规模代码重构，v0.2.0的运行部分应该没有问题，请参阅 #2
| Feature 功能 | |
|---|---|
| ✅ Done | GVHMR, WiLoR |
| 🚀 国内镜像加速 | 🚧 |
| 🐧Linux | 🚧 Implementing |
| 🪟Windows | ❓ Need tested |
| 🍎 OSX | ❓ |
| 📔Jupyter Notebook | ❓ |
| 🤖 MCP | 🕒 TODO |
stage-by-stage scheme
For budget, you can start with a cheap scheme and then upgrade to a more expensive one.
interactable virtual scene & record video & post-calculate → realtime calculate with better GPU → realtime with hardware support
GPU → iphone → quest3
| minimum | medium | higher | |
|---|---|---|---|
| ♿🖥sit | 🖱⌨️🎮user input as body motion/interact in game |
GPU(face+hand) +cam📷 |
UE live link(face) +mini tripod🔭 |
| sit cost | $0 game like VRchat | ≥$200 GPU; $0 if use phone as cam | $0 live link app; ≥$350 iphone; $1 mini tripod |
| 🧍♂️🎣stand | tripod cam🔭+ GPU(body+hand) |
UE live link(face) +Headrig | UE vcam to interact |
| stand cost $520 | $20 tripod; ≥$200 GPU | $0 live link app; $300 rokoko headrig | $0 vcam app |
| 👓VR | quest3 realtime hand mocap & natural interact |
pico 4 pro or DIY hardware for face |
tracker hardware or GPU software for body |
| VR cost $850 | $400 | ? | $450 tracker or ≥$200 GPU |
Rank: body🕺 hand👋 face👤 text to motion文
| model | paper | commit | issue | comment |
|---|---|---|---|---|
✅🕺GVHMR  |
  |
  |
2024, VRAM > 3GB | |
🚧🕺TRAM  |
  |
  |
2024, suit for fast-motion, but VRAM > 6GB | |
🕒🕺CoMotion  |
  |
  |
2025, belongs to Apple | |
🕒🕺SAT-HMR  |
  |
  |
2025 | |
✅👋WiLoR  |
  |
  |
2024, use mini, fast, VRAM > 2.5GB | |
🚧👋Dyn-HaMR  |
  |
  |
2025 | |
🕒👋Hamba  |
  |
  |
2025 | |
🕒👋OmniHands  |
  |
  |
2024 | |
🕒👋HaMeR  |
  |
  |
2023 | |
🕒👋HOISDF  |
  |
  |
2024, better on occulusion | |
🕒👤SPIGA  |
  |
  |
2022 | |
🕒👤mediapipe  |
  |
  |
realtime | |
🕒文🎵 MotionAnything  |
  |
  |
2025, waiting code release | |
🕒文 momask-codes  |
  |
  |
2024 |
- hand: no constant tracking for video(just no yolo, ready for photo but not video)
- 🕺👋👤-文🎵 Genmo （Nvidia Lab）
- 🕺👋👤Look Ma, no markers: holistic performance capture without the hassle
- 👤D-ViT
| Solution | Comment | |
|---|---|---|
| 👤face | 🍎iFacialMocap (iPhone X + PC(win/Mac)) 🤖Meowface (free, Android, can work with iFacialMocap PC client) |
Shape key |
| 🍎+💻Unreal Live Link | Bone | |
| hand/body | VR headset or VR trackers |
- 🍎
iphone≥X(12/13 best)for better face mocap result on UE live link, though you can use android🤖 to do live link.
The scripts will smartly skip or update pixi,uv,mocap-wrapper,7z,aria2c,ffmpeg,git if they're installed or in system $PATH.
# sudo -i; bash <(curl -sSL https://gitee.com/SuperManito/LinuxMirrors/raw/main/ChangeMirrors.sh) # 一键设置linux镜像(可选)
curl https://raw.githubusercontent.com/AClon314/mocap-wrapper/refs/heads/master/src/mocap_wrapper/install/pixi.py | python -- -y
mocap --install -b gvhmr,wilorThe python scripts are equivalent to the following:
#!/bin/bash -eou pipefail
# 1. pixi.py: use system python
curl -fsSL https://pixi.sh/install.sh | sh
pixi global install uv
uv python install
~/.pixi/bin/uv pip install git+https://github.com/AClon314/mocap-wrapper
# 2. mocap --install -b ''
sudo apt install 7z aria2 ffmpeg git # pixi global install 7z aria2 ffmpeg git
git clone https://github.com/zju3dv/GVHMR
aria2c hmr4d.ckpt # download pre-trained
# 3. mocap-wrapper in uv; gvhmr/wilor... in pixi env seperately
. ~/.venv/bin/activate
mocap -i input.mp4 -b gvhmr
cd $SEARCH_DIR/GVHMR
pixi run run/gvhmr.py%%{init:{'flowchart':{'padding':0, 'htmlLabels':false}, 'htmlLabels':false, 'theme':'base', 'themeVariables':{'primaryColor':'#fff','clusterBkg':'#fff','edgeLabelBackground':'#fff','lineColor':'#888','primaryTextColor':'#000','primaryBorderColor':'#000','secondaryTextColor':'#000', 'clusterBorder':'#888','tertiaryTextColor':'#000'} }}%%
graph TD
p[pixi]
u[uv]
m[mocap]
p --global install--> u
u --global venv--> m
p --search_dir--> gvhmr,wilor...
m --> 7z,aria2c,ffmpeg...
See mocap -h for more options.
mocap -i input.mp4
mocap -i input.mp4 -b gvhmr,wilor -o outdirA useful data visualize tool to expand .pt/.npy/.npz
You have to read these if you want to modify code.
LOG=debug mocap -I# docker build -t mocap -f docker/Dockerfile .
podman build -t mocap -f docker/Dockerfile . --security-opt label=disable
# github action local
act -j test -v --action-offline-mode --bind --reuse --env LOG=D # --rm=falsekey: Armature mapping from;Algorithm run;who;begin;prop[0];prop[1]...
example:
- smplx;gvhmr;person0;0;body_pose = array([...], dtype=...)
- smplx;wilor;person1;9;hand_pose = ...
- smplx;wilor;person1;1;bbox = ...
ps: the blender addon use Armature mapping to
- pose: thetas, θ
- betas: shape, β
- expression: psi, ψ
- trans(lation) 平移
- global_orient: rotate 旋转
- bbox: yolo_bbox_xyXY
By using this repository, you must also comply with the terms of these external licenses:
| Repo | License |
|---|---|
| GVHMR | Copyright 2022-2023 3D Vision Group at the State Key Lab of CAD&CG, Zhejiang University. All Rights Reserved.  |
| WiLoR |  |
| mocap-wrapper | AGPL v3 |