Official PyTorch implementation of "MoDiTalker: Motion-Disentangled Diffusion Model for High-Fidelity Talking Head Generation".
Seyeon Kim∗1,2,
Siyoon Jin∗1,
Jihye Park∗1,2,
Kihong Kim3,
Jiyoung Kim1,
Jisu Nam4 and
Seungryong Kim†4.
∗ Equal contribution ,†Corresponding author
1Korea University, 2Samsung Electronics, 3VIVE STUDIOS, 4KAIST
conda create -n MoDiTalker python=3.8 -y
conda activate MoDiTalker
python -m pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu116
python -m pip install natsort tqdm gdown omegaconf einops lpips pyspng tensorboard imageio av moviepy numba p_tqdm soundfile face_alignemntWe utilized two datasets for training each stage. Please refer and follow the dataset preparation from here
Get the BFM_model_front.mat, similarity_Lm3D_all.mat and Exp_Pca.bin, and place them to the MoDiTalker/data/data_utils/deep_3drecon/BFM directory.
Obtain 'BaselFaceModel.tgz and extract a file named 01_MorphableModel.mat and place it to the MoDiTalker/data/data_utils/deep_3drecon/BFM directory.
We had to revise a single code inside the package accelerate due to version conflicts. If some conflicts occur during loading data, please revise the code in accelerate/dataloader.py following this;
from
batch_size = dataloader.batch_size if dataloader.batch_size is not None else dataloader.batch_sampler.batch_sizeto
batch_size = dataloader.batch_size if dataloader.batch_size is not None else len(dataloader.batch_sampler[0])cd AToM
bash scripts/train.shThe checkpoints of AToM will be saved in ./runs
The checkpoints of AToM will be saved in ./runs
First, execute the following script:
cd MToV
bash scripts/train/first_stg.sh Then the script will automatically create the folder in ./log_dir to save logs and checkpoints.
Second, execute the following script:
cd MToV
bash scripts/train/first_stg_ldmk.sh You may change the model configs via modifying configs/autoencoder. Moreover, one needs early-stopping to further train the model with the GAN loss (typically 8k-14k iterations with a batch size of 8).
cd MToV
bash scripts/train/second_stg.shWe provide the corresponding checkpoints in the below:
Download and place them in the ./checkpoints/ directory.
| Link to download | |
|---|---|
| AToM | link |
| MToV motion autoencoder | link |
| MToV RGB autoencoder | link |
Before producing the motions from audio, there's need to preprocess the audio since we process audio in the type of HuBeRT. To produce hubert feature of audio you want, please follow the script below:
cd data
python data_utils/preprocess/process_audio.py \
--audio path to audio \
--ref_dir path to directory of reference images Then the processed audio hubert(npy) will be saved in data/inference/hubert/{sampling rate}
Note that, you need to specify the path to (1) reference images (2) processed hubert and (3) checkpoint in the following bash script.
cd AToM
bash scripts/inference.shThe results of AToM will be saved in AToM/results/frontalized_npy and this path should be consistent with the ldmk_path of the following step.
Note that, you need to specify the path to (1) reference images and (2) produced landmark.
cd data/data_utils
python motion_align/align_face_recon.py \
--ldmk_path path to directory of generated landmark \
--driv_video_path path to directory of reference images The final landmarks will be saved in AToM/results/aligned_npy.
cd MToV
bash scripts/inference/sample.shThe final videos will be saved in MToV/results.
@misc{kim2024moditalker,
title={MoDiTalker: Motion-Disentangled Diffusion Model for High-Fidelity Talking Head Generation},
author={Seyeon Kim and Siyoon Jin and Jihye Park and Kihong Kim and Jiyoung Kim and Jisu Nam and Seungryong Kim},
year={2024},
eprint={2403.19144},
archivePrefix={arXiv},
primaryClass={cs.CV}
}This code is mainly built upon EDGE and PVDM.
We also used the code from following repository: GeneFace.