-
You can create an anaconda environment called adnerf with:
conda env create -f environment.yml conda init conda activate adnerf pip install torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html -
Recommend install from a local clone
conda install -c bottler nvidiacub cd pytorch3d && pip install -e .或者通过下载包的方式安装nvidiacub。不过最好的方法是将一些依赖包删掉
pip uninstall tqdm termcolor tabulate pyyaml protalocker yacs iopath fvcore，即使采用了镜像迁移也最好删掉....，然后让pytorch3d去自动收集并安装。但是torch或者pytorch需要提前安好。dfrf使用的torch，可以使用torch。pytorch不清楚
To rebuild after installing from a local clone run,rm -rf build/ **/*.sothenpip install -e ..
总之，pytorch3d的安装是玄学，ubuntu的版本也许也有关系. 换源的方法不太靠谱。直接纯净环境下安装依旧可能安不上。Ubuntu20.04可能安不上. 换成18.04应该能安上 -
Put "01_MorphableModel.mat" to data_util/face_tracking/3DMM/; cd data_util/face_tracking; run
python convert_BFM.py
\如果得到结果为Nan，最好在face tracking文件夹中的render 3dmm中设置为blur_radius=0和faces_per_pixel=1
-
Data Preprocess ($id Obama for example)
bash process_data.sh Obama- Input: A portrait video at 25fps containing voice audio. (dataset/vids/$id.mp4)
- Output: folder dataset/$id that contains all files for training
-
Train Two NeRFs (Head-NeRF and Torso-NeRF)
- Train Head-NeRF with command
python NeRFs/HeadNeRF/run_nerf.py --config dataset/$id/HeadNeRF_config.txt - Copy latest trainied model from dataset/$id/logs/$id_head to dataset/$id/logs/$id_com
- Train Torso-NeRF with command
python NeRFs/TorsoNeRF/run_nerf.py --config dataset/$id/TorsoNeRF_config.txt - You may need the pretrained models to avoid bad initialization. #3
- Train Head-NeRF with command
- Reconstruct original video with audio input
python NeRFs/TorsoNeRF/run_nerf.py --config dataset/$id/TorsoNeRFTest_config.txt --aud_file=dataset/$id/aud.npy --test_size=300 - Drive the target person with another audio input
python NeRFs/TorsoNeRF/run_nerf.py --config dataset/$id/TorsoNeRFTest_config.txt --aud_file=${deepspeechfile.npy} --test_size=-1
If you find our work useful in your research, please consider citing our paper:
@inproceedings{guo2021adnerf,
title={AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis},
author={Yudong Guo and Keyu Chen and Sen Liang and Yongjin Liu and Hujun Bao and Juyong Zhang},
booktitle={IEEE/CVF International Conference on Computer Vision (ICCV)},
year={2021}
}
If you have questions, feel free to contact gyd2011@mail.ustc.edu.cn.
We use face-parsing.PyTorch for parsing head and torso maps, and DeepSpeech for audio feature extraction. The NeRF model is implemented based on NeRF-pytorch.