This repository is the official implementation of the CVPR 2024 paper 3D Facial Expressions through Analysis-by-Neural Synthesis.
SMIRK reconstructs 3D faces from monocular images with facial geometry that faithfully recover extreme, asymmetric, and subtle expressions.
You need to have a working version of PyTorch and Pytorch3D installed. We provide a requirements.txt file that can be used to install the necessary dependencies for a Python 3.9 setup with CUDA 11.7:
conda create -n smirk python=3.9
pip install -r requirements.txt
# install pytorch3d now
pip install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py39_cu117_pyt201/download.htmlThen, in order to download the required models, run:
bash quick_install.shThe above installation includes downloading the FLAME model. This requires registration. If you do not have an account you can register at https://flame.is.tue.mpg.de/
This command will also download the SMIRK pretrained model which can also be found on Google Drive.
We provide two demos. One that can be used to test the model on a single image,
python demo.py --input_path samples/test_image2.png --out_path results/ --checkpoint pretrained_models/SMIRK_em1.pt --cropand one that can be used to test the model on a video,
python demo_video.py --input_path samples/dafoe.mp4 --out_path results/ --checkpoint pretrained_models/SMIRK_em1.pt --crop --render_origDataset Preparation
SMIRK was trained on a combination of the following datasets: LRS3, MEAD, CelebA, and FFHQ.
-
§§Download the LRS3 dataset from here.We are aware that currently this dataset has been removed from the website. It can be replaced with any other similar dataset, e.g. LRS2. -
Download the MEAD dataset from here.
-
Download the CelebA dataset from here. You can download directly the aligned images
img_align_celeba.zip. -
Download the FFHQ256 dataset from here.
After downloading the datasets we need to extract the landmarks using mediapipe and FAN. We provide the scripts for preprocessing in datasets/preprocess_scripts. Example usage:
python datasets/preprocess_scripts/apply_mediapipe_to_dataset.py --input_dir PATH_TO_FFHQ256/images --output_dir PATH_TO_FFHQ256/mediapipe_landmarksand for FAN:
python datasets/preprocess_scripts/apply_fan_to_dataset.py --input_dir PATH_TO_FFHQ256/images --output_dir PATH_TO_FFHQ256/fan_landmarksNote that for obtaining the FAN landmarks we use the implementation in https://github.com/hhj1897/face_alignment.
Next, make sure to update the config files in configs with the correct paths to the datasets and their landmarks.
At the pretraining stage, we train all 3 encoders (pose, shape, and expression) using only the extracted landmarks and the output of MICA.
python train.py configs/config_pretrain.yaml train.log_path="logs/pretrain"After pretraining, at the core stage of SMIRK, we freeze the shape and pose encoders and train the expression encoder with the full SMIRK framework (reconstruction path and cycle path).
python train.py configs/config_train.yaml resume=logs/pretrain/first_stage_pretrained_encoder.pt train.loss_weights.emotion_loss=1.0If you find this work useful, please consider citing:
@inproceedings{SMIRK:CVPR:2024,
title = {3D Facial Expressions through Analysis-by-Neural-Synthesis},
author = {Retsinas, George and Filntisis, Panagiotis P., and Danecek, Radek and Abrevaya, Victoria F. and Roussos, Anastasios and Bolkart, Timo and Maragos, Petros},
booktitle = {Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2024}
}We acknowledge the following repositories and papers that were used in this work: