This repository contains the implementation of the following paper:
FreeInit: Bridging Initialization Gap in Video Diffusion Models
Tianxing Wu, Chenyang Si, Yuming Jiang, Ziqi Huang, Ziwei Liu
From MMLab@NTU affiliated with S-Lab, Nanyang Technological University
We propose FreeInit, a concise yet effective method to improve temporal consistency of videos generated by diffusion models. FreeInit requires no additional training and introduces no learnable parameters, and can be easily incorporated into arbitrary video diffusion models at inference time.
- [01/2024] FreeInit has been integrated into Diffusers and ComfyUI-AnimateDiff-Evolved. Big thanks to @a-r-r-o-w, @DN6 and @Kosinkadink!
- [12/2023] We released a demo on Hugging Faceπ€! Feel free to try it out: [demo link].
- [12/2023] Arxiv paper available.
- [12/2023] Code released.
- [12/2023] Project page and video available.
In this repository, we use AnimateDiff as an example to demonstrate how to integrate FreeInit into current text-to-video inference pipelines.
In pipeline_animation.py, we define a class AnimationFreeInitPipeline
inherited from AnimationPipeline
, showing how to modify the original pipeline.
In freeinit_utils.py, we provide frequency filtering code for Noise Reinitialization.
An example inference script is provided at animate_with_freeinit.py.
Please refer to the above scripts as a reference when integrating FreeInit into other video diffusion models.
git clone https://github.com/TianxingWu/FreeInit.git
cd FreeInit
cd examples/AnimateDiff
conda env create -f environment.yaml
conda activate animatediff