StableAnimator: generates high quality video animation that maintains the character's features.

General Introduction

StableAnimator is an innovative end-to-end identity-preserving video diffusion framework capable of synthesizing high-quality videos based on a reference image and a series of poses without any post-processing. Developed in collaboration with researchers from Fudan University, Microsoft Research Asia, Huya Inc and Carnegie Mellon University, StableAnimator ensures video identity consistency in video. The framework performs well in multiple benchmarks and generates high-fidelity, identity-preserving portrait animations.

Function List

• Identity Preservation Video Generation: Generate high-quality videos based on reference images and pose sequences to ensure identity consistency.
• No post-processing: The resulting video does not require post-processing with any face swapping tools or face restoration models.
• Global Content-Aware Facial Encoder: Further optimizing facial embedding by interacting with image embedding.
• Distribution-Aware Identity Adapter: Prevent time layer interference while maintaining identity consistency.
• Hamilton-Jacobi-Bellman optimization: Facial quality is further enhanced by solving the HJB equation during the inference process.

Using Help

Installation process

1. clone warehouse: Clone the StableAnimator repository by running the following command in a terminal:

   git clone https://github.com/Francis-Rings/StableAnimator.git

2. Installation of dependencies: Go to the repository directory and install the required dependencies:

   cd StableAnimator
pip install -r requirements.txt

3. Running the application: Use the following command to start the application:

   python app.py

Usage Process

1. Preparing reference images and pose sequences: Make sure you have a reference image and a series of pose images that will be used to generate the animation.
2. Running inference scripts: Use the following command to run the inference script to generate the video animation:

   python inference_basic.py --reference_image path_to_reference_image --pose_sequence path_to_pose_sequence

3. View Generated Results: The generated video will be saved in the specified output directory and you can view the result using any video player.

Detailed Function Operation

• Identity Preservation Video Generation: By providing a reference image and a sequence of poses, StableAnimator is able to generate high-quality video animations, ensuring that the identity of the people in the video is consistent with the reference image.
• Global Content-Aware Facial Encoder: This module further optimizes the facial embedding by interacting with the image embedding to ensure high fidelity of facial details in the generated video.
• Distribution-Aware Identity Adapter: During the generation of the video, the module ensures identity consistency in the video through alignment techniques that prevent time layer interference.
• Hamilton-Jacobi-Bellman optimization: The quality and consistency of faces in the generated video is further enhanced by solving the HJB equation during the inference process.

The layman's guide to getting started with AI

Help you learn how to utilize AI tools at a low cost and from a zero base.AI, like office software, is an essential skill for everyone. Mastering AI will give you an edge in your job search and half the effort in your future work and studies.

View Details>