LatentSync: Enabling Audio-Driven Precise Lip Synchronization for AI Mouth Swap Video Generation

General Introduction

LatentSync is an innovative audio conditional potential diffusion modeling framework open-sourced by ByteDance, specifically designed to enable high-quality video lip-synchronization. Unlike traditional methods, LatentSync uses an end-to-end approach to directly generate natural, smooth lip-synchronization effects without intermediate action representations. The project uses the Whisper model to convert speech into audio embeddings, which are integrated into U-Net through a cross-attention layer to enable accurate generation of video frames. The system not only supports real-life video processing, but also handles lip synchronization of anime characters, which has a wide range of application prospects. The project is fully open source, providing inference code, data processing flow and training code, enabling researchers and developers to easily reproduce and improve this technology. Finally, there is a solution other than Wav2Lip An alternative to the

Experience: https://huggingface.co/spaces/fffiloni/LatentSync

API demo address: https://fal.ai/models/fal-ai/latentsync

Function List

• End-to-end audio driven lip sync generation
• Support lip synchronization for live action video and anime characters
• Automatic audio and video alignment and synchronization correction
• High quality face detection and alignment
• Automatic scene segmentation and video segmentation processing
• Video quality assessment and filtering
• Provide a complete data processing pipeline
• Support for custom model training

Using Help

Environment Configuration

1. System Requirements:
   • GPU Memory Requirement: at least 6.5GB
   • NVIDIA graphics cards with CUDA support
   • Python environment
2. Installation Steps:

source setup_env.sh

After a successful installation, the checkpoint file structure should look like the following:

. /checkpoints/
|-- latentsync_unet.pt # Main model file
|-- latentsync_syncnet.pt # Synchronization Network Model
|-- whisper
| `-- tiny.pt # Speech Processing Models
|-- auxiliary/ # auxiliary model directory

Usage Process

1. Basic Reasoning Usage:
   • (of a computer) run . /inference.sh draw basic inferences
   • This can be done by adjusting the guidance_scale parameter to 1.5 to improve lip synchronization accuracy
2. Data Processing Flow:
   • Video Preprocessing:
     • Automatically corrects video frame rate to 25fps
     • Audio resampling to 16000Hz
     • Automatic scene detection and segmentation
     • Split video into 5-10 second segments
   • Face Processing:
     • Detecting and filtering face size (>256×256 required)
     • Remove multi-face scenes
     • Affine transformation based on facial feature points
     • Uniform resizing to 256 x 256
   • Quality control:
     • Synchronized confidence score screening (threshold of 3)
     • Automatic adjustment of audio and video offsets
     • Image quality assessment using hyperIQA
3. Advanced Features:
   • Model Training:
     • U-Net training: using . /train_unet.sh
     • SyncNet training: using . /train_syncnet.sh
   • Parameters in the configuration file can be adjusted as needed, such as data directory, checkpoint save path, etc.

caveat

• Ensure sufficient video memory when reasoning (at least 6.5GB)
• Make sure the video is in good quality before processing
• It is recommended to do small scale testing before processing large amounts of video
• A complete data processing flow needs to be completed before training a custom model
• Please comply with the relevant license requirements