TANGO: A tool for generating coordinated gesture videos based on audio, dictation

General Introduction

TANGO is an open source co-lingual gesture video generation tool designed to accurately synchronize audio with gestures in video. Developed by the University of Tokyo and CyberAgent AI Lab, the tool utilizes advanced AI techniques to enable high-quality gesture generation in a variety of environments.Suitable for research, academic, and personal use, TANGO provides complete training code, inference code, and pre-trained models.

Work with voice lip synchronization tools for a complete digital human project:Sync ,Wav2Lip

Function List

• Highly Accurate Gesture Synchronization : Accurately synchronize any audio with the gestures in the video.
• Multi-language support: Works with a variety of languages and sounds, including CGI faces and synthesized sounds.
• Open source and free : The code is completely public, and users are free to use and modify it.
• Interactive Demo: Provides an online demo where users can upload video and audio files to experience.
• Pre-training models: Provide a variety of pre-training models, users can directly use or secondary training.
• Complete training code: Includes training code for gesture synchronization discriminator and TANGO model.

Using Help

Installation process

1. Cloning Warehouse :
   bash copy

git clonehttps://github.com/CyberAgentAILab/TANGO

1. Install dependencies :
   bash copy

pip install -r requirements.txt

1. Download pre-trained model: Download the pre-trained model to a specified directory, e.g. face_detection/detection/sfd/s3fd.pthThe
2. Run the inference code :
   bash copy

python inference.py --checkpoint_path <ckpt> --face <video.mp4> --audio <an-audio-source>

Usage Process

1. To access the local server: Open the http://localhost:3000The
2. Upload Video and Audio : Upload the audio and video files you want to synchronize in the input box.
3. Perform gesture synchronization : Tap the "Synchronize" button, the system will automatically perform the gesture synchronization process.
4. Viewing and Downloading Results : After synchronization is complete, you can preview the results and download the synchronized video files.
5. Use Interactive Demo : Upload video and audio files on the Demo page to experience the gesture synchronization effect in real time.
6. Manage Projects : View and manage all uploaded projects on the My Projects page, supporting version control and collaboration.

Advanced Features

• Smart Gesture Synchronization : Improve the presentation of your video content with smart gesture synchronization provided by AI.
• Multi-language support : Select different languages and voices according to your project needs.
• Customized development: Since TANGO is open source, users can develop it according to their needs.

The layman's guide to getting started with AI

Help you learn how to utilize AI tools at a low cost and from a zero base.AI, like office software, is an essential skill for everyone. Mastering AI will give you an edge in your job search and half the effort in your future work and studies.

View Details>