DH live: AI digital people for smooth real-time interaction on 30/40-series graphics cards

General Introduction

DH_live is a real-time live digital human project based on sample less learning, aiming to provide users with a smooth and interactive live streaming experience. The project supports NVIDIA 30 and 40 series graphics cards and is capable of running in real-time at 25+ fps. Users can create and use digital people in simple steps, enabling audio-driven video generation and real-time interaction.

Function List

• Real-time performance: Supports NVIDIA 30 and 40 series graphics cards for a smooth real-time interactive experience.
• Few-shot learning: The system is able to learn from a small number of examples to generate realistic responses.
• Video Preparation: Prepare the video data using the data_preparation script.
• Audio Driver: Supports driving digital people through audio files to generate synchronized video.
• Real-time microphone input: Supports real-time operation via microphone.

Using Help

Environment creation and model file decompression

1. Create a virtual environment and activate it:

   conda create -n dh_live python=3.12
   conda activate dh_live
   

2. Install the dependencies:

   pip install torch --index-url https://download.pytorch.org/whl/cu124
   pip install -r requirements.txt
   

3. Unzip the model file:
   • Linux.

     cd checkpoint
     cat render.pth.gz.001 render.pth.gz.002 > render.pth.gz
     gzip -d -c render.pth.gz > render.pth
     

   • Windows: Extract the checkpoint file using 7zip or WinRAR.

Video preparation

1. utilization data_preparation.py Script preparation video:

   python data_preparation.py YOUR_VIDEO_PATH
   

   The results will be stored in the . /video_data Catalog.

Running with audio files

1. Make sure the audio file is in .wav format, with a sample rate of 16kHz, 16-bit mono.
2. Run the demo script:

   python demo.py video_data/test video_data/audio0.wav 1.mp4
   

real time operation

1. Use the microphone for real-time operation:

   python demo_avatar.py
   

common problems

• Failed to unzip the model file: Please make sure that all sub-volume files are complete and properly unzipped.
• Incorrect audio file format: Please use a conforming .wav file.

DH live one-click installer

This content has been hidden by the author, please enter the verification code to view the content

Captcha:

Please pay attention to this site WeChat public number, reply "CAPTCHA, a type of challenge-response test (computing)", get the verification code. Search in WeChat for "Chief AI Sharing Circle"or"Looks-AI"or WeChat scanning the right side of the QR code can be concerned about this site WeChat public number.

The layman's guide to getting started with AI

Help you learn how to utilize AI tools at a low cost and from a zero base.AI, like office software, is an essential skill for everyone. Mastering AI will give you an edge in your job search and half the effort in your future work and studies.

View Details>