Fay Digital Human Framework: Integrated language modeling and 3D digital characters to support multiple application scenarios

General Introduction

Fay is an open source 3D virtual digital human framework, integrated with language models and digital characters, suitable for a variety of application scenarios, such as virtual shopping guides, virtual anchors, assistants, waiters, teachers, and voice- or text-based mobile assistants.Fay framework supports full offline use, providing milliseconds to reply, and can be freely matched with a variety of digital human models, large language models, ASR (Automatic Speech Recognition) and TTS (Text-to-Speech) models. Fay is dedicated to providing comprehensive digital human application interfaces for microcontrollers, applications, websites, etc. It supports multi-user and multi-channel concurrency, and provides text and voice communication interfaces, management and control interfaces, and auto-play interfaces, etc. Fay's framework supports full offline use and provides millisecond responses.

Function List

• Completely open source and commercially available
• Supports full offline use
• millisecond response
• Free-matching digitizer models, large language models, ASR, TTS models
• Support virtual teacher, virtual anchor, news broadcast and other automatic broadcast mode
• Support microcontroller, application, website, big screen, mature system access
• Multi-user concurrent support
• Provide text communication interface, voice communication interface, digital human modeling interface, management control interface, automatic playback interface
• Supports flexible configuration of voice command execution
• Customized knowledge base, customized Q&A pairs, customized persona information
• Support for waking up and interrupting conversations
• Supports server and stand-alone modes
• Support for robot expression output
• Support for react agent autonomous decision execution
• Support for active dialog
• Supports silent background startup

Using Help

Installation process

1. Ensure that the system environment is Python 3.9, 3.10, 3.11, or 3.12, with support for Windows, macOS, and Linux.
2. Clone Fay Warehouse:

   git clone https://github.com/xszyou/Fay.git

3. Go to the Fay directory and install the dependencies:

   cd Fay
pip install -r requirements.txt

4. Modify as necessary. /system.confDocumentation.
5. Start the Fay controller:

   python main.py

or boot with Docker:

   docker build -t fay .
docker run -it --rm -p 5000:5000 -p 10001:10001 -p 10002:10002 -p 10003:10003 fay

Guidelines for use

Basic Functions

• text communication interface: Send a text message through the API to get a digital person to reply.
• VCI: Send a voice message through the API to get a digital human voice response.
• Management Control Interface: Manage and configure digital human behavior and response.
• AutoPlay interface: Set up autoplay modes such as virtual teacher, virtual anchor, etc.

Featured Functions

• Customized Knowledge Base: Users can create and manage their own knowledge base, enhancing the professionalism and personalization of digital people.
• Customized Q&A pairs: Configure specific questions and answers to improve the responsiveness of digital people to specific domains.
• Customized persona information: Set the role and background information of the digital person according to the application scenario.
• Voice Command Configuration: Flexible configuration of voice commands for complex interactions.
• Wake up and interrupt conversations: Support for waking up or interrupting a digital human conversation with specific commands.
• Robot Expression Output: The digital person can output corresponding expressions according to the content of the dialog to enhance the interactive experience.
• react agent autonomous decision making: Digital people can make decisions and perform tasks autonomously based on preset rules.
• active dialog: Digital people can initiate conversations to enhance the user interaction experience.
• Silent start in the background: Supports silent startup in the background to minimize interference with foreground operations.

Advanced Play

• Integration into your own products: Fay provides a rich set of interfaces that allow users to integrate it into their products, such as microcontrollers, applications, websites, and more.
• Use of digital people: Fay supports a wide range of digital human models, so users can choose the right model for their application according to their needs.
• virtual anchor (TV): Configure virtual anchor mode for automated news broadcasting, merchandising and other functions.
• Remote voice assistant: Remotely control devices through voice commands to realize smart home and other application scenarios.