LongCat-Audio-Codec - LongCat open source voice codec solution for Meituan

What is LongCat-Audio-Codec?

LongCat-Audio-Codec is an open source speech codec solution from the LongCat team of Meituan. The program is designed for Speech Large Language Model (Speech LLM), and takes into account the semantic and acoustic features of speech through the semantic and acoustic dual Token parallel extraction mechanism, which solves the problem of the difficulty of balancing the semantic and acoustic information in the traditional program. The low-latency streaming decoder supports real-time interaction to meet the needs of in-vehicle voice assistant, real-time translation and other scenarios. With ultra-low bit rate high fidelity and integrated super resolution design, it realizes high fidelity audio reconstruction at very low bit rate.

Features of LongCat-Audio-Codec

• Efficient Semantic and Acoustic Parallel Processing: through semantic and acoustic dual Token Parallel extraction, taking into account the semantic information and acoustic features of speech, improves the accuracy and naturalness of speech processing.
• Low-latency streaming decodingThe frame-level incremental processing mode is used to control the decoding latency in hundred milliseconds to meet the real-time interaction requirements, such as in-vehicle voice assistant and real-time translation.
• Ultra-low bit rate high fidelity audio reconstruction: High-fidelity audio reconstruction at very low bit rates with an integrated super-resolution design that enhances the sampling rate and naturalness of the audio.
• Flexible codebook configuration: Support for adjusting the number of acoustic codebooks according to downstream tasks, adapting to different application scenarios, such as less-tone or more-tone scenarios.
• Multi-stage training strategy: Optimized by multi-stage training, balancing high compression rate and high sound quality requirements to meet diversified application needs.
• one-stop-shop toolchain: Provide a complete tool chain of Token generators and reducers to lower the development threshold and accelerate the application of speech grand models on the ground.

LongCat-Audio-Codec's Core Advantages

• Parallel Semantic-Acoustic Dual Token Extraction Mechanism: For the first time, the parallel extraction of semantic and acoustic information is realized, taking into account the semantic understanding of speech and the retention of acoustic features, and solving the problem that it is difficult to balance the semantic and acoustic information in the traditional scheme.
• Low Latency Streaming DecoderThe innovative frame-level incremental processing mode controls the decoding delay at the hundred millisecond level, significantly improving the real-time nature of voice interaction to meet the needs of in-vehicle voice assistants, real-time translation and other scenarios.
• Ultra-low bitrate high fidelity with integrated super-resolution design: High-fidelity audio reconstruction at very low bit rates, while super-resolution processing is integrated into the decoder to enhance the sampling rate and naturalness of the output audio, and to enhance the detail of the speech.
• Flexible Acoustic Codebook Configuration: Supports dynamic adjustment of the number of acoustic codebooks according to the downstream task, adapting to different scene requirements, such as less-tone or more-tone scenarios, to provide a more flexible solution.
• Multi-stage training strategy: A multi-stage training strategy is designed to meet the reconstruction demand under high compression rate, high sound quality synthesis demand, and personalized customization demand, respectively, to further optimize the model performance.

What is LongCat-Audio-Codec's official website?

• Github repository:: https://github.com/meituan-longcat/LongCat-Audio-Codec
• Hugging Face Model Library:: https://huggingface.co/meituan-longcat/LongCat-Audio-Codec

Who is LongCat-Audio-Codec for?

• Speech technology developers: Efficient audio processing tools are needed to develop Speech Large Language Models (Speech LLM) and related applications such as intelligent voice assistants, voice translation, etc.
• Artificial intelligence researchers: Focusing on research in the areas of speech recognition, speech synthesis, and speech interaction, advanced audio coding and decoding technologies are needed to support experiments and research.
• Product Team: Teams responsible for developing voice interaction products such as in-vehicle voice assistants, smart speakers, and real-time translation tools need low-latency, high-quality audio processing solutions.
• Audio Engineer: Professionals with needs in audio processing, audio compression and audio quality enhancement require flexible audio codec tools to optimize audio processing processes.
• technology enthusiast: Interested in speech technology and audio processing and want to explore and use the latest audio codecs for project development or personal study.