AudioGen-Omni - Multimodal Audio Generation Model from Racer

What is AudioGen-Omni?

AudioGen-Omni is a multimodal audio generation model launched by Racer, which can generate high-quality audio, voice and songs based on video, text and other inputs.AudioGen-Omni is based on advanced technologies, such as Multimodal Diffusion Transformer and Phase-Aligned Anisotropic Position Injection, to achieve accurate audio-visual alignment and cross-modal synchronization. The model supports multi-language inputs and has an outstanding inference speed of 1.91 seconds to generate 8 seconds of audio.AudioGen-Omni is suitable for a variety of scenarios such as video dubbing, speech synthesis, and song creation, which can significantly improve the efficiency of creation and the richness of content.

Key Features of AudioGen-Omni

• Multimodal Audio Generation: It can generate high-quality audio, voice and songs based on video, text or a combination of both to meet diverse content creation needs.
• Precision audio-visual alignment: Based on phase-aligned anisotropic position injection technology, it ensures that audio and video are highly matched in terms of lip sync and rhythmic alignment, enhancing the audiovisual experience.
• Multi-language support: Supports multiple language inputs to generate speech and songs in corresponding languages, adapting to the creative needs of different language environments.
• Efficient Reasoning: Fast inference, 1.91 seconds can generate 8 seconds of audio, significantly better than similar models, suitable for efficient creation scenarios.
• Flexible input conditions: Generate stable audio output even with video-only or text-only inputs, adapting to different creative conditions.
• High quality audio generation: The generated audio is highly matched to the input in terms of semantic and acoustic performance, and supports high-fidelity audio generation to ensure excellent sound quality.

AudioGen-Omni's project address

• Project website:: https://ciyou2.github.io/AudioGen-Omni/
• arXiv Technical Paper:: https://ciyou2.github.io/AudioGen-Omni/

Core Advantages of AudioGen-Omni

• Efficient generation speed: AudioGen-Omni's inference is extremely fast, taking only 1.91 seconds to generate 8 seconds of audio, significantly outperforming similar models, and dramatically improving the efficiency of authoring for scenarios that require fast audio generation.
• Powerful multimodal processing: The model can handle multiple input modalities, including video, text, or a combination of both. The ability to generate high-quality audio when some modalities are missing (e.g., video only or text only) shows great adaptability.
• Precise audio-visual alignmentBased on Phase Aligned Anisotropic Position Injection (PAAPI) technology, AudioGen-Omni enables precise lip-synchronization and tempo alignment between audio and video, ensuring a high degree of consistency in audio-visual content and greatly enhancing the user experience.
• Multi-language support: AudioGen-Omni supports multi-language input and can generate speech and songs in corresponding languages, adapting to the needs of creation in different language environments, with a wide range of internationalization application potential.
• High quality audio output: The generated audio is highly matched to the input in terms of semantic and acoustic performance, and supports high-fidelity audio generation to ensure excellent sound quality and meet the needs of professional creation.
• Flexible application scenariosIt is suitable for a variety of scenarios, including video dubbing, speech synthesis, song creation and sound effect generation, etc. It can provide powerful technical support for creators in different fields.

Who is AudioGen-Omni for?

• Video Creators: Used by self-publishers, short video creators and film production teams to quickly generate video voiceovers, background music or sound effects to enhance creative efficiency and content appeal.
• music producer: Helps indie musicians and music studios generate backing tracks or full songs based on lyrics or video content to aid in music creation.
• Language Service Providers: Generate multilingual speech content for translation companies and speech synthesis service providers for use in audiobooks, voice navigation, and other services.
• educator: Help online education platforms and educational content creators to generate accurate voiceovers for instructional videos, enhancing the attractiveness and comprehensibility of educational content.
• Companies & Brands: Apply to brand marketing team and customer service team to generate brand promotion voiceover, background music or intelligent customer service voice content to enhance brand appeal and user experience.