YuE: Transforms lyrics into a base model of a complete song, supporting a wide range of musical styles

General Introduction

YuE is an open source full-song generation base model that focuses on transforming lyrics into full songs. Unlike other models that can only generate short snippets of non-vocal music, YuE is able to generate full songs with lead and backing vocals that can be up to several minutes long. The model addresses the challenges of long contexts, complex musical signals, distorted linguistic content, and lack of parallel data in music generation.YuE employs a variety of techniques, including a semantically-enhanced audio lexer, a bisection technique, lyrics-chained thought generation, and a three-phase training scheme, to ensure that the generated songs have a coherent musical structure, an engaging vocal melody, and an appropriate accompaniment.YuE supports a wide range of musical styles and vocal genres for pop, metal and many other music genres.

Function List

• Lyrics to Songs: Converts input lyrics into a complete song, including lead and backing vocals.
• Various music styles: Supports a wide range of music styles such as pop and metal.
• High-quality generation: Ensure that the songs generated are coherent and of high quality.
• open source project: The code and model are open source and can be freely used and modified by the user.
• GPU optimization: Supports large-scale GPU parallel computing to improve generation efficiency.

Using Help

Installation process

1. environmental preparation::
   • Make sure Python version >= 3.8.
   • Install the dependencies:pip install -r requirements.txtThe
   • Install FlashAttention 2 to save GPU memory:pip install flash-attn --no-build-isolationThe
   • Configure the CUDA environment (CUDA 11.8 as an example):

    export PATH=/usr/local/cuda-11.8/bin:$PATH
   export LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib64:$LD_LIBRARY_PATH
   

2. Download Code::
   • Cloning Project Warehouse:git clone https://github.com/multimodal-art-projection/YuE.git
   • Go to the project catalog:cd YuE

Usage Process

1. Prepare the lyrics.::
   • Save the lyrics as a text file, e.g.lyrics.txtThe
2. Run the generated script::
   • Use the following command to generate a song:

    python generate_song.py --lyrics lyrics.txt --output song.wav
   

3. Configuration Generation Parameters::
   • The details of the generation can be adjusted via command line parameters, such as music style, vocal type, etc: bash
python generate_song.py --lyrics lyrics.txt --output song.wav --style pop --vocal male


Detailed Function Operation

• Lyrics to Songs::
  • Enter the lyrics file and the model will automatically generate a complete song with lead and backing vocals.
  • A wide range of music styles and vocal types are supported, and users can choose different parameters according to their needs.
• Various music styles::
  • It supports various music styles such as pop, metal, etc. Users can select the type of music to be generated through parameters.
• High-quality generation::
  • The models are optimized to ensure that the resulting songs are coherent and of high quality for professional music creation.
• open source project::
  • The code and model are open source and can be freely used and modified by users for scientific research and commercial applications.
• GPU optimization::
  • Supports large-scale GPU parallel computing to improve generation efficiency, suitable for generating songs for a long time.