VACE: Open source model for video authoring and editing (not open)

General Introduction

VACE is an open source project developed by Alitongyi Visual Intelligence Lab (ali-vilab), focusing on video creation and editing. It is an all-in-one tool that integrates a variety of functions, such as generating videos based on references, editing existing video content, localization modifications, and more. Officially released in March 2025, the project's goal is to enable users to accomplish complex video processing tasks with a single model.Designed with flexibility and efficiency in mind, VACE supports users in freely combining functions, making it suitable for creators, developers, and even ordinary users. Its code is hosted on GitHub, so anyone can download, study or improve it for free. The team behind it relies on Ali's strong technical support and combines the latest achievements in artificial intelligence and video processing to provide users with a powerful yet practical creation platform.

Function List

• Reference Generation Video (R2V): Generate brand-new video content based on a given reference material, such as an image or video clip.
• Video to video editing (V2V): Modify the overall style or content of an existing video, such as adjusting the color tone or replacing the background.
• Masked video editing (MV2V): Localized editing is supported, allowing users to select specific areas of the video for modification with a mask.
• motion control: Adjusts the motion trajectory of objects in a video, such as making a character move to a specified position.
• Main body replacement: Replace a subject (such as a person or object) in a video with something else.
• Screen Expansion: Expand the video frame to generate frames beyond the original boundaries.
• Animation Generation: Convert still images into dynamic animation effects.

Using Help

VACE is an open source project , mainly for users with a certain technical basis . It has no ready-made graphical interface and needs to be run through code. The following is a detailed guide to help you get started quickly.

Installation process

1. Preparing the environment
   • Make sure you have Python 3.9 or above installed on your computer. You can do this at the command line by typing python --version Check.
   • Install the Git tool for downloading code from GitHub, which can be downloaded from git-scm.com for Windows users and is available for Mac users. brew install git Installation.
   • NVIDIA GPU and CUDA support is required as VACE relies on GPU acceleration. If you don't have a GPU, you can try CPU mode, but performance will be degraded.
2. Download Code
   • Open a terminal (CMD or PowerShell for Windows, Terminal for Mac).
   • Enter the following command to clone the VACE repository:

     git clone https://github.com/ali-vilab/VACE.git
     

   • Once the cloning is complete, go to the project folder:

     cd VACE
     

3. Installation of dependencies
   • VACE requires some Python library support. Run the following command to install it:

     pip install -r requirements.txt
     

   • If you get an error, check your network connection or update your pip (using the pip install --upgrade pip).
   • For GPU users, make sure to install PyTorch and the corresponding CUDA version, for example:

     pip install torch torchvision --extra-index-url https://download.pytorch.org/whl/cu118
     

4. Verify Installation
   • Once the installation is complete, run a simple test command to confirm that the environment is OK:

     python -c "import torch; print(torch.cuda.is_available())"
     

   • If the return TrueIf it is FalseIf you do not have a CUDA installation, you may need to check the CUDA installation.

How to use

The operation of VACE is mainly done through the command line, where the user needs to write configuration files and run Python scripts. The following is a detailed flow of the main functions.

Function 1: Reference Generation Video (R2V)

• move
  1. Prepare reference material, such as an image (JPG, PNG) or a video (MP4 format).
  2. Create a configuration file, e.g. config_r2v.yamlThe contents of this section refer to the examples on GitHub (usually found in the configs (folder). The configuration file specifies the input path and output parameters such as resolution, frame rate, etc.
  3. Run the command in the terminal:

     python inference.py --cfg config_r2v.yaml
     

  4. The generated video is saved in the specified output folder (default is output (folder).
• typical example
  • Enter a picture of a little boy with a description in the configuration file: "Little boy running to the right in the sunshine". When run, VACE generates a video of the boy running.

Function 2: Video to Video Editing (V2V)

• move
  1. Prepare a video that needs to be edited, such as input.mp4The
  2. Edit Configuration File config_v2v.yamlIf you want to change the background to grassland, set the target of editing, for example, "Change the background to grassland".
  3. Run command:

     python inference.py --cfg config_v2v.yaml
     

  4. The edited video is automatically saved.
• typical example
  • Input a video of a city street, the configuration file says "Replace background with forest", the result is a character walking in the forest.

Function 3: Masked Video Editing (MV2V)

• move
  1. Prepare the video and the mask file (the mask is a black and white image, the white area indicates the editing part).
  2. In the configuration file config_mv2v.yaml Specify the video path and the mask path in the "Mask" section, and set the editing content, such as "Replace the mask area with a lion".
  3. Run command:

     python inference.py --cfg config_mv2v.yaml
     

  4. View the results and the masked area will be replaced.
• typical example
  • Enter the coffee shop video, mask select the cups on the table, replace with "a kitten", the result is a cat on the table.

Function 4: Motion Control and Subject Replacement

• move
  1. Prepare a video and description of the target maneuver, such as "Make the horse run to the left."
  2. Set the action parameters or replace the body (e.g. "horse replaced by cow") in the configuration file.
  3. Run the command to generate a new video.
• typical example
  • Enter a horseback riding video, set "Horse running to the right, replace with purple-haired rider", and the result is a purple-haired rider running to the right on a horse.

Handling Precautions

• Configuration file format: YAML files are strict, watch indentation and syntax.
• hardware requirement: GPUs with at least 12GB of RAM are recommended, otherwise you may need to adjust the parameters to reduce the memory footprint.
• Debugging Tips: If something goes wrong, check the terminal logs, common problems are path errors or missing dependencies.
• Community Support: You can ask questions on GitHub's Issues page, and the developers and community will help answer them.

With these steps, you can easily use VACE to create and edit videos. Its power lies in its flexibility, which allows you to realize various creative ideas as long as you are familiar with the configuration.