MatAnyone: Extract video to specify the target portrait of the open-source tool to generate the target portrait video

General Introduction

MatAnyone is an open source project focusing on video keying, developed by the research team at S-Lab, Nanyang Technological University, Singapore and released on GitHub. It provides users with stable and efficient video processing capabilities through consistent memory propagation techniques, and is particularly good at handling target-specific keying tasks in complex backgrounds. Launched in 2025 by Peiqing Yang and other researchers, the project combines advanced computer vision algorithms for scenarios requiring high-quality video segmentation, such as film and TV post-production, virtual background replacement, etc. MatAnyone's core strength is its memory fusion module, which is capable of preserving fine details of object boundaries while keeping the core region semantically stable. The project has attracted attention in the academic and open source communities, and is released under the NTU S-Lab License 1.0, which allows users to download, use and modify the code for free.

Function List

• Target-specific video keying: Supports user-specified keying of specific objects, suitable for video segmentation of people or other dynamic targets.
• Coherent Memory Dissemination: Ensure coherent keying results between video frames through regionally adaptive memory fusion.
• High-quality boundary processing: Fine preservation of object edge details and improved keying accuracy for professional video editing.
• First frame mask prediction: Predicts the alpha matte of subsequent frames based on the first frame's segmentation mask, without additional auxiliary input.
• Open Source Support: Full code and documentation is provided, allowing for user-defined optimization or secondary development.
• Cross-platform compatibility: Runs on multiple operating systems and is easy for developers to integrate into existing workflows.

Using Help

Installation process

MatAnyone is a GitHub open source project that requires a basic Python programming environment and Git tools. Here are the steps to install MatAnyone:

1. Environmental preparation

• operating system: Windows, Linux or macOS are supported.
• software dependency::
  • Python 3.8 or above.
  • Git (for cloning code from GitHub).
  • Conda (recommended for creating virtual environments).
• hardware requirement: A GPU (e.g. NVIDIA graphics card) is recommended to speed up the inference process; a CPU can also run but at a slower speed.

2. Downloading code

Open a terminal or command line and enter the following command to clone the MatAnyone repository:

git clone https://github.com/pq-yang/MatAnyone.git
cd MatAnyone

This will download the project files to a local directory.

3. Creating virtual environments

Use Conda to create and activate a separate Python environment and avoid dependency conflicts:

conda create -n matanyone python=3.8 -y
conda activate matanyone

4. Installation of dependencies

In the project root directory, run the following command to install the required Python libraries:

pip install -r requirements.txt

Dependencies requirements.txt All libraries needed for the project are included, such as PyTorch, OpenCV, etc. If you encounter network problems, try changing the pip source (e.g., using the pip install -i https://pypi.tuna.tsinghua.edu.cn/simple -r requirements.txt).

5. Downloading pre-trained models

MatAnyone requires pre-trained model files to run. The authors provide a link to download the model on the project page (usually in README.md), which the user has to download manually and place in a specified folder (e.g. models/ (Catalog). Specific Steps:

• Visit the GitHub project page (https://github.com/pq-yang/MatAnyone).
• Find the model download link in the README (which may point to Google Drive or Hugging Face).
• Download and unzip the model file into the MatAnyone/models/ Catalog.

6. Verification of installation

Run the following command to check if the environment is configured successfully:

python test.py

If no errors are reported, the installation is complete and you can start using it.

Main function operation flow

MatAnyone's core feature is target-specified video keying, and here are the main steps on how to use it:

Function 1: Targeted video keying

1. Preparing to input video::
   • Place the video file to be processed (e.g. input_video.mp4) into the project directory under the data/ folder (you can create this folder manually if it does not exist).
   • Ensure that the video format is supported (e.g. MP4, AVI) and that the resolution is moderate (too high may require more computing resources).
2. Generate first frame mask::
   • Use an external tool (e.g. Photoshop or an open source segmentation tool) to generate a segmentation mask of the target object for the first frame of the video (in PNG format, with white for the target area and black for the background).
   • Name the mask file mask_frame1.pngput into data/masks/ Folder.
3. Run the keying command::
   Go to the project directory in the terminal and execute the following command:

   python inference.py --video data/input_video.mp4 --mask data/masks/mask_frame1.png --output output/
   

   • --video Specifies the input video path.
   • ---mask Specifies the first frame mask path.
   • --output Specify the output folder and the result will be saved as a video file with transparent background.
4. View Results::
   • After processing is complete, open the output/ folder, the generated keying video will be stored as a frame sequence or as a full video (depending on the configuration).

Function 2: Consistent Memory Dissemination

• principle: MatAnyone ensures that keying results are coherent in the time dimension by memorizing features from the previous frame and fusing them into the processing of the current frame.
• manipulate: No additional configuration is required, this feature is built into the inference process. As long as the first frame mask is provided, the program will automatically propagate the memory frame by frame.
• Optimization Tips::
  • If there are sudden lighting changes in the video, adjust the parameters in the configuration file (e.g. memory_fusion_rate), for a description of the specific parameters see config.yaml Documentation.
  • Example Adjustment Command:

    python inference.py --video data/input_video.mp4 --mask data/masks/mask_frame1.png --config config.yaml --output output/
    

Function 3: High Quality Boundary Processing

• Enabling method: It is turned on by default and requires no additional action. The program automatically optimizes the edge details.
• verify the effectiveness of sth.: When processing video with complex backgrounds (e.g., a character's hair swinging in the wind), observe whether the borders are natural in the output video.
• enhancement effect: If the results are not satisfactory, try increasing the inference resolution by adding the command --resolution 1080 Parameters:

  python inference.py --video data/input_video.mp4 --mask data/masks/mask_frame1.png --resolution 1080 --output output/
  

Precautions for use

• computing resource: Processing is faster in a GPU environment. If using a CPU, it is recommended to shorten the length of the video (less than 30 seconds) to minimize the waiting time.
• First frame mask quality: The accuracy of the mask directly affects the results of subsequent frames and careful plotting is recommended, especially in the edge regions.
• documentation reference: In case of problems, consult the README.md Or contact the author at peiqingyang99@outlook.com for assistance.
• Community Support: The GitHub Issues page has user feedback and solutions, and it's recommended to check it regularly for updates.

With the above steps, users can quickly get started with MatAnyone and complete the whole process from installation to video keying. Whether for professional editing or research and development, MatAnyone provides stable technical support.