AI Personal Learning
and practical guidance
Beanbag Marscode1

AnimeGamer: An Open Source Tool for Generating Anime Videos and Character Interactions with Language Commands

General Introduction

AnimeGamer is an open source tool from Tencent's ARC Lab. Users can generate anime videos with simple language commands, such as "Sousuke drive around in a purple car," and have different anime characters interact with each other, such as Kiki from Magical Girl's Adventure and Pazu from Castle in the Sky. It's based on the Multimodal Large Language Model (MLLM), which automatically creates coherent animated segments while updating the character's status, such as stamina or social values. The project code and model are free and open on GitHub for anime fans and developers to use for creation or experimentation.

AnimeGamer: An Open Source Tool for Generating Anime Videos and Character Interactions with Language Commands-1


 

Function List

  • Generate animation video: Input language commands to automatically generate animation clips of character movements and scenes.
  • Character Interaction Support: Let different anime characters meet and interact to create new stories.
  • Updated character status: real-time record of changes in character values such as stamina, socialization and entertainment.
  • Keep content coherent: ensure consistent video and status based on historical instructions.
  • Open source extensions: complete code and models are provided and developers are free to adapt them.

 

Using Help

AnimeGamer requires a bit of programming basics, but the installation and usage steps are not difficult. Here are detailed instructions to help you get started quickly.

Installation process

  1. Preparing the environment
    You'll need a Python-enabled computer, preferably with a GPU (at least 24GB of video memory). Install Git and Anaconda first, then type in the terminal:
git clone https://github.com/TencentARC/AnimeGamer.git
cd AnimeGamer

Create a virtual environment:

conda create -n animegamer python=3.10 -y
conda activate animegamer
  1. Installation of dependencies
    Runs in a virtual environment:
pip install -r requirements.txt

This will install the necessary libraries such as PyTorch.

  1. Download model
    Download the three model files to ./checkpoints Folder:
  • AnimeGamer model:Hugging FaceThe
  • Mistral-7B model:Hugging FaceThe
  • CogvideoX's 3D-VAE model: go to checkpoints folder, run:
    cd checkpoints
    wget https://cloud.tsinghua.edu.cn/f/fdba7608a49c463ba754/?dl=1 -O vae.zip
    unzip vae.zip
    

Make sure the models are all in the right place.

  1. test installation
    Return to the home directory and run:
python inference_MLLM.py

No error means the installation was successful.

How to use the main features

At its core, AnimeGamer generates videos and character interactions using verbal commands. Here's how it works:

Generate anime videos

  • move
  1. compiler ./game_demo command file in a folder such as instructions.txtThe
  2. Enter a command, e.g. "Sousuke is driving around in a purple car in the forest".
  3. Run MLLM to generate a representation:
python inference_MLLM.py --instruction "宗介在森林里开紫色车兜风"
  1. Decode to video:
python inference_Decoder.py
  1. The video will be saved in the ./outputs Folder.
  • take note of
    Instructions should be written with clear characters, actions and scenes so that the video is more in line with expectations.

Character Interaction

  • move
  1. Enter an interactive command, such as, "Kiki teach Pazuzu to fly a broom."
  2. Run through the steps above to generate an interactive video.
  • specificities
    Supports mixing and interacting with different anime characters to create unique scenes.

Update Character Status

  • move
  1. Add a state description to the command, e.g. "Sousuke is tired after running".
  2. (of a computer) run inference_MLLM.pyThe status will be updated to ./outputs/state.jsonThe
  • draw attention to sth.
    The status is automatically adjusted according to historical instructions to maintain consistency.

Customization and Technical Details

Want to change a feature? You can edit it directly ./game_demo AnimeGamer's technique is a three-step process:

  1. Processing the action representation with an encoder, the diffusion decoder generates the video.
  2. MLLM predicts the next state based on historical instructions.
  3. Optimize the decoder to improve video quality.
    More details are in GitHub's README.md.

latest developments

  • April 2, 2025: release of model weights and papers for The Witch's House and Goldfish Girl on the Cliff (arXiv).
  • April 1, 2025: inference code released.
  • Future plans: launch Gradio interactive demos and training code.

Frequently Asked Questions

  • Slow generation? Verify that the GPU has enough memory (24GB), or update the drivers.
  • Model download failed? Download manually from Hugging Face.
  • Report an error? Check Python version (3.10 required) and dependencies.

With these steps, you'll be able to generate anime videos and character interactions with AnimeGamer.

 

application scenario

  1. anime and manga creation
    Anime fans can use it to generate videos, such as having different characters interact and share them with friends.
  2. Gaming Tests
    Developers can use it to quickly prototype dynamic content and test ideas.
  3. Learning Practice
    Students can use it to learn multimodal technology and video generation for hands-on AI experience.

 

QA

  1. Programming knowledge required?
    Yes, basic Python knowledge is required for installation and tuning, but simple commands will work.
  2. What roles are supported?
    Support for Magical Girl's Home Companion and Goldfish Hime on the Cliff now, with expansion in the future.
  3. Is it commercially available?
    Yes, but follow the Apache-2.0 protocol, see GitHub for details.
May not be reproduced without permission:Chief AI Sharing Circle " AnimeGamer: An Open Source Tool for Generating Anime Videos and Character Interactions with Language Commands
en_USEnglish