MagicTryOn - Video Virtual Try-On Framework from ZJU and Vivo and others

What is MagicTryOn?

MagicTryOn is an advanced video virtual try-on framework launched by the School of Computer Science and Technology of Zhejiang University in collaboration with vivo and other organizations. The framework replaces the traditional U-Net architecture with an innovative Diffusion Transformer (DiT) architecture, combined with a full self-attention mechanism, to achieve spatio-temporal consistency modeling of the video, ensuring that the fitting effect remains smooth during the character movement, and avoiding flickering and shaking of the garment. magicTryOn is based on a coarse-to-fine garment retention strategy, integrating garment markers in the embedding stage, and introducing multiple conditions such as semantic and texture and contour lines in the denoising stage, to effectively retain the details of the garment and improve visualization. MagicTryOn introduces multi-conditions such as semantics, texture and contour lines to effectively preserve garment details and enhance visual quality.MagicTryOn demonstrates performance beyond existing state-of-the-art methods on image and video try-on datasets, and is widely used in online shopping, fashion design, virtual fitting rooms, advertising and marketing, and gaming and entertainment to bring immersive virtual try-on experiences to users.

Key Features of MagicTryOn

• Clothing details retained: Accurately renders the texture, pattern and silhouette of a garment, and maintains the natural feel and clarity of detail of the garment even when the figure is in motion.
• Spatio-temporal coherence modeling: Based on the full self-attention mechanism, it ensures the coherence of the frames in the video, avoids flickering or jittering of garments, and realizes a smooth fitting effect.
• Multi-conditional guidance: Supports guidance with a variety of conditions such as text, image features, garment markers and contour line markers to generate a more realistic and detailed fitting effect and improve the overall visual quality.

MagicTryOn's official website address

• Project website::https://vivocameraresearch.github.io/magictryon/
• GitHub repository::https://github.com/vivoCameraResearch/Magic-TryOn/
• arXiv Technical Paper::https://arxiv.org/pdf/2505.21325

How to use MagicTryOn

• environmental preparation: MagicTryOn is a deep learning-based framework that requires a high-performance GPU (such as NVIDIA's RTX-series or A-series graphics cards) to accelerate computation.
• software environment::
  • Install Python (Python 3.8 or higher recommended).
  • Install a deep learning framework (e.g. PyTorch), making sure the version matches the requirements of MagicTryOn.
  • Install other dependent libraries (e.g. OpenCV, NumPy, Torchvision, etc.), based on pip install -r requirements.txt installation (the requirements.txt file usually contains all dependencies).
• Getting Code and Data::
  • Cloning GitHub Repositories::

git clone https://github.com/vivoCameraResearch/Magic-TryOn.git
cd Magic-TryOn

• • Preparing the dataset::
    • MagicTryOn requires video data and costume data. The dataset is downloaded from the link provided with the project, or use your own dataset.
    • Datasets usually need to be organized in a specific format, for example:

dataset/
├── videos/          # 视频文件
├── garments/        # 服装图像
├── masks/           # 服装掩码（可选，用于分割）
└── annotations/     # 注释文件（如服装标记等）

• Model Reasoning (Trying on)::
  • Loading pre-trained models: If you use the pre-trained model provided by the project, you can load it directly:

from magictryon import MagicTryOnModel
model = MagicTryOnModel.load_from_checkpoint("path/to/pretrained_model.ckpt")

• • Preparing to enter data: The input data typically consists of, video frames (character images), garment images and their masks (used to specify garment areas) and optionally textual descriptions or other conditional information.
  • running inference::

output = model.inference(video_frames, garment_image, mask, text_description)

• • output is the result of the generated virtual fitting, usually a video or image sequence.
• Visualization of results: Save the generated fitting results as a video or image sequence for visualization based on OpenCV or other tools:

import cv2
for frame in output:
    cv2.imshow("Virtual TryOn", frame)
    cv2.waitKey(30)
cv2.destroyAllWindows()

MagicTryOn's Core Benefits

• Excellent presentation of garment details: Accurately simulate the texture, pattern and silhouette of a garment to maintain its realism and stability as the character moves.
• Strong temporal consistency: Jointly modeling the spatio-temporal coherence of the video based on a fully self-attentive mechanism to ensure coherence between frames in the video, avoiding costume flickering, jittering or unnatural transitions.
• Flexible multi-conditional guidance: Supports guidance based on a variety of conditions such as text, image features, garment markers, and contour line markers to generate more realistic and detailed fitting effects.
• Outperforms existing methods: Demonstrate performance beyond existing state-of-the-art methods on both image and video try-on datasets, both in terms of evaluation metrics, visual quality, and generalizability to field scenes.
• Wide range of application scenariosIt can be used for online shopping and virtual fitting rooms, as well as fashion design, advertising and marketing, gaming and entertainment, etc. It provides efficient solutions for different industries.
• Open Source and Ease of Use: Open source code and detailed documentation are provided for developers and researchers to get up and running quickly.

Who is MagicTryOn for?

• Online shopping platforms and e-commerce companies: Helps users visualize how garments look on the body, enhancing the shopping experience and reducing return rates.
• Fashion designers and clothing brandsMagicTryOn: Quickly preview garment designs with MagicTryOn, speeding up the design process and reducing prototyping costs.
• Brick and mortar stores and retailers: Reduce the use of physical fitting rooms and improve store operational efficiency by providing virtual fitting services.
• Advertising and marketing staff: Create personalized try-on ads to capture consumer attention and enhance brand impact.
• Gaming and entertainment industry: Enhance player and audience immersion by trying on virtual costumes in real time in gaming and entertainment scenarios.