Vidi2 - ByteHop's open source multimodal video understanding and generation of large models

What is Vidi2?

Vidi2 is a second-generation multimodal video understanding and generation big model open-sourced by ByteDance, focusing on video content understanding, analysis and creation. It supports the joint input of text, video, and audio modalities, and can simultaneously understand picture content, sound information, and natural language commands to achieve cross-modal interaction and reasoning. Accurately locate the time range and spatial position of a specific event or target object in a video, and the model can automatically label the corresponding time period and target area in the screen, with an error that can be accurate to the millisecond level. It can process hours of raw video footage and quickly retrieve clips that match specific semantics.

Features of Vidi2

• Multimodal processing capability: It supports the joint input of text, video, and audio modalities, and is able to understand screen content, sound information, and natural language commands simultaneously, realizing cross-modal interaction and reasoning.
• Fine spatio-temporal localization (STG)The model can accurately locate the time range and spatial location of a specific event or target object in the video, for example, according to the text description "someone did a certain action in the Xth minute", the model can automatically mark the corresponding time period and target area in the screen, and the error can be accurate to the millisecond level.
• Long video comprehension and retrieval: Can process hours-long raw video footage, quickly retrieve segments that match specific semantics, and maintain high accuracy even when the video content is complex and the scene switches frequently. In ultra-long video (>1 hour) scenarios, the performance is ahead of mainstream commercial models.
• Video Quiz & Reasoning: Supports open-ended Q&A based on video content, which can answer questions about the plot, character relationships, cause and effect of events, and generate reasonable answers through multiple rounds of reasoning to help users quickly access key information in the video.
• Intelligent editing and creative assistanceIt can automatically extract highlight clips and generate short video titles, as well as perform intelligent composition cropping and automatic multi-camera switching according to users' needs, which significantly reduces the threshold of video creation and improves the efficiency of creation.

Core Benefits of Vidi2

• Fine spatial and temporal positioning capability: Vidi2 recognizes both the timestamp and the bounding box of the target object in the video, and given a text query, it can not only find the corresponding time period, but also accurately mark the location of specific objects within these time frames, realize tracking of specified objects and characters with one-second granularity, and support the tasks of tracking a specific character in a crowd or separating props in discontinuous shots.
• Powerful video comprehension and generation capabilities: Vidi2 can process hours of raw footage, understand the story line within it, and generate full TikTok short videos or movie clips based on simple prompts.
• Advanced Technology Architecture: The use of Gemma-3 as the backbone network, combined with re-engineered adaptive markup compression, ensures that efficiency is maintained while processing long videos without losing critical details. In addition, a cross-modal processing flow is unified by jointly processing text, visuals and audio to understand and create videos.
• Excellent performance: On the VUE-TR-V2 benchmark for open-ended temporal retrieval, Vidi2 achieves an overall IoU of 48.75, and in particular outperforms the commercial model by 17.5 percentage points on ultra-long videos (more than 1 hour). On the localization task (VUE-STG), Vidi2 achieves the best performance of 32.57 for vIoU and 53.19 for tIoU.
• Efficient Data Training Strategies: The training process of Vidi2 emphasizes real, diverse video data, combined with synthetic localization data and carefully curated annotations to align spatial and temporal reasoning on a large scale. In addition, a Temporal-aware Multimodal Alignment (TAMA) strategy is employed to enhance the performance of the model through a staged, bi-directional reinforcement training mechanism.

What is the official website for Vidi2

• Project website:: https://bytedance.github.io/vidi-website/
• Github repository:: https://github.com/bytedance/vidi
• arXiv Technical Paper:: https://arxiv.org/pdf/2511.19529

People for whom Vidi2 is intended

• Video Creators: Vidi2 can help video creators quickly generate video scripts, outlines and titles, and can automatically edit long videos into short videos suitable for platform publishing, greatly improving creative efficiency.
• Content Editorial Team: For editing teams that have to deal with a large amount of video footage, Vidi2 can automatically identify and extract key clips in the video to generate highlight moments, saving time on manual screening and editing.
• Social Media Operators: Vidi2 can quickly convert long video content into short videos suitable for social media platforms, helping operators to publish content more efficiently and improve its dissemination.
• moviemakerVidi2 can assist with plot comprehension, camera editing and subtitling in post-production to enhance production efficiency.
• Advertising & Marketing Team: Vidi2 can quickly generate engaging video content to help advertising teams create more attractive advertising videos and improve advertising effectiveness.
• educator: Educators can use Vidi2 to optimize and process teaching videos to generate short video clips suitable for teaching and improve the utilization efficiency of teaching resources.