NitroGen - NVIDIA's open-source gaming AI model in conjunction with Stanford, Caltech, and others

What is NitroGen?

NitroGen is an open-source gaming AI model developed by NVIDIA in conjunction with Stanford University, Caltech and other organizations, capable of playing over 1,000 different types of games. The model is based on the GROOT N1.5 architecture, and by analyzing 40,000 hours of game video data (including handle operation annotations), it realizes the ability to rapidly migrate from zero sample games to new games. The innovation lies in the adoption of Diffusion Transformer technology to generate action commands directly from pixel inputs, which supports the adaptation of all kinds of games through the Gymnasium API. Tests show that NitroGen can still accomplish non-trivial tasks without fine-tuning, and the task success rate is improved to 52% during migration learning, which is expected to be applied to robotics and other embodied intelligence fields in the future. The project has open-sourced the complete code, dataset and pre-trained models.

Features of NitroGen

• Multi-Game Adaptation: The NitroGen is capable of playing more than 1,000 games, covering a wide range of genres including role-playing, platform jumping, battle royale, racing, and virtually everything else, whether it's a 2D or 3D game.
• input and outputThe model directly takes game video frames as input and outputs real joystick operation signals, naturally adapting to all games that support joysticks.
• Post-training capacity: NitroGen supports post-training, so when faced with a new game, you don't need to learn the rules from scratch, and can quickly get up to speed with a small amount of fine-tuning or light adaptation, with the potential for cross-game generalization.
• Architectural Foundations: Its underlying architecture is GROOT N1.5, an architecture originally designed for robotics and adapted for use in gaming.

NitroGen's core strengths

• Wide range of adaptability: It can adapt more than 1,000 different types of games, covering a wide range of game genres such as role-playing, platform jumping, battle royale, racing, etc., and can run effectively in both 2D and 3D games.
• Efficient post-training capabilities: When facing a new game, there is no need to learn the rules from scratch, only a small amount of fine-tuning or light adaptation, can quickly get started, showing a strong cross-game generalization ability.
• Excellent mission success rate: In procedurally generated game worlds as well as new games that have never been touched before, the task success rate improves by 521 TP3T compared to the model trained from scratch, which is an excellent performance.
• open source sharing: The research results, including pre-trained model weights, complete action datasets, and related code have been open-sourced to facilitate further research and development by developers.
• Video frame-based input and outputThis end-to-end processing makes it possible to directly adapt to all games that support joysticks, by directly taking game video frames as input and outputting real joystick operation signals.
• Massive Behavioral Cloning Training: Trained using over 40,000 hours of game master operation videos covering more than 1,000 games, it learns how to play and strategize different types of games by mimicking the operation behavior of human players.

What is the official website of NitroGen

• Project website:: https://nitrogen.minedojo.org
• Github repository:: https://github.com/MineDojo/NitroGen
• HuggingFace Model Library:: https://huggingface.co/nvidia/NitroGen
• paper address:: https://nitrogen.minedojo.org/assets/documents/nitrogen.pdf
• HuggingFace dataset:: https://huggingface.co/datasets/nvidia/NitroGen

Who NitroGen is for

• game developer: You can use NitroGen to quickly test new gameplay and mechanics, optimize game design, and improve development efficiency.
• Game Tester: With NitroGen to simulate player actions, automated tests are conducted to find and fix in-game problems.
• Game content creators: Get inspired to create game guides, tutorials, or entertainment videos with NitroGen-generated gameplay and actions.
• gamer: Especially new players can use NitroGen as an aid to learn gaming skills and get up to speed quickly with complex games.
• AI researchers: NitroGen serves as a platform for studying generalized AI and cross-game generalization capabilities, providing the experimental basis and data support for related research.
• Educators and students: In education, NitroGen can be used to develop educational games or simulated training environments to aid teaching and learning.