Matrix-Game: Interactive World Foundation Model
📝 Overview
Matrix-Game is a 17B-parameter interactive world foundation model for controllable game world generation.
✨ Key Features
- 🎯 Feature 1: Interactive Generation. A diffusion-based image-to-world model that generates high-quality videos conditioned on keyboard and mouse inputs, enabling fine-grained control and dynamic scene evolution.
- 🚀 Feature 2: GameWorld Score. A comprehensive benchmark for evaluating Minecraft world models across four key dimensions, including visual quality, temporal quality, action controllability, and physical rule understanding.
- 💡 Feature 3: Matrix-Game Dataset A large-scale Minecraft dataset with fine-grained action annotations, supporting scalable training for interactive and physically grounded world modeling.
🔥 Latest Updates
- [2025-05] 🎉 Initial release of Matrix-Game Model
🚀 Performance Comparison
GameWorld Score Benchmark Comparison
| Model | Image Quality ↑ | Aesthetic Quality ↑ | Temporal Cons. ↑ | Motion Smooth. ↑ | Keyboard Acc. ↑ | Mouse Acc. ↑ | Object Cons. | Scenario Cons. |
|---|---|---|---|---|---|---|---|---|
| Oasis | 0.65 | 0.48 | 0.94 | 0.98 | 0.77 | 0.56 | 0.56 | 0.86 |
| MineWorld | 0.69 | 0.47 | 0.95 | 0.98 | 0.86 | 0.64 | 0.51 | 0.92 |
| Ours | 0.72 | 0.49 | 0.97 | 0.98 | 0.95 | 0.95 | 0.76 | 0.93 |
Metric Descriptions:
-
Image Quality / Aesthetic: Visual fidelity and perceptual appeal of generated frames
-
Temporal Consistency / Motion Smoothness: Temporal coherence and smoothness between frames
-
Keyboard Accuracy / Mouse Accuracy: Accuracy in following user control signals
-
Object Consistency: Geometric stability and consistency of objects over time
-
Scenario Consistency: Scenario consistency over time
Please check our GameWorld benchmark for detailed implementation.
Human Evaluation
Double-blind human evaluation by two independent groups across four key dimensions: Overall Quality, Controllability, Visual Quality, and Temporal Consistency.
Scores represent the percentage of pairwise comparisons in which each method was preferred. Matrix-Game consistently outperforms prior models across all metrics and both groups.
🚀 Quick Start
# clone the repository:
git clone https://github.com/SkyworkAI/Matrix-Game.git
cd Matrix-Game
# install dependencies:
pip install -r requirements.txt
# install apex and FlashAttention-3
# Our project also depends on [apex](https://github.com/NVIDIA/apex) and [FlashAttention-3](https://github.com/Dao-AILab/flash-attention)
# inference
bash run_inference.sh
⭐ Acknowledgements
We would like to express our gratitude to:
- Diffusers for their excellent diffusion model framework
- HunyuanVideo for their strong base model
- MineDojo for their Minecraft video dataset
- MineRL for their excellent gym framework
- Video-Pre-Training for their accurate Inverse Dynamics Model
- GameFactory for their idea of action control module
We are grateful to the broader research community for their open exploration and contributions to the field of interactive world generation.
📎 Citation
If you find this project useful, please cite our paper:
@article{zhang2025matrixgame,
title = {Matrix-Game: Interactive World Foundation Model},
author = {Yifan Zhang and Chunli Peng and Boyang Wang and Puyi Wang and Qingcheng Zhu and Zedong Gao and Eric Li and Yang Liu and Yahui Zhou},
journal = {arXiv preprint arXiv:2506.18701},
year = {2025}
}