open-interactive-video-models technique sourced
Open real-time interactive video world models
Open and startup interactive video models (Matrix-Game, Hunyuan-GameCraft, WorldPlay, Yan, Mirage, Odyssey) adapt pretrained video diffusion transformers to keyboard and mouse control and stream frames in real time, adding pose-indexed memory and text edits during play.
mechanism
A pretrained text- or image-to-video diffusion transformer (SkyReels-V2 for Matrix-Game 2.0, HunyuanVideo for GameCraft) gets action modules: keyboard and mouse mapped to camera translation and rotation, injected through cross-attention, token addition, or Plucker camera embeddings. Training uses gameplay recordings from Unreal Engine scenes, GTA V, Minecraft, Temple Run, or 100+ AAA titles. The bidirectional model is then distilled into a causal few-step generator (Self-Forcing with distribution matching, or phased consistency models) that streams chunks with a rolling KV cache. 2026 systems add memory by retrieving past frames whose camera pose overlaps the current view (Matrix-Game 3.0, WorldPlay). Yan, Mirage 2, and Odyssey-2 accept typed text during play to edit the world.
limitations
- Resolution and speed trade against hardware: Matrix-Game 2.0 is 352x640 on one H100, Matrix-Game 3.0's 720p 40 fps needs 9 GPUs, and GameCraft runs at 6.6 fps after distillation.
- Consistency horizon is about a minute even with retrieval memory; earlier open models have no explicit memory at all.
- Training data includes recordings of commercial games (GTA V, 100+ AAA titles), so output provenance is contested.
- Stronger history conditioning improves consistency but reduces responsiveness to new actions (GameCraft ablation).
- Action spaces cover camera movement and basic locomotion; game verbs such as shooting, building, or trading are not modeled.
relations
7 stored edgesContent sharing
Mirage 2 lets players save and share generated worlds; what is shared is a prompt and seed image for a model rather than an asset bundle, so the receiver needs the same hosted model.
Direct avatar control
Walking, turning, and jumping are rendered from a unified camera-action space; GameCraft's action space lacks shooting, throwing, and explosions.
Generative content on demand
Typed text during play adds or restyles objects and scenes (Yan editing, Mirage 2 commands, Odyssey-2 prompts), so content is requested at runtime rather than authored.
Procedural terrain
A world starts from any image, sketch, or photo (Mirage 2, GameCraft); pose-indexed memory in Matrix-Game 3.0 and WorldPlay lets revisited areas stay consistent for about a minute, still far short of a save file.
Rigid-body physics
Contact and motion responses come from video priors; out-of-domain views degrade (Matrix-Game 2.0 reports over-saturation under sustained upward or forward camera motion).
Vehicle dynamics
Matrix-Game 2.0 was fine-tuned on 574 hours of GTA driving; car handling is reproduced from video rather than tire and suspension models, with no guarantee of consistent grip or mass.
backlinks
computed from other recordsclaims
5 with sourcesMatrix-Game 2.0 (Skywork, August 2025) streams 352x640 video at 25 fps on one H100 with about 1.8B parameters trained on about 1,200 hours of Unreal Engine, GTA driving, Minecraft, and Temple Run data, and releases weights and code; it has no explicit memory.
Hunyuan-GameCraft trained on over 1M recordings from more than 100 AAA titles; distillation raised speed from 0.25 to 6.6 fps with under 5 seconds latency per action, and weights released in August 2025 need at least 24 GB of GPU memory (80 GB recommended).
src-li-2025-hunyuan-gamecraft, src-tencent-2025-hunyuan-gamecraft-weights
Yan (Tencent, August 2025) reports 1080p at 50 to 60 fps for its simulation module trained on the game Yuan Meng Star and an editing module that separates interactive mechanics simulation from visual rendering so text can restyle the world during play.
Matrix-Game 3.0 (April 2026) reaches up to 40 fps at 720p with a 5B model on an 8+1 GPU setup and keeps minute-long consistency by retrieving memory frames by camera pose; WorldPlay (Tencent Hunyuan) streams 720p at 24 fps with a reconstituted context memory for long-term geometric consistency.
Mirage 2 (Dynamics Lab, August 2025) lets players turn photos or sketches into browser-playable worlds and change them by typing commands, with saved and shared worlds, but struggles with precise controls and visual stability; Odyssey-2 (October 2025) generates each frame in under 50 ms from text prompts.