diffusion-game-engines technique sourced
Diffusion and autoregressive neural game engines
Action-conditioned generative models trained on recordings of one existing game predict each next frame from recent frames and player input, so the model itself plays the role of the game engine.
mechanism
A game is recorded as pairs of frames and player actions (from an RL agent in GameNGen, human play in DIAMOND's CS:GO model, or gameplay video in Oasis). A generator is trained to predict frame t+1 from the last N frames and the action at t: a latent diffusion model in GameNGen, DIAMOND, and Oasis, or a transformer over interleaved image and action tokens in MineWorld. Context frames are noised during training (noise augmentation, Diffusion Forcing, dynamic noising) so the model can recover from its own errors during long rollouts, and sampling is cut to a few denoising steps for real-time play. There is no explicit game state: position, health, enemies, and map layout exist only as pixels or latents inside the context window, so anything that leaves the window is forgotten unless it is visible on screen.
limitations
- Memory is a few seconds: GameNGen's 64-frame context covers about 3 seconds, and Oasis shows about 3 seconds of memory in one example, so off-screen state such as map layout or enemy positions is lost.
- Requires an existing game to record: GameNGen needed about 70M agent frames from DOOM, DIAMOND 87 hours of CS:GO play; the approach clones a game rather than authoring a new one.
- Each player needs a dedicated accelerator (one TPU-v5 for GameNGen, H100s for Oasis); Maginative reports cost remains a significant barrier to deploying such models at scale, and no per-player-hour figure is published.
- State such as health, ammo, inventory, and score is only as reliable as the pixels that show it; Oasis inventory control is imprecise and distant geometry is fuzzy.
- Training on footage of commercial games raises unresolved licensing questions.
relations
5 stored edgesDirect avatar control
Avatar movement, camera response, and weapon use are predicted frame by frame from keyboard and mouse input instead of computed by engine code; responsiveness depends on denoising steps and frame rate (20 fps GameNGen and Oasis, about 10 fps DIAMOND CS:GO).
Generative content on demand
Every frame, enemy, and pickup is generated content conditioned on play, so a game can in principle contain situations its source game never had; in practice output stays inside the training game's distribution.
Procedural terrain
Terrain is invented on demand from the learned distribution of the training game with no seed or noise function (Oasis), but it is not persistent: regions that leave the short context window are regenerated differently on return.
Rigid-body physics
Collision, gravity, and jumping are learned statistically from pixels rather than integrated; rules the data rarely violates can still be broken, as with DIAMOND's repeated mid-air jumps in CS:GO.
Voxel destruction
In Oasis and MineWorld, breaking and placing Minecraft blocks is predicted as pixels with no stored voxel grid, so edits are not durable; Oasis placing a dirt block can bring up an entirely different environment.
backlinks
computed from other recordsclaims
5 with sourcesGameNGen simulates DOOM at 20 frames per second on a single TPU, with next-frame PSNR of 29.43; human raters picked the real game over the simulation only 58 percent (1.6 s clips) and 60 percent (3.2 s clips) of the time.
GameNGen conditions on 64 past frames (about 3 seconds) and was trained on about 70M frames recorded from a PPO agent trained for 50M environment steps; the authors state the model cannot be used to create new games.
DIAMOND reached a mean human-normalized score of 1.46 on Atari 100k for agents trained entirely inside a diffusion world model, and a CS:GO world model trained on 87 hours of human play runs at about 10 fps on an RTX 3090 but lets the player jump repeatedly in mid-air, which the real game forbids.
Oasis (Decart and Etched, October 2024) generated a Minecraft-like open world at 20 fps in a public browser demo with 500M-parameter open weights; reviewers found the layout rearranged after turning around, and Decart did not say it had Microsoft's permission to train on Minecraft footage.
src-decart-2024-oasis, src-techcrunch-2024-decart-oasis, src-infoq-2024-decart-etched-oasis
MineWorld (Microsoft Research, 2025) released an open visual-action autoregressive transformer for Minecraft that generates 4 to 7 frames per second depending on model scale.