muse-wham technique sourced
Muse World and Human Action Model (WHAM)
Microsoft Research's WHAM is a transformer trained on seven years of Bleeding Edge play that generates both future frames and the player's controller actions, aimed at gameplay ideation, with a real-time Quake II follow-up (WHAMM).
mechanism
Frames are encoded into discrete image tokens and controller inputs into action tokens; a decoder-only transformer models the interleaved sequence, so it predicts both how the world evolves and what a human player is likely to do next. Prompting with a few frames yields many plausible continuations (diversity); a designer can edit a frame, such as inserting a character, and the model carries the edit forward (persistency). WHAM-1.6B produces about one frame per second at 300x180. WHAMM replaces token-by-token decoding with MaskGIT-style parallel prediction (a 500M backbone plus a 250M refiner) to reach more than 10 fps at 640x360 on Quake II.
limitations
- WHAM-1.6B generates about one image per second at 300x180, too slow and too small for play; WHAMM trades data scope (one level section) for speed.
- Needs years of recorded human play with consent terms for a single game; few studios hold such datasets.
- Context of under a second (WHAMM) means objects out of view are forgotten.
- Counts, damage, and other exact quantities are unreliable in generated frames.
relations
3 stored edgesDirect avatar control
In WHAMM the avatar's movement and camera are rendered by the model from controller input, at 10+ fps with a 0.9 second context.
Generative content on demand
Designers insert an object or character into a frame and the model generates plausible gameplay around it (persistency), giving a preview of how a content change might play before it is built.
Rigid-body physics
Movement and collisions follow learned game dynamics; the Nature paper's consistency criterion is that generated play respects controller input and the game's physics, which is measured statistically rather than enforced.
claims
3 with sourcesThe Nature paper (February 2025) frames consistency, diversity, and persistency of user modifications as the three capabilities generative models need to support game ideation, and shows WHAM provides them.
WHAM was trained on more than 1 billion images and controller actions, over 7 years of continuous play, from Bleeding Edge matches recorded under the game's EULA; model weights, sample data, and the WHAM Demonstrator were released on Azure AI Foundry.
WHAMM ran a playable section of one Quake II level at 10+ fps and 640x360 after training on one week of gameplay data; enemies were blurry, damage could be wrong, context was 0.9 seconds, and health counters were unreliable.