# Self-play planning with learned models (AlphaZero, MuZero, Dreamer) *Technique* · `model-based-planning-muzero` · status: **sourced** Agents that combine tree search or imagined rollouts with neural policy and value networks trained by self-play, where MuZero and Dreamer also learn the environment's dynamics, reach superhuman play from rules or pixels alone. | field | value | |---|---| | aliases | AlphaZero, MuZero, EfficientZero, DreamerV3, Dreamer 4 | | category | Reinforcement learning | | year | 2020 | | maturity | Research | | trl | 4 | | runtime | Offline authoring, Client real-time | | determinism impact | Mitigable | | horizon | 3-5 years | | risks | Compute cost, Controllability, Legibility, Evaluation difficulty | | builds on | [Monte Carlo tree search for game AI](../technique/monte-carlo-tree-search.md) | | affects count *(computed)* | 5 | ## Summary Agents that combine tree search or imagined rollouts with neural policy and value networks trained by self-play, where MuZero and Dreamer also learn the environment's dynamics, reach superhuman play from rules or pixels alone. ## Mechanism AlphaZero pairs MCTS with a network that outputs a move prior p and a value v for a state; games generated by the search train p toward search visit counts and v toward outcomes, so search and network improve each other given only the rules. MuZero drops the need for rules at planning time: a representation network encodes observations into a latent state, a dynamics network predicts the next latent state and reward for an action, and a prediction network outputs policy and value, so MCTS runs entirely in the learned latent space. EfficientZero adds self-supervised consistency and value-prefix losses for sample efficiency. Dreamer learns a recurrent world model and trains an actor-critic on imagined trajectories; DreamerV3 adds symlog and return normalization so one hyperparameter set works across domains, and Dreamer 4 trains its world model mostly on unlabeled video and learns the policy offline inside it. For design work the same loop can be retrained on each rule variant and the converged play compared. ## Limitations - Training is expensive: MuZero's board-game runs used 16 TPUs for training and 1,000 for self-play, and each significant rule change needs retraining or fine-tuning. - Self-play converges to strong play, not human play, so balance conclusions can reflect exploits humans would never find or use. - Learned dynamics drift over long rollouts; MuZero did poorly on 6 Atari games and DreamerV3 needed about 30M steps (17 days of play) for its first Minecraft diamond. - Most simulation games are many-player, simultaneous-move, partly hidden, and long-horizon, which the two-player perfect-information AlphaZero setup does not cover directly. - Policies are opaque, so designers cannot read why an AI rival chose a strategy, and giving it a personality requires reward shaping rather than editing rules. ## Sources - src-silver-2017-alphazero - src-schrittwieser-2020-muzero - src-wikipedia-muzero - src-ye-2021-efficientzero - src-tomasev-2020-alphazero-chess-variants - src-hafner-2025-dreamerv3 - src-hafner-2025-dreamer-4 - src-deepmind-2022-muzero-youtube ## Relations **affects** - [Rival firms](../mechanic/rival-firms.md) (Replaces substrate): Competitor AI becomes a policy trained by self-play against the game's own rules, so it re-adapts when economic rules change in a patch instead of needing rewritten heuristics; training compute per rule set is the cost. - [Tech tree](../mechanic/tech-tree.md) (Tunes / balances): Self-play agents trained on each candidate tree reveal dominant research orders and unused branches, the same comparison AlphaZero made across chess rule variants. - [Policy toggles](../mechanic/policy-toggles.md) (Tunes / balances): Each toggle combination is a rule variant; converged self-play under each variant measures how much a policy shifts outcomes and flags toggles that are strictly dominant or never worth enacting. - [Creature learning](../mechanic/creature-learning.md) (Deepens): Sample-efficient model-based learners (EfficientZero needed about two hours of experience) make it plausible for a creature to learn new behaviors from player reward within a session, by planning in a learned model of its world rather than reacting with a fixed network. - [Military operations](../mechanic/military-operations.md) (Deepens): Opponents plan with a learned value function plus search over unit actions, giving strong play in operational simulations where hand-written doctrine is brittle; latent models allow planning where exact rollouts are too slow. **builds on** - [Monte Carlo tree search for game AI](../technique/monte-carlo-tree-search.md) ## Backlinks (computed) **applied in:** [Explained delegation to AI managers](../opportunity/explained-delegation.md), [Trainable creatures revival](../opportunity/trainable-creatures-revival.md) ## Claims - AlphaZero reached superhuman chess, shogi, and Go within 24 hours from random play given only the rules, beating a world-champion program in each game. *(confidence: high)* [[314](../source/src-silver-2017-alphazero.md)] - MuZero matched AlphaZero in Go, chess, and shogi and surpassed R2D2 on Atari while learning its own model instead of being given the rules (Nature, December 2020). *(confidence: high)* [[315](../source/src-schrittwieser-2020-muzero.md)] - MuZero's board-game training used 16 TPUs for training and 1,000 for self-play at 800 simulations per move, and it performed poorly on 6 Atari games. *(confidence: medium)* [[316](../source/src-wikipedia-muzero.md)] - EfficientZero reached 194.3% mean and 109.0% median human-normalized score on Atari 100k from about two hours of real-time game experience. *(confidence: high)* [[317](../source/src-ye-2021-efficientzero.md)] - AlphaZero trained from scratch on nine chess rule variants was used to compare decisiveness, piece values, and dynamism across variants as an in silico game balance assessment. *(confidence: high)* [[318](../source/src-tomasev-2020-alphazero-chess-variants.md)] - DreamerV3 (Nature, April 2025) beat specialized methods on over 150 tasks with one configuration and was the first algorithm to collect Minecraft diamonds from scratch without human data, first at about 30M environment steps (about 17 days of play). *(confidence: high)* [[319](../source/src-hafner-2025-dreamerv3.md)] - Dreamer 4 (September 2025) learned a Minecraft world model that runs interactively on one GPU and obtained diamonds from offline data only, choosing sequences of over 20,000 keyboard and mouse actions from pixels. *(confidence: medium)* [[320](../source/src-hafner-2025-dreamer-4.md)] - MuZero-RC chose per-frame VP9 quantisation on part of YouTube live traffic for an average 4% bitrate reduction, its first production deployment, outside games. *(confidence: high)* [[321](../source/src-deepmind-2022-muzero-youtube.md)] ## Sources - [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](../source/src-silver-2017-alphazero.md) - [Mastering Atari, Go, chess and shogi by planning with a learned model](../source/src-schrittwieser-2020-muzero.md) - [MuZero](../source/src-wikipedia-muzero.md) - [Mastering Atari Games with Limited Data](../source/src-ye-2021-efficientzero.md) - [Assessing Game Balance with AlphaZero: Exploring Alternative Rule Sets in Chess](../source/src-tomasev-2020-alphazero-chess-variants.md) - [Mastering diverse control tasks through world models](../source/src-hafner-2025-dreamerv3.md) - [Training Agents Inside of Scalable World Models](../source/src-hafner-2025-dreamer-4.md) - [MuZero's first step from research into the real world](../source/src-deepmind-2022-muzero-youtube.md) --- Provenance: created in run `2026-10-24-techniques-rl-design`.