11-what-breaks 3,569 words about 16 min

Chapter 11. What breaks

You and a friend are playing a strategy game in lockstep multiplayer against a computer opponent. Neither machine sends the other the state of the world. Each runs the full simulation itself, and only the players' orders travel over the network; as long as both machines start from the same state and apply the same orders on the same ticks, they stay identical. The opponent plans with Monte Carlo tree search, given a fixed slice of time to think before each decision.

Your friend's computer is faster. In the same slice of time it plays out more futures, and on one decision the move it has visited most is a different move from the one your machine settled on. For a few seconds nothing looks wrong. The two worlds have already split: on one screen the enemy's cavalry turns north, on the other it holds. Then the machines compare checksums, the numbers disagree, and the match stops with a desync error that neither of you can do anything about.

The dataset's record for the technique lists this exact failure: time-budgeted search returns different moves on different hardware, which breaks replays and lockstep multiplayer unless budgets are counted in iterations with seeded randomness. The fix is cheap once you know it. Count playouts instead of milliseconds, seed the random numbers, and every machine computes the same move. The record therefore marks the technique as mitigable.

AI techniques tend to fail inside simulation games in the same few places, because simulation games demand things that most software does not: the same inputs must produce the same world, every output a player sees becomes evidence for how the system works, and a single purchase has to pay for hundreds of hours of play.

The same inputs, the same world

Factorio's record notes that its item movement is discrete and deterministic, belts carrying a fixed number of items per second on two lanes, and that this is what lets its lockstep multiplayer simulate identical worlds on every machine. The same property supports a long list of things players never think about. A replay is a recording of inputs, played back through the same rules. A bug report from a player can be reproduced by loading the save and replaying the inputs. A leaderboard can be trusted because a submitted run can be re-simulated. A save file loads into the same world that was saved.

Each technique in the dataset records its effect on that property, and chapter 6 counted them. The ones that break it share a cause. A frame-generating world model, a language-model game master, a society of language agents, a realtime voice model: each samples its output, the sample depends on hardware and timing and the model's version, and its output decides what happens next.

A mitigable technique can be pinned down by seeding its randomness, storing its output, or bounding its computation by work instead of time. Infinite Craft stores the output, as chapter 8 described. A small model running on the player's own GPU can be seeded, which is why the record for on-device language models marks them mitigable too.

The techniques that leave reproducibility intact mostly never touch the simulation. Neural upscaling and frame generation work on the rendered image after the simulation has decided everything, which is part of why they became the one learned technique that is a norm in games (chapter 7). The others are deterministic algorithms run at play time, such as wave function collapse, which Townscaper runs in real time to work out how the houses you place fit together. A deterministic algorithm can generate as much as it likes; given the same seed and the same clicks, it generates the same town.

The bets in chapter 10 inherit these properties from the techniques they apply. Of the 42 opportunities, 17 rely on at least one technique that breaks reproducibility outright. Each of them needs a plan for what the log records, or it gives up replays and multiplayer.

The requirement reaches mods too. The record for modding interfaces lists sandboxing and determinism requirements among a designer's levers, because in a lockstep game a mod runs on every machine and has to compute the same thing on each. A mod written from a sentence, the language-modding bet in chapter 10, inherits that requirement: the code has to be deterministic even though the model that wrote it is not. The record states the open problem directly, as a question about what interface design keeps generated mods sandboxed, deterministic, and compatible across updates.

Where the computation happens

Each technique record also says where the technique runs: offline while content is made, as an assistant to a designer, on the player's machine during play, on a server during play, or over logged data after the fact. The place decides who pays and what can go wrong. 18 techniques can run on a server during play, and 10 of them break reproducibility outright; 21 can run on the player's machine, and 3 of those break it. 11 never run during play; they are used while the game is made or on its telemetry, like the satellite reconstruction of chapter 7, whose output reaches the player as fixed data.

Figure: What runtime techniques do to reproducibility. Techniques that can run during play, on a server or the player's device, grouped by what they do to reproducibility. Within each group, shipped techniques come first and are set in dark text. Interactive version: https://sim.yohei.me/#fig-determinism-runtime

The techniques that run only on a server are where the breaks gather: of 13, 7 break reproducibility, and every technique that breaks it can run on a server. A server adds a network round trip to every call and a GPU bill per player, as the record for real-time restyling notes of its cloud stream. A hosted model may not be available to build on at all: the record for Google's world models describes access as a hosted research prototype for paying subscribers, with no public interface, hourly price, or version for the player's own machine published.

Contradiction

A contradiction does more damage than a gap, because players use what the game shows them to build their model of how it works. A colonist who mourns a friend who is alive, a road that was there a moment ago, a price that changes when nothing changed: each teaches the player something false.

Consistency is listed as a risk on 20 of the 45 techniques in the dataset, and chapter 8 gave the measured cases: a language model predicting text-game states, a world model letting a player jump in mid-air, language-model characters inventing quests. Characters forget, too. In a 28-player study with five language-model characters, players reported characters forgetting their own earlier statements in 10 percent of reports [1].

Generated pictures can contradict the state they stand for. Real-time restyling, whose door-shaped solid block chapter 7 described, can also change an object's appearance between views, because its record notes a short frame window and no long-term memory. In a frame-generating engine, health, ammunition, inventory, and score are only as reliable as the pixels that show them. Generated levels fail in their structure: an A* agent could not complete about 12 percent of the levels MarioGPT generated, so every output needs a solver to check it [2], and EA's match-3 generator, conditioned on difficulty, matched the requested difficulty a third of the time [3].

The pattern points to where a language model can safely sit. Asked to predict what the simulation will do, a model gets it wrong often enough to mislead. Asked to describe what the simulation did, from a log of events the simulation computed, it has the right answer in front of it. The record for language-model explanation of state draws the same line: explanations of dynamics should be computed by the simulation and only narrated by the model, and because the explanation layer reads state without writing it, the simulation stays deterministic.

Opaque replacements

Deep simulations already struggle to be understood. Chapter 5 showed that the mechanics rated most emergent are rated hardest to read, and designers answered with a whole family of mechanics that own no simulation state and exist to explain the rest: overlays, inspectors, tooltips, chronicles. A learned component can undo that work. A policy trained to run a city's traffic or a rival's pricing may play well and still offer no reason for what it does, which moves the mechanic it replaced down the legibility scale.

Legibility is listed as a risk on 9 techniques, including Monte Carlo tree search, whose choices come from thousands of playouts no player will see, and mean-field games, which replace a crowd of individual decisions with one representative agent responding to the population as a whole, so there is no individual decision to inspect. The remedy the dataset proposes is the same for every case: each learned component that affects state needs an explanation path, and the cheapest one is a log of what it decided and on what inputs.

The bill

Most simulation games are sold once and played for a very long time. A feature that costs money every time a model is called turns each player-hour into a running cost. Compute cost is listed as a risk on 29 techniques, second only to controllability.

The numbers behind that risk vary by orders of magnitude. The 25-agent Smallville simulation behind the original generative agents paper cost thousands of dollars in model credits and took several days of wall-clock time [4]; a later study cut token use for the same 25-agent town to 42.7 percent of the baseline by replaying cached plans [5]. When Google opened Project Genie, built on its Genie 3 world model, to subscribers in January 2026, each generated world was limited to 60 seconds, a limit DeepMind attributed to limited compute [6]. At the other end are the small on-device models of chapter 7, which fit beside the game's rendering on the player's own card. Local inference moves the cost onto the player's hardware and off the developer's bill, which is the case the local-first opportunity in chapter 10 makes.

Latency

Latency is listed as a risk on 13 techniques. The record for realtime voice, whose speech models chapter 7 timed, notes a trade: the most expressive delivery and the lowest latency pull against each other, and the most expressive speech synthesis was not suited to real time when it launched. Interactive video models are slower still: distillation took Hunyuan-GameCraft from 0.25 to 6.6 frames per second, with under five seconds of latency for each action [7], and the record notes that conditioning more strongly on history makes such a model more consistent and less responsive to new actions.

Small models on the player's machine trade the latency problem for a capability problem. The record for on-device language models notes that they reason less well over several steps, so vendors send strategic planning to cloud models, which brings back the cost and the latency the local model was meant to remove. It also notes that model sizes and latencies in shipped titles are mostly undisclosed.

Steering and checking

The risk listed most often across all 45 techniques is controllability: the designer cannot reliably make the technique do what the design needs. It appears on 34 of them. A language-model game master tends to accommodate requests a human game master would refuse [8]. A world model accepts a prompt for weather or a new character mid-session and decides for itself what that looks like. A level generator produces something valid that no designer would have chosen.

Close behind is evaluation, listed on 24: when a system can produce an unbounded variety of outputs, nobody can check them all. Automated playtesting can check that generated content is valid and balanced; it says little about whether a generated event is any good. Human review does not scale to generated variety. This dataset faces the same problem with its own records and answers it with a verification queue: sampled, scored against anchors, and logged (chapter 13). Games will need something similar for what their generators produce.

Playtesting agents differ from players in skill and style, and agents that imitate players need large logs of real moves, which a new game lacks before launch. Rewards for coverage or curiosity stand in for the designer's intent, since, as chapter 7 noted, an open-ended sim gives them no single goal, and they can miss balance problems that appear only over long play. Societies of language agents are validated by matching stylized findings, which makes it hard for a designer to tune or test what the society does as a whole, and agents built on pretrained models reproduce institutions people already have, such as taxes, votes, and religion, so their outcomes are bounded by their training data.

Trust, authorship, and safety

Players and the people who make games have their own reasons to distrust generated content, and chapter 7 covered how reception has gone: a moderation failure at AI Dungeon, a strike over voice replicas, and store rules that ask live generators to describe their guardrails.

These risks fall unevenly across the bets. Opportunities that help players understand a game or author more of it, such as explanation, modding by description, and orders given as intentions, replace no one's work and generate nothing a player did not ask for. Opportunities that replace authored writing, voices, or art carry the whole weight of the labor and trust questions.

The table lists every risk the dataset records, with the number of techniques that carry it.

Provenance

Intellectual property and provenance are listed as a risk on 9 techniques, and several records trace it to training data. The open interactive video models learned from recordings of commercial games: Hunyuan-GameCraft from more than a million recordings of over a hundred AAA titles [7], Matrix-Game 2.0 from about 1,200 hours that include Grand Theft Auto driving and Minecraft [9]. Their record calls the provenance of the output contested. The frame-predicting engines of chapter 7 are trained on footage of the game they reproduce, so their record describes the approach as cloning a game rather than authoring a new one and lists the licensing of that footage as unresolved. Learned level generators trained on commercial levels such as Super Mario Bros. raise the same questions for a shipped game, and text-to-3D models differ in where their training data came from and under what licence; TRELLIS is released under the MIT licence [10].

Some of the risk sits in data a studio would collect itself. The game-world model of chapter 7 shows the cost: its record says few studios hold years of consented play from one game. Models of human behaviour built from interviews need hours of data per person, which raises consent and privacy questions for player data. Even a technique that leaves reproducibility intact can carry the risk: rebuilding a planet from imagery needs a licensed imagery source.

Labor and safety

Labor displacement is listed on 7 techniques and content safety on 7. The record for realtime voice carries both. Beyond the consent terms the strike won (chapter 7), the record notes that pay for replicas is now contracted, and that players have marked games down in reviews for AI voices. It also notes the control problem chapter 7 described: speech-native models are hard to hold to the actions a game allows.

A model on the player's machine needs its filters there too, and the record notes that players can extract or modify local weights. A shared cache has the opposite exposure: the record for language-model rule adjudication says that because every cached answer becomes permanent shared content, outputs must be filtered for offensive results. A hosted world model enforces its own limits; Genie's guardrails refuse copyrighted settings and nudity, which narrows the worlds players can make.

Risks the bets name

The opportunity records list risks of their own, and many of them concern what a feature does to the game itself.

• A design that removes a chore can remove a skill. Intent-level orders risk hiding the systems some players enjoy mastering, and an ambiguous intent compiles to a surprising rule. Explained delegation risks a late game spent approving a manager's proposals, and a manager that imitates a novice repeats the novice's mistakes.
• A design that shapes events can feel rigged. The drama-aware storyteller lists players perceiving arc-shaped events as rigged, and the same arcs recurring across runs if the scoring is too deterministic. Placement that tells stories lists players arranging their towns for favourable events.
• A design that adds realism can take away play. Realistic weather can be less fun than authored seasons, research that depends on which scientists work together frustrates a player planning a build, and a workforce to manage can bury the logistics puzzle a factory player came for.
• The serious-sim bet lists users over-trusting calibrated outputs, and the planning sandbox lists simulated residents standing in for real community input, and political misuse.
• Balancing by search has blind spots. Bots find exploits humans never would and miss ones humans find, and balancing too hard removes asymmetries that make a game interesting.

A log as the fix

One design answers several of these failures at once, and chapter 10 ranked it among the top bets. The event-sourced simulation runtime records every state change as an event with its inputs, its output, and its cause, including the changes a model decided. The log is the source of truth and the game state is a projection of it.

Each failure above has a corresponding use of the log. A replay or a multiplayer client reads the logged output of a model call instead of calling the model again, so a technique that breaks reproducibility becomes reproducible after the fact; the opponent's move in the desynced match would have been decided once and sent to both machines as an event. A bug report includes the log, so the developer can step through what the model said. A why-query can cite the events that caused an outcome, and a branch preview can fork from any event. The record lists the precedents for the parts: Dwarf Fortress's legends, Factorio's deterministic lockstep, and EVE Online's persistent world.

The validation experiment is plain engineering. Wrap the model calls of an agent prototype in an event log, replay a hundred sessions, and check that the state is reconstructed byte for byte and that forked branches differ only after the fork point. Success is every replay reproducing, with the log adding less than 20 percent to the size of a save per hour of play. The prediction is that by the end of 2028 an engine, a middleware vendor, or an open-source project will ship an event-log replay layer designed for games with generative components, used by at least one title with 1,000 Steam reviews.

A second tool limits what a model can write in the first place. Constrained decoding masks every token that would take the output outside a schema or grammar, so a model asked for a game action can only produce an action the engine can execute, against targets that exist. A hosted structured-output mode reached 100 percent on a complex schema-following test, against under 40 percent for an earlier model without constraints [11]. Strict formats have a cost: they reduced models' reasoning accuracy compared with free-form answers, and stricter formats cost more [12]. The record also states that no source documents constrained decoding in a specific shipped game; it is scored as research for games despite being routine elsewhere.

Together the two tools describe an architecture. The authoritative simulation stays symbolic and deterministic. Models propose changes in a typed format that a validator accepts or rejects, every accepted change is logged, and everything a player sees can be traced back to an entry in the log. The narrative engine behind Hidden Door applies the validator half of this design: it checks every story turn against its canon and rules before showing it [13].

Exercise: trace a failure

Pick an AI feature you have seen in a game, or one from chapter 10.

1. Write down what state it writes. If the answer is "none", say what it reads and what the player sees.
2. Decide whether a replay of the session would show the same output. If not, write what the log would have to record for it to do so.
3. Describe the first contradiction a player is likely to see, and how the player would find out that the game was wrong and not their understanding of it.
4. Estimate the cost per player-hour, and who pays it.

The exercise sorts features into two groups: those that can be fixed with a log and a schema, and those whose value depends on doing something no log can make repeatable. Whether the first group ships first is one of the questions the predictions in chapter 12 are set up to answer, starting on 31 December 2027.

Sources

[1] Xiangyu Peng, Jessica Quaye, Sudha Rao, Weijia Xu, et al., Bill Dolan (2024-04-25). Player-Driven Emergence in LLM-Driven Game Narrative. arXiv https://arxiv.org/abs/2404.17027
[2] Shyam Sudhakaran, Miguel González-Duque, Claire Glanois, Matthias Freiberger, Elias Najarro, Sebastian Risi (2023-02-12). MarioGPT: Open-Ended Text2Level Generation through Large Language Models. arXiv https://arxiv.org/abs/2302.05981
[3] Monica Villanueva Aylagas, Joakim Bergdahl, Jonas Gillberg, Alessandro Sestini, Theodor Tolstoy, Linus Gisslén (2024). Improving Conditional Level Generation using Automated Validation in Match-3 Games. IEEE Transactions on Games https://media.contentapi.ea.com/content/dam/ea/seed/presentations/seed-tog-conditional-level-generation-paper.pdf
[4] Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, Michael S. Bernstein (2023-04-07). Generative Agents: Interactive Simulacra of Human Behavior. UIST '23 (ACM Symposium on User Interface Software and Technology) https://arxiv.org/abs/2304.03442
[5] Yangbin Yu, Qin Zhang, Junyou Li, Qiang Fu, Deheng Ye (2024-02). Affordable Generative Agents. arXiv https://arxiv.org/abs/2402.02053
[6] Rebecca Bellan (2026-01-29). I built marshmallow castles in Google's new AI world generator Project Genie. TechCrunch https://techcrunch.com/2026/01/29/i-built-marshmallow-castles-in-googles-new-ai-world-generator-project-genie
[7] Jiaqi Li, Junshu Tang, Zhiyong Xu, and others (Tencent Hunyuan) (2025-06). Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Condition. https://arxiv.org/abs/2506.17201
[8] Roberto Gallotta, Graham Todd, Marvin Zammit, Sam Earle, Antonios Liapis, Julian Togelius et al. (2024). Large Language Models and Games: A Survey and Roadmap. IEEE Transactions on Games https://arxiv.org/abs/2402.18659
[9] Xianglong He, Chunli Peng, Zexiang Liu, and others (Skywork AI) (2025-08). Matrix-Game 2.0: An Open-Source, Real-Time, and Streaming Interactive World Model. https://arxiv.org/abs/2508.13009
[10] Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang et al. (2024-12). Structured 3D Latents for Scalable and Versatile 3D Generation. CVPR 2025 https://github.com/microsoft/TRELLIS
[11] OpenAI (2024-08-06). Introducing Structured Outputs in the API. OpenAI blog https://openai.com/index/introducing-structured-outputs-in-the-api/
[12] Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, Yun-Nung Chen (2024-08-05). Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models. arXiv https://arxiv.org/abs/2408.02442
[13] Hidden Door (2026). Hidden Door. hiddendoor.co https://www.hiddendoor.co/

Dataset dfde7895aa6d.

mechanics in this chapter

2 linked or named

SeasonsWeather