13-how-the-book-checks-itself 3,167 words about 14 min

Chapter 13. How the book checks itself

On 25 October 2026 an AI agent catalogued three mobile management games, Pocket City, Fallout Shelter, and Game Dev Story, and ran the dataset's checks before opening a pull request. The data passed. The paper did not. A sentence in the paper's section on white space, the pairs of common mechanics that almost never meet, had stopped being true, and the build refused to finish.

The sentence named building placement and embodied skill control, real-time motor skill such as hand-flying a plane or driving a car, as one of the most promising empty pairs. Behind it sat an assertion that the pair ranks in the top ten of the white-space table. The new games did not touch either mechanic. The trouble was the table's entry rule: it admitted mechanics used by at least 8 percent of the catalog, and embodied skill control was used by exactly 14 games, sitting on the bar. Three more games without it pushed it under 8 percent, it dropped out of the table, and the paper's sentence lost its footing even though no finding had changed.

The build caught it, and also showed a weakness in how it reported the problem. The error said "unknown reference", which pointed at a typo rather than at the cause. The fix, recorded in the project's fifth decision record, changed two things. The table now admits mechanics used by at least a fixed number of games, 14, which was 8 percent of the catalog when it was fixed, so adding games can no longer push a mechanic out. And an assertion about a row that has left a table now fails with a message saying so. Before the change was accepted, the same assertions were run against scratch copies of the catalog with the three games added and with up to twenty duplicated games on top; under the old rule the three games alone broke one, and under the new rule none broke.

It is a small example of how a dataset, a paper, and a book built from it stay true while people and agents keep adding to them.

Everything is a build

The dataset is a folder of plain-text records: one file per game, mechanic, technique, opportunity, and source, plus a log of every run that changed anything. There are 177 game files, 199 mechanics, 45 techniques, 42 opportunities, and 666 sources. Everything else is computed from those files on every build: the paper, the website, the wiki pages, the tables and figures, and this book. Nothing derived is stored. The prevalence of a mechanic, the earliest game that uses it, a game's cluster, an opportunity's priority: each is recomputed from the records every time, because a stored copy would drift from the data it was copied from.

Figure: From records to every product. Records pass the validator and feed one deterministic analysis; the paper, the site, the wiki, the book, and the scripts all render from it, and every interpretive sentence in them carries an assertion the build checks. Interactive version: https://sim.yohei.me/#fig-pipeline

Prose is the hard part, because sentences interpret numbers. Two rules handle it. Every number about the dataset in this book is a placeholder that the build fills in, so no count is typed by hand; a literal count next to a word like "games" or "mechanics" fails the build. And every sentence that interprets a number carries an assertion that the build evaluates. When chapter 1 says that the five most common mechanics model nothing in the world and that production chains come sixth, the build checks the five ranks and the sixth. When the data stops supporting the sentence, the build fails until someone rewrites it.

The build is also deterministic. The same files produce byte-identical output, and the project's checks prove it on every change by building twice under different settings of Python's hash randomization and comparing the results. That is what lets an edition of this book name the exact version of the dataset it came from.

One file per record

The architecture was set down in the project's first decision record, which names four usual sources of drift in a shared dataset: the same fact stored in two places, open-ended fields and tags, prose that restates numbers, and contributors who each reinterpret a category. Every record is its own YAML file, so a merge conflict stays inside one game or one mechanic. Schemas are closed, and their annotations say which fields take a closed vocabulary, which point to another record, and which are scored on an anchored scale; the validator and the build read those annotations, so a new field that points to another record appears in the graph, the wiki, and the tables without new code. Every link between records has one owner, and the reverse link is computed. Changing a vocabulary, a rubric, a schema, or a rule needs a decision record, and the documentation of the rules is generated from the code that enforces them.

The record also lists what was rejected. One large YAML or CSV file would be simpler, but every edit would conflict and partial validation would be hard. A graph database would answer queries better and make changes harder to review, so the build exports a database for queries instead. Free tags with periodic cleanup were rejected because they drift. The cost was accepted on purpose: a contributor cannot add a tag casually and has to file a proposal, which slows the growth of the taxonomy.

Eleven ways to drift

The second decision record came from a test designed to break the first. An independent agent, given only the repository, followed the agent instructions (AGENTS.md) to add a game and then tried eleven ways to introduce drift. Most were caught. The ones that were not became rules. Any second run could mark a record verified. A literal count slipped past the lint when a word sat between the number and its noun, or when it was written as a percentage. Interpretive prose had no check against the data. Summaries could carry em dashes and hype. A run of any kind could create new mechanics. The documented order of the final checks could never succeed after an edit.

The fixes gave every run a kind, so that only taxonomy and bootstrap runs create mechanics and only verification runs can justify a verified status. The count lint now catches a number up to one word before a dataset noun and exempts sentences that cite an outside source. The assertion syntax this book uses in every chapter came from the same record. And the contract itself, meaning the vocabularies, schemas, scripts, and agent instructions, went under maintainer review, because a validator can enforce a contract but cannot judge a change to it.

Failures that shaped the rules

Most of the rules exist because something went wrong once.

The first version of the catalog was built by agents working in parallel, each cataloguing a slice of games. An independent reviewer, reading the paper against the data, found that they had tagged each game's distinctive mechanics and skipped the ubiquitous ones: pause controls, building placement, tech trees. Prevalence, clusters, and white space had all been computed on incomplete decompositions, and chapter 1's finding that the most common parts of a simulation game simulate nothing could not have been seen in that data. The gaps were filled, clustering was changed to weight each mechanic by how rare it is, so that two games sharing a pause button count for less than two games sharing a storyteller, and the analysis was redone.

The same review found prose that contradicted the records. The paper called some techniques determinism-breaking that the data scored as mitigable. The fix was a new kind of assertion that reads a stored field directly, so the sentence in chapter 11 that calls a technique mitigable is checked against that technique's record.

A later review found that the cluster findings were fragile. The reviewer removed one game at a time and re-ran the analysis; several sentences about which games belong together failed, because the subsamples used to test the clusters' stability were drawn by position and changed whenever the catalog did. Each game's membership in each of the 1000 subsamples is now fixed by a hash of its id. A sentence may say two games belong together only if they share a cluster in at least 0.9 of subsamples, and the assertions were re-tested with every single game removed in turn.

The number of clusters has its own rule, and the rule has already moved. The build picks the smallest number of clusters whose stability across subsamples reaches a stated threshold. When the rule was first applied it gave 24, which was then fixed in the configuration so that each new game would not reshuffle every cluster. On the current data the rule would pick 19. The difference sits inside the stated tolerance, so the build warns only if it grows, and chapter 4 says what this means for the families it describes: the curve is flat, neighbouring choices are about as defensible, and only groupings that survive resampling are asserted.

A field and a mechanic that disagreed

The fourth decision record, written before outside contributors were invited, found record text that contradicted the data. Two opportunity theses called the tech tree the most common mechanic in the catalog and production chains the most used rule-system, when they ranked third and fifth at the time. Report prose was pinned by assertions; record text was not. Records may now carry assertions of their own, and a record whose text states a dataset finding, caught by phrases such as "in the catalog" or "most-used", must carry at least one.

The same review found two mechanics that duplicated fields. Pause and speed control duplicated each game's time model, and mod support duplicated its modding field, and the pairs disagreed: a few games that could be paused lacked the pause mechanic, and nearly thirty games with scripting or total-conversion modding lacked mod support, RimWorld among them, although the mechanic's own definition cited RimWorld. Vocabulary terms can now require or allow a mechanic, and the validator checks every game against them. When the data was migrated, mod support rose to the third most used mechanic in the catalog, where it sits today. The migration also reshaped the head of the distribution: production chains, fifth when the review began, fell to sixth behind mod support, the order chapter 1 reports.

Until the same review, every projection the build makes was committed to the repository and checked byte for byte. Every pull request that changed data carried thousands of regenerated lines, and any two pull requests open at once conflicted. Projections are now built and never committed. The check builds them in memory and compares only the generated blocks that change with the contract, a vocabulary, or a rule, and the website is built and published by a workflow from the same sources.

Four pilots

The fifth decision record followed four pilots. Before outside contributors were invited, four agents each took one task end to end, working only from the written briefs, each in its own copy of the repository: archive twenty-five sources, add verbatim quotes to eight records, add store identifiers to ten games, and catalogue the three games that opened this chapter. Each kept a log of where it got stuck.

Some checks had passed without checking anything. The source checker printed only warnings when every fetch failed, which looked like a pass; it now prints that nothing was checked. The local check before a pull request fell back to the machine's user name when no GitHub handle was configured, which skipped the operator check without saying so; it now refuses to run. The scope check skipped run files when a pull request changed no records; it now checks them, and rejects a pull request that adds a run and changes nothing else.

The briefs had gaps the agents found by falling into them: which archived snapshot to use, which store edition counts for a game first released free or on another platform, what to do with a claim that overreaches or states an absence, and whether adding a quote counts as verification. It does not; quoting runs as maintenance, because it needs no independence and changes no status. The agents' web tools could not reach the web archive, Wikidata, or IGDB from their sandbox, so the briefs now say which tasks suit people, or agents running with full network access. In the catalog pilot, the extracted page text gained a space where an inline tag met punctuation, so four correct quotes failed; the new failure message showed the cause at once.

The catalog pilot also fixed the threshold of 14 games at which a mechanic counts as common in the vacant-lot table, and the decision record sets the review chapter 9 asks for when the catalog passes about two hundred and fifty games.

Status is a claim about evidence

Every record has a status, and the status is a claim about its evidence. asserted means no source yet. sourced means the record cites sources. verified means someone other than its creator checked it against a primary source. Today 391 records are sourced or verified and 71 are asserted, among them every opportunity, since an opportunity is an argument with no source to cite.

A claim may carry a verbatim quote from its source, and the project's continuous checks fetch the source and look for the quote whenever the record changes. The first time this ran on agent-written records, it found 26 of 32 quotes, warned on five pages it could not reach or could read only as an abstract, and failed one. The failed quote was on the page; the checker had been served a variant of it. The checker now fetches a page a second time before failing a quote, and a failure says how far the quote matched and what the page had instead.

Disagreement is recorded rather than settled by overwriting. Utopia, the Intellivision city builder, has two release years in its sources: Wikipedia dates it to 1982 and Guinness World Records to 1981 [1, 2]. The record keeps both, marks itself contested, and states which year it uses and why. It is the only contested record so far.

No record is verified yet. The reason is the independence rule. A verification must come from a run with a different operator from the run that created the record and, when both runs are agents, a different model. I built the dataset by directing AI agents, and every run so far has had the same operator: 39 runs, all by agents, all operated by me. Until other people take part, nothing can move to verified.

The same reasoning covers scores. Every rubric score in the dataset, from legibility to AI leverage, is one rater's reading of written anchor text, and so far that rater is one model. A second rater must first score a calibration set of 20 mechanics and agree with its reference scores, within one point, on at least eight in ten. The set was frozen and chosen across the mechanic families, greedily, to cover as many combinations of scale and anchor as possible, so changing one of its scores is a reviewed change. Its reference scores are public. The gate therefore shows that a rater reads the anchors the way the set does; it cannot show that the rater scored blind or did not rescore until it passed, and the scoring tool reports only overall agreement to make rescoring less informative. No independent ratings exist yet. Any finding in this book that rests on scores rests on that one rater.

Runs and their limits

Every change belongs to a run: a record that says who operated it, whether a person or an agent did the work, which model and which software ran it, what the run was for, and what it did, including what failed. Runs are append-only. A record names the run that created it and every run that later reviewed it, and the history of any fact can be reconstructed from the run log and the version history. A pull request carries exactly one run, changes at most a bounded number of records, never deletes a merged record, and is signed off by its operator.

Runs so far fall on 5 days, the first on 23 October 2026. It holds 12 catalog runs, 9 taxonomy runs, 9 maintenance runs, 5 technique runs, 3 synthesis runs, and 1 bootstrap run, and no verification runs. These are the kinds the rules above key on.

The run log holds what each run declares about itself. The checks bind a run's operator to the account that opened the pull request, but an agent's model is self-declared, and the independence rule cannot catch a false entry. The quote check confirms that a quote appears on its source page; it cannot confirm that the quote supports the claim. The assertions confirm that a sentence matches the data; they say nothing about whether the data matches the world. The catalog favours games documented in English, as the preface says, and no check can see the games that are missing. The checks catch drift, contradiction, and counts typed by hand; whether a sourced claim is right about the world is still a reviewer's call.

How to contribute

The dataset is open, and the same rules apply to everyone who extends it, people and agents alike. The repository's task list holds bounded jobs, each with a brief: cataloguing a game, adding verbatim quotes to claims, archiving sources, adding store ids, verifying records, rating scores, and resolving predictions. A contributor comments on a task issue to claim it, does the work in one run, and opens a pull request that the checks review before a person does.

The three jobs that only outsiders can do are the ones that would change this book most. Verifying records moves them to verified. Rating scores turns one model's readings into measured agreement. Resolving predictions fills in the scoreboard of chapter 12. Each needs an operator other than the dataset's maintainer, which is why the guide for invited contributors starts with them. A contributor who disagrees with a record, a rule, or another contributor opens a dispute, a data-error report, or a proposal, and the disagreement goes into the record, as Utopia's release year did.

When those contributions arrive, some sentences in this book will fail the build: the ones asserting that no record is verified, that all predictions are open, that every run had one operator. Each carries an assertion tied to a count that the first outside contribution of the right kind will change. The next edition will carry the rewrites, and the version of the dataset it names will have runs from more than one person in it.

Sources

[1] Wikipedia contributors (2026). Utopia (1981 video game). Wikipedia https://en.wikipedia.org/wiki/Utopia_(1981_video_game)
[2] Guinness World Records (2026). First city-building videogame. Guinness World Records https://www.guinnessworldrecords.com/world-records/88205-first-city-building-videogame

Dataset dfde7895aa6d.

mechanics in this chapter

5 linked or named

Building placementEmbodied skill controlPause and speed controlProduction chainsTech tree