Keeping score

Cold open

On December 31, 2027, 9 predictions about AI in simulation games come due on the same day. The first says that a widely played simulation or strategy game will let you ask, in plain language, why something happened in your own game, and answer from your save. Somebody is going to have to check.

Reading it literally

The sentence on the card is the headline. The criteria underneath are what count, and they're narrower. The feature has to have first shipped after the 24th of October 2026, so a game that already had it doesn't count. The game needs at least ten thousand Steam reviews, counted on the resolve date, so a game that crosses the line a month late doesn't count. And the patch notes or store page have to describe free-text questions about causes, answered from the player's own save.

Say you find a strategy game that added an advisor you can type to. You read the details: it answers questions about the rules, from the manual. That fails the last clause, because nothing comes from your save. If nothing else qualifies by the date, the prediction is refuted, and the top-ranked bet in the dataset takes the loss.

Bets that can lose

A bet with no date can't lose, so every bet in the dataset carries one. They weren't always this strict. When an outside reviewer read the first version, the predictions said things like "a major studio" or "a popular game", leaned on store rankings, and had no start date, so a feature that already existed could have settled them. One was already true on the day it was written. The fix, and the reasons for it, are written down in the project's decision records, where anyone can read why the rules are what they are.

The fix set three rules for every prediction. A feature only counts if it first shipped after the dataset was published. Size is a stated number of Steam reviews, or a console release. And the evidence has to come from the developer: patch notes, a store page, documentation, a talk.

Two tests per bet

Each bet in the dataset carries two tests. The experiment tests the idea: what a small team would build, who would play it, and the number that counts as success. For the why button, that's a correct diagnosis forty percent faster with nearly every sentence backed by the log. Anyone with a build can run it.

The prediction tests the world: whether anyone ships the thing by a date. An idea can pass the experiment and still lose the prediction, because studios didn't build it in time. It can also win the prediction with a design the experiment would have rejected. The record keeps both, so a reader can tell which kind of wrong a bet turned out to be.

The calendar

All 42 are open right now. 9 come due by the end of 2027, 28 by the end of 2028, and the last on December 31, 2030. The 9 due first cover the why button, events generated against a schema, endless inspection shifts for a Papers, Please-style game, furniture generated from text, run histories written up automatically, local language models, automated balancing, a voice instructor in a flight or racing sim, and simulation games released as test beds for AI agents. Most of the early dates belong to bets rated buildable now. The late ones are the long shots from episode 9: a game drawn by a generative model, a mechanic invented by a search program.

Four more cards

The other cards due that day show what checking involves. One counts games: at least five, each with over a thousand reviews, using a language model that runs on your own machine with no internet connection. Each one needs two facts checked, the date and the offline part.

One excludes things. An AI voice instructor in a flight or racing sim counts only if the developer ships it; third-party apps don't count, however good they are, and the sim needs at least five thousand reviews or a console release.

One excludes a specific game. Generated furniture and scenery from text has to show up in a life or management sim other than inZOI, because inZOI did it before the window opened.

And one counts studios: two different studios have to describe in public using large-scale automated play to balance a game's economy. That one can fail because a studio does it and never says so, and the record will still say refuted.

Who is allowed to check

Resolving a prediction is a change to the dataset, like any other. The checker finds the evidence and adds each page that settles it as a source in the dataset, so anyone can open the patch note or store page that decided the bet. Then comes the resolution: confirmed, refuted, ambiguous, or withdrawn, with the date, the evidence, and a note saying which game met which clause.

The build enforces the timing. You can't call a bet refuted before its date, and you can't withdraw it after. And the checker has to be someone else. Every prediction was written by one agent, running one model, working for the dataset's maintainer. So a resolution has to come from a different person and, if an agent does the work, a different model. Whoever made the bet doesn't get to settle it, and neither does another copy of the model that wrote it.

A refuted prediction counts against the bet's argument as well as its timing. The rules ask that nobody rewrite the argument in the same change that records the loss, so the losing case stays readable next to the evidence.

Every time the dataset is built, a warning lists each prediction whose date has passed without a resolution, and it keeps appearing until someone checks.

What the scoreboard tests

Once results come in, the scoreboard also tests the rankings. If the scores behind the table mean anything, confirmations should gather near the top. A run of refutations among the explanation bets would mean this series was wrong about what studios would build, even if it was right about what players need.

It also tests the dataset's maturity scores. Every technique is rated research, prototype, shipped in a niche, or shipped as a norm. If a bet that depends on a research-stage technique comes true on time, that's evidence the scale was running behind the field.

Try one

You can do this yourself today. Pick a prediction that's more than a year out. Turn every clause into a yes-or-no question. Search for each answer, and write down where you found it. Decide what the outcome would be if the date were today. Then write the two-sentence note you'd attach, so that someone who never saw your search could check it. Most people find a clause the criteria leave open, and that's worth a proposal to sharpen the prediction before it comes due.

Close

Every number in this episode is computed from the dataset, so a version built after the first results will say something different. The last episode shows how that works, and why the sentence saying every prediction is still open will break the build the day one isn't.