sims-as-agent-benchmarks opportunity asserted
Simulation games as agent benchmarks
Use factory, colony, and economy sims as long-horizon evaluation and training environments for AI agents, with game studios as environment providers.
thesis
AI agents now need environments that test planning over thousands of steps, resource management, and recovery from compounding errors. Simulation games are exactly that, with measurable outcomes (throughput, survival, profit). Minecraft became a research environment through MineDojo and Voyager, and factory games have been proposed as agent benchmarks. Games with headless modes and clean observation APIs can become a licensed research category.
design sketch
A studio ships a headless build with a typed observation and action API, standard scenarios, and leaderboards for agents. Researchers measure long-horizon performance; players watch agent runs as content.
validation experiment
Wrap an open-source colony or factory sim with a typed API, define five scenarios with throughput and survival metrics, and evaluate three frontier agents. Success: scores that separate agents and do not saturate within one model generation.
key risks
- Benchmarks saturate quickly.
- Studios may not want agent play to define their game publicly.
- Engine licensing for research use.
falsifiable prediction
A simulation game developer or publisher will officially release a headless research API or benchmark for AI agents.
Released after 2026-10-24 by the developer or publisher of a title with at least 1,000 Steam user reviews, as an official announcement or repository; community wrappers and academic forks do not count.