Can Language Models Serve as Text-Based World Simulators?

https://arxiv.org/abs/2406.06485

ByteSized32-State-Prediction: 76,369 transitions from 31 text games. GPT-4 whole-state accuracy at most 59.9%; action-driven up to 77.1%; environment-driven at most 49.7%; humans about 80% on a subset where GPT-4 averaged about 50%. Failures in arithmetic, common sense, scientific knowledge.

LLM explanation of simulation state, LLM-adjudicated state transitions (LLM as game mechanic)