Scaling Instructable Agents Across Many Simulated Worlds

https://arxiv.org/abs/2404.10179

Agents take image observations and language instructions and output keyboard and mouse actions through a generic human-like interface in real time, across research environments and commercial games.

Generalist instructable game agents (SIMA)