Paper 2604.07429
GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents
- Published
- Apr 2026
- Research lab
- Independent
- Citations
- 10
- GitHub
- 216 stars
01 In brief
Summary
GameWorld is a benchmark for evaluating multimodal large language models (MLLMs) as game agents in browser environments.
It includes 34 diverse games across five genres (Runner, Arcade, Platformer, Puzzle, Simulation) and 170 tasks, each with outcome-based, state-verifiable metrics computed from serialized gameAPI state.
The benchmark supports two agent interfaces: Computer-Use Agents (CUAs) that emit raw keyboard/mouse controls, and Generalist Multimodal Agents that use deterministic Semantic Action Parsing.
A browser-based sandbox pauses game execution during model inference to decouple decision quality from response speed.
Evaluation of 18 model-interface pairs shows that even the best agents (e.g., Gemini-3-Flash-Preview with 41.9% overall progress) remain far below human performance (Novice: 64.1% PG, Expert: 82.6% PG).
Repeated full-benchmark reruns demonstrate stability, with low standard deviations in overall progress.
Additional analyses include a real-time variant (GameWorld-RT), context-memory sensitivity, and action validity, revealing challenges in timing grounding, spatial navigation, and long-horizon coordination.
GameWorld provides a standardized, verifiable, and reproducible framework for advancing game agent research.
02 From the paper
Abstract
Towards an embodied generalist for real-world interaction, Multimodal Large Language Model (MLLM) agents still suffer from challenging latency, sparse feedback, and irreversible mistakes. Video games offer an ideal testbed with rich visual observations and closed-loop interaction, demanding fine-grained perception, long-horizon planning, and precise control. However, systematically evaluating these capabilities is currently hindered by heterogeneous action interfaces and heuristic verification. To this end, we introduce GameWorld, a benchmark designed for standardized and verifiable evaluation of MLLMs as generalist game agents in browser environments. Two game agent interfaces are studied: (i) computer-use agents that directly emit keyboard and mouse controls, and (ii) generalist multimodal agents that act in a semantic action space via deterministic Semantic Action Parsing. GameWorld contains 34 diverse games and 170 tasks, each paired with state-verifiable metrics for outcome-based evaluation. The results across 18 model-interface pairs suggest that even the best performing agent is far from achieving human capabilities on video games. Extensive experiments of repeated full-benchmark reruns demonstrate the robustness of the benchmark, while further studies on real-time interaction, context-memory sensitivity, and action validity expose more challenges ahead for game agents. Together, by offering a standardized, verifiable, and reproducible evaluation framework, GameWorld lays a robust foundation for advancing research on multimodal game agents and beyond. The project page is at https://gameworld-bench.github.io.