Paper 2606.17861
GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?
- Published
- Jun 2026
- Research lab
- Independent
- Citations
- 1
- GitHub
- 191 stars
01 In brief
Summary
GameCraft-Bench is a benchmark for evaluating coding agents on end-to-end game generation in the Godot engine.
It formalizes the task as transforming natural-language specifications into complete, playable game projects, requiring three desiderata: Engine Grounding, Artifact Completeness, and Interactive Verification.
The benchmark includes 140 tasks across 15 game families, with evaluation via replayed interaction traces and rubric-guided multimodal judging.
Results show frontier agents perform poorly, with the best (Opus-4.7) scoring only 41.46% overall.
Agents often produce recognizable mechanics but fail to deliver complete games with sufficient content, visual feedback, and presentation.
Analysis reveals that visual feedback aids debugging, but tool usage volume does not correlate with quality.
The judge is stable and slightly more permissive than humans.
Game generation ability is partially factorized, with mechanics, content, visuals, and presentation not fully coupled.
02 From the paper
Abstract
Game generation is an emerging application of coding agents, requiring models to transform natural-language specifications into playable interactive systems. Unlike traditional coding tasks, game generation takes place within a game engine, where scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay. We formalize end-to-end game generation as the problem of producing a complete game artifact that realizes a specification through observable player-game interaction in a target environment. We argue that evaluating this setting requires three desiderata: Engine Grounding, Artifact Completeness, and Interactive Verification. We propose an interaction-grounded evaluation framework that assesses executable gameplay through replayed demonstrations and rubric-guided multimodal judging. We instantiate this framework as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families. Evaluations of frontier coding agents show that end-to-end game generation remains highly challenging: the strongest agent achieves only 41.46%, and most agents score below 40%. Further analysis reveals that while agents often implement recognizable mechanics, they struggle to deliver complete games with sufficient content, functional visual feedback, and coherent presentation. See https://tongxuluo.github.io/gamecraft-bench-website for demos, code, and data.