SWE-Game benchmark tests whether coding agents can build games
An arXiv paper introduces SWE-Game, a benchmark of 247 tasks grounded in 41 executable reference Godot games spanning 13 gameplay categories in 2D and 3D, with five task types: brief-to-game, implementation from a design document, skeleton completion, repair of 83 injected-fault cases, and Godot-to-Unity porting. Across six evaluated models, Opus5 scored highest overall in all five task types, yet best overall scores on the three construction tasks stayed below 60 out of 100, with Brief-to-Game at 50.38; the authors flag requirement omissions and gameplay logic errors as the predominant problems. The paper reports executable checks reaching 92.59% balanced accuracy on human-labeled behaviors from 100 agent-built games versus 78.41% for a video-based VLM judge, and rubric visual scores with a 0.829 Spearman correlation to human ratings.
Game development gives coding agents a long-horizon, automatically verifiable testbed, and this benchmark suggests current models still struggle to ship complete games that match the brief.
Computer Science > Artificial Intelligence
arXiv:2609.33678 (cs)
[Submitted on 27 Sep 2026 (v1), last revised 7 Oct 2026 (this version, v3)]
Title:SWE-Game: Can Coding Agents Build the Games We Want?
Authors:Xiaoyu Chen, Lai Wei, Jin Wang, Xiangyu Zou, Ruochen Fan, Enze Luo, Mingzhe Yao, Jiahui Zhu, Yuhua Wen, Linghe Kong, Weiran Huang
View a PDF of the paper titled SWE-Game: Can Coding Agents Build the Games We Want?, by Xiaoyu Chen and 10 other authors