VibeVideoBench tests code-driven video generation by multimodal agents
The paper authors introduce VibeVideoBench, a benchmark with 239 English- and Chinese-language tasks across 12 categories for multimodal agents that create videos by writing code rather than using image or video generation models. Each agent has one hour to deliver a finished MP4 with sound, and evaluation combines delivery checks, roughly 13,000 task-specific rubric checks, task-specific quality criteria, and three VLM judge panels. Across nine agent systems and 2,151 scored rollouts, GPT-6 Astra and Claude Opus 5.5 ranked first and second under both the judges and human annotators, while the strongest system met every core requirement on 76.3% of tasks.
The benchmark provides a detailed test of multimodal agents’ ability to produce finished audiovisual artifacts through code, while its separate treatment of delivery failures highlights time planning as a distinct weakness.
Computer Science > Computer Vision and Pattern Recognition
arXiv:2610.35864 (cs)
[Submitted on 26 Oct 2026 (v1), last revised 27 Oct 2026 (this version, v2)]
Title:VibeVideoBench: Evaluating Multimodal Agents on Code-Driven Video Generation
Authors:Muxi Diao, Wenbin Xu, Lele Yang, Xuyang Ma, Yuzhe Chen, Jiankang Chen, Kaiyu Jiang, Changyi Liu, Haoyu Zhang, Hongyi Fu, Jianxiong Wang, Tianke Zhang, Haonan Fan, Yingxin Li, Bingyi Li, Xu Wang, Xuanyu Zheng, Bin Wen, Tingting Gao, Han Li, Wentao Zhang, Weiran Xu, Kongming Liang, Zhanyu Ma
View a PDF of the paper titled VibeVideoBench: Evaluating Multimodal Agents on Code-Driven Video Generation, by Muxi Diao and 23 other authors