WorkflowArena评测智能体团队的协调能力
据Einsia团队介绍,WorkflowArena是一项新基准,覆盖八个领域的145项长程任务。该基准记录智能体如何分工,并检查最终交付物是否满足任务要求。在作者报告的一项实验中,Opus 5.5和GPT-6 Astra针对四个执行模型,在相同的32项任务上编写工作流。与让执行模型自行编排相比,Opus 5.5的静态工作流使四个执行模型的连续得分平均提高6.41分;GPT-6 Astra的工作流则使平均分降低2.53分,尽管Astra在主要基准中的表现强于这四个执行模型。
Einsia团队的实验结果表明,仅依据模型自身的任务表现来选择协调者,可能无法判断其工作流是否真正有助于执行模型开展工作;这一问题与混合模型智能体团队直接相关。
Suppose you’re putting together a team of agents. Picking the strongest model as the coordinator seems reasonable. We wanted to test whether its workflows would actually help other models do the work.
In one experiment, we gave Opus 5.5 and GPT-6 Astra the same 32 challenging tasks and asked them to write workflows for four execution models. They could divide the work and instruct workers, but couldn’t edit the workspace themselves.
Opus’s static workflows improved all four executors, adding 6.41 points on average to their continuous scores compared with letting them orchestrate themselves. Astra’s static workflows lowered the average by 2.53 points, despite Astra being a stronger system than all four executors in the main benchmark.
So choosing a coordinator based on its own task performance can miss something consequential: whether the way it organizes work is useful to the models carrying it out.
This is part of WorkflowArena, our team’s new benchmark at Einsia. It covers 145 long-horizon tasks across eight domains. Agents choose their own division of work; we record the delegation and check whether the final deliverable meets the task’s requirements.
For anyone building mixed-model teams, I’d be interested in how you evaluate that choice today. Do you test the coordinator separately, or mostly judge the whole setup by its final output?
Project and workflow replays: https://lab.einsia.ai/workflowarena/home.html Paper: https://arxiv.org/abs/2611.12856 Code: https://github.com/Einsia/WorkflowArena
submitted by /u/OliviaYii [link] [comments]