Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

SWE-CoWork evaluates coding agents in simulated software organizations

AI summary

An arXiv paper introduces SWE-CoWork, a benchmark with 101 development assignments drawn from 101 real repositories, 11 programming languages and 12 application domains. Each assignment places a coding agent in a simulated engineering organization with 26 colleagues, testing its ability to recover requirements, reconcile conflicting information, collaborate, implement changes and verify the work with private executable tests. The paper reports that, among 11 evaluated models, the two leading systems passed roughly 63% of fail-to-pass tests but completed fewer than 8% of assignments. These findings come from the paper and have not been independently verified here.

In-depth insight

The benchmark highlights sustained requirement tracking—from clarification through implementation, verification and handoff—as a key challenge for reliable autonomous software engineering. Its reported results also show a substantial gap between passing individual tests and delivering complete assignments.

Computer Science > Software Engineering

arXiv:2611.20218 (cs)

[Submitted on 15 Nov 2026]

Title:SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations

Authors:Xinping Lei, Minghao Liu, Ruijie Zhu, Xiaohui Hu, Xingqiao Li, Ao Zhang, Wenhao Chai, Han Li, Tao Wang, Jingyuan Zhang, Enduo Zhao, Yunfeng Shi, Yiming Liang, Zetao Yang, Xingjian Wang, Yifan Yao, Xinkai Ma, Yingru Li, Tianhan Yang, Huan Zhou, Weiran Shi, Shen Yan, Jiaheng Liu, Stephen Huang, Eli Cheung, Weihao Xuan

View a PDF of the paper titled SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations, by Xinping Lei and 25 other authors

#SWE-CoWork#Seed Evolving#GPT-6 Astra#Claude-Opus-5.5

Original · 3 min readRead the original
Group chat

No comments yet. Start the conversation.