Four frontier models each got $100 to build a PDF editor — nearly all shipped bugs
Researcher nielstron gave Gemini 3.8 Flash, GPT Astra 6, Opus 5 and Fable 5 $100 each (about $400 total) and asked them to autonomously build an open-source PDF editor with an excellent user experience. According to OSChina's report, bugs could be found within a few clicks in nearly every result. The author argues coding agents cannot interact with software the way humans do, which is a key reason their flaws surface so quickly.
Small comparative experiments like this suggest coding agents' weak point may lie as much in interacting with and verifying software as in generating code.
一个研究 LLM for Code 的研究者，拿 400 美元做了个实验：给 Gemini 3.8 Flash、GPT Astra 6、Opus 5、Fable 5 四个前沿模型各 $100 预算，让它们自主开发一款「用户体验极好」的开源 PDF 编辑器——结果是「点几下就能在几乎每个里找到 bug」。作者 nielstron 据此提出：coding agent 不能像人一样和软件交互，所以它们...
#Gemini 3.8 Flash#GPT Astra 6#Opus 5#Fable 5#Google#OpenAI#Anthropic