The Bitter Lesson and the AI Factory
An opinion article argues that improving models can make elaborate workflows, domain rules, prompts, and harnesses less useful than simpler, model-led approaches. The author cites reported comparisons in which minimal instructions performed well on Zstandard tasks, test-driven development instructions worsened performance, and a near-bare mini-swe-agent matched or outperformed Claude Code and Codex CLI on DeepSWE. The author also claims that a concise prompt to Claude Code with Opus 5.5 can beat an “AI factory.” The supplied report does not provide the underlying benchmark data or independently verify these comparisons and broader conclusions.
The article highlights a practical debate among AI developers: whether to continually build layered, constrained workflows or adopt simpler approaches as models become more capable. The supplied evidence does not establish which approach is generally superior.
For every AI workflow we define, the next model makes it useless.
What's worse: most of us keep clinging to the outdated workflows for months after they've outlived their usefulness.
Remember how everyone was excited about AI code completion? How about prompt engineering? Harness engineering? Spec-driven development? Loop engineering? Software factory? Dark factory? They are all dead because of the bitter lesson. To summarize it: no matter what "domain knowledge", "harness", or "workflow" you try to integrate above your AI model, it will be outshone by just a larger, smarter model. This happened in chess when the program that beat Kasparov in 1997 won with massive, deep search, not with the human understanding of chess that most researchers had spent years encoding. And it keeps happening to us today.
That's why whenever somebody tries to build yet another factory or a really constraining skill, I roll my eyes since modern models perform worse with these tools. There are even benchmark runs showing that: Dan Luu tried 26 testing instructions and a few popular testing skills on agents implementing Zstd, and plain "no additional instructions" did well above average while the popular skills underperformed; Kun Chen measured that asking agents to do TDD makes them worse. On DeepSWE, near-bare mini-swe-agent did as well as or better than Claude Code and Codex CLI.
These days a naive and concise prompt into Claude Code with Opus 5.5 will perform significantly better than every AI factory because factories needlessly constrain Opus. A default Opus 5.5 can make much better decisions by going beyond the constraints and rules that factories commonly put on it since the written rules cannot foresee every situation and invariant.
https://zmievsa.org/posts/the-bitter-lesson-ate-your-ai-factory/
submitted by /u/Varabe [link] [comments]
#Claude Opus 5.5#Claude Code#Codex CLI#DeepSWE#mini-swe-agent#Zstandard