Coding-Agent Harness Evolution Can Trigger Catastrophic Forgetting
A paper by Shangziqi Zhao, Yitong Zhang, and Jia Li studies continual harness self-evolution across three software-engineering datasets, six evolution directions, and two harness-evolution methods. The authors report that later evolution can substantially reduce performance on earlier tasks: with DeepSeek-V4-Flash-0731, Meta-Harness reportedly fell from 94.0% on SWE-Bench after evolution to 32.0% by the fifth round of subsequent FeatureBench evolution. The paper also presents insights intended to guide continual harness self-evolution.
The study highlights a reliability problem for coding agents whose surrounding harness evolves across multiple task types, while the reported 66% relative performance decrease shows that gains on a new dataset may coincide with severe forgetting of previous capabilities.
Computer Science > Software Engineering
arXiv:2611.03219 (cs)
[Submitted on 2 Nov 2026]
Title:Can Coding Agents Sustain Continual Evolution without Forgetting?
Authors:Shangziqi Zhao, Yitong Zhang, Jia Li
View a PDF of the paper titled Can Coding Agents Sustain Continual Evolution without Forgetting?, by Shangziqi Zhao and 2 other authors