Study tests small open-weight models on real coding-agent work
A retrospective empirical systems paper submitted to arXiv examines whether small open-weight models can complete real coding work rather than merely generate coherent text. Based on 4,165 gateway turns, 623 debug jobs, and three delegation studies, it reports major differences in tool-call formatting, session continuity, artifact validation, and verification across models and runtimes. The author explicitly says the studies were not pooled and do not rank models or runtimes.
The paper offers concrete evidence that reliable coding-agent performance depends on serving, protocol, continuity, and independent artifact checks—not just whether a model produces coherent text or executes a tool call.
Computer Science > Software Engineering
arXiv:2610.34241 (cs)
[Submitted on 25 Oct 2026]
Title:Beyond Benchmarks: Can Small Open-Weight Models Get Real Coding Work Done?
Authors:Vaibhav Jain
View a PDF of the paper titled Beyond Benchmarks: Can Small Open-Weight Models Get Real Coding Work Done?, by Vaibhav Jain
#Qwen3-Coder 30B-A3B#Claude Opus 5.5#Nemotron Cascade 2#vLLM#SGLang#MLX