Local quantized models vs Claude on nine real CUDA-engine bug fixes
A Reddit author ran nine real regression bugs from the private history of their C++/CUDA inference engine (capy-flash) through coding agents, with Claude models as fixed reference points. Local quantized models ran on the author's own engine on 2x RTX 5070 Ti 16GB, a Ryzen 9900X and 96GB DDR5, while Claude models ran in Claude Code and local models in OpenCode. Patches were scored automatically against a hidden key from the real fix and by Claude Opus 5 as a blind judge, which correlated at r = 0.79; Claude Opus 5.5 led at 0.74, Sonnet 5.5 scored 0.58, Qwen3.8-Flash-Next 3-bit scored 0.48, and Gemma 4 26B-A4B scored 0.07. The author reports that 3-bit quantization left scores largely unchanged (Flash-Next -0.02, 27B +0.05, inside the noise) while making the 27B generate about 40% more tokens per task.
The evaluation argues that quantization cost shows up as longer reasoning rather than lower patch scores, a distinction that pure tok/s comparisons miss, though the author cautions that nine tasks from one codebase, different harnesses per model family and roughly ±0.2 confidence intervals mean the ordering is not a general ranking.
#Claude Opus 5.5#Claude Sonnet 5.5#Claude Haiku 5.5#Claude Sonnet 5#Claude Sonnet 4.5#Claude Haiku 4.5#Qwen3.8-Flash-Next#Qwen3.8-27B#Gemma 4 26B-A4B#capy-flash#OpenCode#Claude Code