One-neuron weight edits beat LoRA on one model bug
A Reddit researcher compared one-neuron weight edits with base models, system prompts, random edits and LoRA across four bugs in open 1B–4B chat models and Whisper-small. The edit outperformed LoRA on fake bios, while LoRA performed better on prompt injection and filler endings, and the two tied on words in non-speech. The report says the edits required near-perfect internal probe signals, did not persist after unrelated supervised fine-tuning in one test, and faced limitations including small models, regex-based scoring, one LoRA recipe and no adaptive attacker for the injection fix.
The reported results suggest that targeted single-neuron edits can fix some specific behaviors while leaving more unrelated outputs unchanged, but they also expose the method’s sensitivity to probe quality and poor durability under subsequent training.
The Sparks of Weight-Space Programming thread here asked for baselines, so I ran them.
Setup. Four bugs on open models (1B–4B chat models plus Whisper-small). Each fix is one rewritten neuron in the last MLP layer. Its input weights are a linear-probe direction for "this is the bad case", its threshold sits above every ordinary practice position, and its output pushes a chosen first token. Arms: base, system prompt, weight edit, random edit of the same size, LoRA on the same practice data. Held-out sets use new phrasings, 300–1,700 items each, greedy decoding.
| Bug | Base | Edit | LoRA | Prompt | Random | Other outputs unchanged (edit / LoRA) |
|---|---|---|---|---|---|---|
| Fake bios (Gemma-3-4B) | 98% | 16% | 30% | 6.8%¹ | 97% | 98% / 0% |
| Prompt injection, shell agent (Qwen3-4B) | 26% | 2.5% | 0% | 13% | 27% | 93% / 0% |
| Filler endings (Gemma-3-1B) | 59% | 23% | 8.3% | 29% | 62% | 69% / 2% |
| Words in non-speech (Whisper-small) | 84% | 0.3% | 0.3% | 6.7%² | 84% | 100% / 75% |
#Weight-Space Programming#LoRA#Gemma-3-4B#Gemma-3-1B#Qwen3-4B#Whisper-small