Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

One-neuron weight edits beat LoRA on one model bug

AI summary

A Reddit researcher compared one-neuron weight edits with base models, system prompts, random edits and LoRA across four bugs in open 1B–4B chat models and Whisper-small. The edit outperformed LoRA on fake bios, while LoRA performed better on prompt injection and filler endings, and the two tied on words in non-speech. The report says the edits required near-perfect internal probe signals, did not persist after unrelated supervised fine-tuning in one test, and faced limitations including small models, regex-based scoring, one LoRA recipe and no adaptive attacker for the injection fix.

In-depth insight

The reported results suggest that targeted single-neuron edits can fix some specific behaviors while leaving more unrelated outputs unchanged, but they also expose the method’s sensitivity to probe quality and poor durability under subsequent training.

The Sparks of Weight-Space Programming thread here asked for baselines, so I ran them.

Setup. Four bugs on open models (1B–4B chat models plus Whisper-small). Each fix is one rewritten neuron in the last MLP layer. Its input weights are a linear-probe direction for "this is the bad case", its threshold sits above every ordinary practice position, and its output pushes a chosen first token. Arms: base, system prompt, weight edit, random edit of the same size, LoRA on the same practice data. Held-out sets use new phrasings, 300–1,700 items each, greedy decoding.

BugBaseEditLoRAPromptRandomOther outputs unchanged (edit / LoRA)
Fake bios (Gemma-3-4B)98%16%30%6.8%¹97%98% / 0%
Prompt injection, shell agent (Qwen3-4B)26%2.5%0%13%27%93% / 0%
Filler endings (Gemma-3-1B)59%23%8.3%29%62%69% / 2%
Words in non-speech (Whisper-small)84%0.3%0.3%6.7%²84%100% / 75%

#Weight-Space Programming#LoRA#Gemma-3-4B#Gemma-3-1B#Qwen3-4B#Whisper-small

Original · 2 min readRead the original
Group chat

No comments yet. Start the conversation.