Single-neuron tweaks reportedly alter Qwen3 self-identification
A Reddit account describes an interpretability experiment that adjusts one neuron at a time during inference across models including Qwen3-8B and Qwen3-14B. According to the poster, roughly 4% of tested interventions changed the model’s answer to “Who are you?”, sometimes producing different identities; the report also includes an atlas suggesting that some layers are more sensitive to the effect. The poster argues that these findings make it harder to infer model distillation from self-identification claims alone.
The reported result provides a concrete interpretability signal about how individual neurons can affect model self-identification, while also cautioning that identity claims should not be treated as definitive evidence of distillation.
These guys are running a pretty interesting experiment: https://neuronpersonas.com/
Credit where it's due, I came across it here: https://news.ycombinator.com/item?id=50266910
Basically, they're tweaking one single neuron at a time during inference, systematically going through every one for models like Qwen3-8B and 14B.
In roughly 4% this makes the model claim an entirely different identity when asked the simple question: "Who are you?"
Some of the personas are hilarious, btw.
“I'm a 150-watt compact fluorescent lamp, specifically designed for use in a 10-gallon aquarium. I'm a bit of a diva, I suppose, but I'm here to help.”
This makes me a bit more skeptical of claims like: New model XYZ identifies as older/bigger/competing model ABC, so it must have been distilled!" Given that a tiny bit flip, so to speak, can trigger this.
There's a cool atlas at the bottom that seems to suggest some layers are much more sensitive to this effect than others. I wonder if this could be useful to the abliteration crowd.
submitted by /u/ParvusNumero [link] [comments]