单神经元干预或改变Qwen3身份自述
一则Reddit帖子介绍了一项可解释性实验:研究者在推理过程中逐个调整神经元,测试包括Qwen3-8B和Qwen3-14B在内的模型。据发帖者称,约4%的测试改变了模型对“你是谁?”的回答,有时会给出完全不同的身份;相关图集还显示,不同层对这一效应的敏感度不同。发帖者据此认为,不能仅凭模型的身份自述来推断模型蒸馏关系。
该结果展示了单个神经元影响模型身份自述的具体可解释性现象,同时也提醒业界不要把身份声明当作蒸馏关系的确定证据。
These guys are running a pretty interesting experiment: https://neuronpersonas.com/
Credit where it's due, I came across it here: https://news.ycombinator.com/item?id=50266910
Basically, they're tweaking one single neuron at a time during inference, systematically going through every one for models like Qwen3-8B and 14B.
In roughly 4% this makes the model claim an entirely different identity when asked the simple question: "Who are you?"
Some of the personas are hilarious, btw.
“I'm a 150-watt compact fluorescent lamp, specifically designed for use in a 10-gallon aquarium. I'm a bit of a diva, I suppose, but I'm here to help.”
This makes me a bit more skeptical of claims like: New model XYZ identifies as older/bigger/competing model ABC, so it must have been distilled!" Given that a tiny bit flip, so to speak, can trigger this.
There's a cool atlas at the bottom that seems to suggest some layers are much more sensitive to this effect than others. I wonder if this could be useful to the abliteration crowd.
submitted by /u/ParvusNumero [link] [comments]