Qwen3.5-4B单神经元引导实用指南
一名社区作者发布指南,介绍如何寻找并操纵Qwen3.5-4B中的单个神经元,内容涵盖PyTorch钩子、固定数值“剂量”、神经元地址,以及随机探测、基于梯度的候选筛选与LLM评审、激活挖掘三种方法。作者称Qwen3.5-4B共有32层,每层含9,216个MLP神经元,并表示修改某些神经元可显著改变模型行为。该指南借鉴了justinrhowell的神经元引导仓库,但未直接使用其代码。作者指出,部分方法和结果仍处于个人探索阶段,报告中没有独立验证。
该指南为开展单神经元激活引导实验提供了具体起点,并给出实现细节以及判断观察到的效果是否源自特定神经元的检查方法。
I made a couple of posts about weird neurons inside Qwen that I'd found and lots of people were asking how it was done so I thought it would be useful for people to have a guide. Generally this was inspired by justinrhowell's neuron-steering repo, although I didn't use his code directly I took a couple of the approaches so props to him for the work!
What is neuron steering?
Each layer of Qwen has an MLP block with thousands of neurons. Qwen3.5-4B has 32 layers with 9,216 neurons each, so roughly 295,000 in total.
Neuron steering is basically choosing one neuron and adding a fixed amount to it (the 'dose') when asking for an output from the model. This is not finetuning, or fancy prompting - it's basically just a really simple form of activation steering that involves one neuron only. But interestingly, some neurons change the model’s behaviour a lot!
To keep track of what neurons you're editing, they have addresses. “L11 N8147” means the 12th layer, neuron 8147 - this is the naming convention justin used, and is basically the same as the neuronpedia address (QWEN3.5-4B · MLP Neurons - Layer 11 · 8147 | Neuronpedia).
#Qwen3.5-4B#Qwen#PyTorch#llama.cpp#Neuronpedia#transformers#justinrhowell neuron-steering repository