Nick Cammarata says neural networks are 'already clean,' sparking debate
Interpretability researcher Nick Cammarata has published a long essay, "Neural Networks Are Already Clean," retracting a decade-old conclusion he helped popularize, claiming that in modern vision models 99–100% of final-layer neurons have clear explanations. He reinterprets InceptionV1's classic 4e:55 "cat/car" neuron as detecting V-shaped versus circular geometry, attributing the earlier misreading to reward hacking in visualization tools and dataset bias; after compressing InceptionV1's layer 3b from 480 to 10 neurons (48x) and retraining, he reports the neurons stayed interpretable, while features produced by sparse autoencoders showed broader responses than native neurons. Google DeepMind's Neel Nanda argued the essay offers no substantial breakthrough, saying it refutes "arbitrary superposition" while "representational superposition" in language models still holds; Cammarata concedes it is a working hypothesis, that the work covers only unimodal vision and language models, and that on uncompressed layers SAE and native neuron shapes match 0.96–1.00.
If native networks are indeed more readable than assumed, interpretability and alignment research may need to rethink its tooling priorities — though the claim remains a contested working hypothesis.