Neel Nanda

Headlines · 2

1. 14h
   Interpretability essay 'Neural Networks Are Already Clean' divides researchers

   Goodfire researcher Nick Cammarata published a long essay, "Neural Networks Are Already Clean," arguing that the internals of large models are far more understandable than assumed. He reinterprets the famous InceptionV1 "cat-car" neuron as a V-shape detector and reads the superposition circuit from "Zoom In" as ordinary generalization, proposing "recursive generalization" to explain remaining ambiguity. He reports preliminary numbers for Qwen3-32B: about 90 percent of neuron poles have one main job, about 7 percent have two related ones and only 2 to 4 percent remain unexplained; Goodfire says it sees similar results in models up to trillions of parameters, with automated agents doing most of the work. OpenAI's Roon called it a potential major moment in scientific history, while Google DeepMind's Neel Nanda said it is a useful tool rather than a breakthrough and still believes superposition is real; the companion technical paper has not yet been released.

   The Decoder · 🔥 7
2. 16h
   Nick Cammarata says neural networks are 'already clean,' sparking debate

   Interpretability researcher Nick Cammarata has published a long essay, "Neural Networks Are Already Clean," retracting a decade-old conclusion he helped popularize, claiming that in modern vision models 99–100% of final-layer neurons have clear explanations. He reinterprets InceptionV1's classic 4e:55 "cat/car" neuron as detecting V-shaped versus circular geometry, attributing the earlier misreading to reward hacking in visualization tools and dataset bias; after compressing InceptionV1's layer 3b from 480 to 10 neurons (48x) and retraining, he reports the neurons stayed interpretable, while features produced by sparse autoencoders showed broader responses than native neurons. Google DeepMind's Neel Nanda argued the essay offers no substantial breakthrough, saying it refutes "arbitrary superposition" while "representational superposition" in language models still holds; Cammarata concedes it is a working hypothesis, that the work covers only unimodal vision and language models, and that on uncompressed layers SAE and native neuron shapes match 0.96–1.00.

   雷峰网 AI · 🔥 6

Experience and discussion from the community

Share my Neel Nanda experienceAsk about Neel Nanda