Ex-OpenAI researcher: neural networks are 'already clean'
Former OpenAI researcher Nick Cammarata and the Goodfire team published a long essay, "Neural Networks Are Already Clean," arguing that much of a network's internal computation is orderly rather than tangled. In a 32-billion-parameter language model, they separated positive and negative neuron responses and roughly estimated that about 90% of neurons each have one main task, with about 2% to 4% still unexplained; in one experiment, ablating 20 neurons labeled "Christmas" left a 76% probability for the model's "Santa" answer, while swapping in eight other key neurons cut it to 6%. The team also released 1,000 reports on randomly sampled neurons, and Cammarata estimated about 99% of the final layer is explainable in the more modern ConvNeXt architecture. In blind tests described in the essay, sentences written from a neuron's "function spec" alone triggered stronger responses than the best samples found across 10 billion tokens; across 183 positive explanations, target examples fired strongly 76% of the time versus 11% for distractors. The authors frame these as working hypotheses awaiting more testing, and envision agents running an automated "science lab" to push the work forward.
If model internals become traceable, researchers may spot risks that outputs alone do not reveal—though these findings remain working hypotheses pending wider testing.