GPT-6 Astra Can Edit Model Weights, but the Result Is Not Superhuman
A Hacker News analysis of a paper reports that GPT-6 Astra could inspect and directly edit neural-network weights. The demonstrations included identifying games learned by a small network and editing a 0.8B Qwen model to stop looping after 32k tokens, allowing it to work on longer mathematical problems. The author notes that the game networks had labeled 8x8-grid inputs, making the first-layer weight patterns relatively easy to interpret. For Qwen, Astra used Python to compare looping and non-looping texts, identify weights associated with looping, trim them and test the patch. The author argues that competent human researchers could perform these tasks and describes direct weight editing as impressive but limited, comparing it with modifying compiled binaries rather than changing how models are normally developed.
The author says a superhuman ability to improve language models could contribute to recursive self-improvement, but cautions that these demonstrations do not establish a radically new research capability. Their view is that LLMs may assist AI research by improving training, not by magically tweaking model weights directly.
A recent paper argues that frontier LLMs can work with neural network weights directly. This seems superhuman, but it isn’t.
The paper shows that GPT-6 Astra can take a small neural network and interact with its weights directly, “well past what we’d expect from humans or conventional algorithms”. Specifically, Astra was able to identify what games a small network1 was trained to play just by looking at the weights, and even tweak the weights itself. The most impressive result is Astra significantly improving a 0.8B Qwen LLM’s ability on mathematics by natively editing its weights. The reason to be excited here is that “superhuman ability to improve language models” is precisely the kind of thing that could cause recursive self-improvement.
I think a smart human ML researcher could perform all of these tasks. Figuring out what game a model was trained to play based on the network sounds impressive until you consider three points:
- The input channels of the model directly listen to each 8x8 grid cell
- The input channels are labeled (e.g. “agent”, “moving objects”)
- It’s thus relatively trivial to calculate2 which parts of the grid the first layer of weights cares about and why