GPT-6 Astra 能编辑模型权重，但结果并非超人级
Hacker News 对一篇论文的分析称，GPT-6 Astra 能够检查并直接编辑神经网络权重。演示包括识别一个小型网络学会的游戏，以及编辑一个 0.8B 的 Qwen 模型，使其不再在 32k tokens 后陷入循环，从而处理更长篇幅的数学问题。作者指出，游戏网络使用带有标签的 8x8 网格输入，因此第一层权重的模式相对容易解读。在 Qwen 的案例中，Astra 使用 Python 比较陷入循环与未陷入循环的文本，找出与循环相关的权重，缩减这些权重并测试补丁。作者认为，有能力的人类研究人员也能完成这些任务，并将直接编辑权重描述为令人印象深刻但能力有限，类比为修改编译后的二进制文件，而非改变模型通常的开发方式。
作者称，改进语言模型的超人级能力可能推动递归式自我改进，但也警告这些演示并未证明一种全新的研究能力。其观点是，LLM 可能通过改进训练来辅助 AI 研究，而不是直接“神奇地”修改模型权重。
A recent paper argues that frontier LLMs can work with neural network weights directly. This seems superhuman, but it isn’t.
The paper shows that GPT-6 Astra can take a small neural network and interact with its weights directly, “well past what we’d expect from humans or conventional algorithms”. Specifically, Astra was able to identify what games a small network1 was trained to play just by looking at the weights, and even tweak the weights itself. The most impressive result is Astra significantly improving a 0.8B Qwen LLM’s ability on mathematics by natively editing its weights. The reason to be excited here is that “superhuman ability to improve language models” is precisely the kind of thing that could cause recursive self-improvement.
I think a smart human ML researcher could perform all of these tasks. Figuring out what game a model was trained to play based on the network sounds impressive until you consider three points:
- The input channels of the model directly listen to each 8x8 grid cell
- The input channels are labeled (e.g. “agent”, “moving objects”)
- It’s thus relatively trivial to calculate2 which parts of the grid the first layer of weights cares about and why