A linear program reproduces Astra’s weight edit
Nathan Barry’s analysis of MIT CSAIL’s “Sparks of Weight-Space Programming” reports that GPT-6 Astra inferred a trained game and generated weights for a small network that outperformed networks trained with PPO, without gradients or a training loop. He also reports that editing 0.016% of Qwen3.5-0.8B’s weights raised MATH-500 accuracy from 7.9% to 48.1% and IFEval from 21.0% to 62.3%. Barry argues that Astra’s roughly eight-hour search for the loop-removal edit can be expressed as a linear program. He reports that a couple of LP solves produced the same effect with one-eighth as many changed weights, while his custom PDHG GPU solver reached within 0.2% of the optimum in about a minute.
Barry’s analysis provides a more compact way to reproduce the reported loop-removal edit, but it also exposes limits: the approach was tested only on removing loops, the per-token constraints had little effect, and choosing the layers without an agent remained unresolved.
Last week, MIT CSAIL published Sparks of Weight-Space Programming. Given the raw weights of a small network, GPT-6 Astra could tell which game it had been trained to play. Given only a game’s rules, it could write the weights of a small network from scratch and beat networks trained with PPO for 100 million steps. No gradients, no training loop, just reasoning about what the weights compute. They coined this weight-space programming.
In another experiment, they asked Astra to make Qwen3.5-0.8B better at math by editing its weights. Astra changed only 0.016% of the weights, and MATH-500 accuracy jumped from 7.9% to 48.1%.1
Looking at Astra’s edit, I realized most of its search was solving a linear program by trial and error. It spent about 8 hours and 28 forward passes choosing which weights to change and by how much. A couple of LP solves, about a minute each, do the same job with 8× fewer changed weights. In this post I’ll walk through the edit, why it’s an LP, and what that says about the fix itself.
#MIT CSAIL#GPT-6 Astra#Qwen3.5-0.8B#Gemma 3 270M#HiGHS#PDHG#PPO#MATH-500#IFEval#GSM8K