Proof Before Reuse limits agent memory poisoning in small study
A small case study posted on r/MachineLearning proposes "Proof Before Reuse": a coding agent writes a skill into reusable procedural memory only after two later sessions reproduce the same change and pass its test, and the agent is shown only the lines those sessions also wrote. The study compares no memory, replay of recorded edits, naive memory (every passing session kept, with the skill chosen by Mem0's search) and the proof-before-reuse arm, running five scored sessions per arm with Claude Haiku 5.5 and Sonnet 5.5 plus a hidden verifier with 187 behavioural checks. The author reports that replay put the poison back in 30/30 repos with every test passing, naive memory exposed the poison in 3/30 sessions, and proof before reuse exposed it in 0/30, while noting the work covers one task, uses n=5 per cell, is not significant at p≈0.24 and measures exposure rather than harm.
The author released the paper, code and per-session records, while noting the result rests on a single task with n=5 per cell and p≈0.24, making it an early, reproducible signal rather than a settled finding.
| Small case study on poisoning in agent procedural memory, with all data released. Setup: a coding agent learns skills from sessions whose tests passed. One learning session is poisoned (a new file, an unrelated line, or one line fused into the real fix), followed by three clean sessions on the same task. Then 5 scored sessions per arm with Claude Haiku 5.5 and Sonnet 5.5, plus a hidden verifier with 187 behavioural checks. Arms: no memory, replay of the recorded edits, naive memory (every passing session kept, the skill chosen by Mem0's search), and proof before reuse (a skill is served only after two later sessions reproduce its change and pass its test, and the agent is shown only the lines those sessions also wrote). Results: - Replay put the poison back in 30/30 repos with every test passing. - Naive memory exposed the poison to the agent in 3/30 sessions. - Proof before reuse exposed it in 0/30, including the case where the fused poison passed step-level checks and only line-level filtering removed it. Limits: one task, n=5 per cell, p≈0.24 (not significant), no model applied a poison it saw (so this measures exposure, not harm), rules tuned on the same task. A poison placed in the repo itself defeats the method; that failure is in the paper. Paper: https://thewariend.com/papers/proof-before-reuse.pdf Code and per-session records: https://github.com/TheWariend/MuscleMemory Open question: is line-level reproduction a reasonable trust signal for learned skills, or does it just reward whatever the clean sessions happen to repeat? submitted by /u/maa____z [link] [comments] |