先复现后复用：智能体记忆投毒暴露降至0/30
AI 总结
r/MachineLearning 上发布的一项小规模案例研究提出「先复现后复用」（Proof Before Reuse）方法：编码智能体只有在后续两次会话复现同一改动并通过测试后，才会把该技能写入可复用的过程记忆，而且只向智能体展示这些会话同样写过的代码行。研究对比了无记忆、回放记录的编辑、朴素记忆（保留所有通过测试的会话，由 Mem0 搜索挑选技能）与该方案，使用 Claude Haiku 5.5 和 Sonnet 5.5 每臂做 5 次计分会话，并配有含 187 项行为检查的隐藏验证器。作者称回放在 30/30 个仓库中重新放回投毒且测试全部通过，朴素记忆在 3/30 次会话中把投毒暴露给智能体，而该方法为 0/30；作者同时说明这只是单一任务、每格 n=5、p≈0.24 不显著，且测量的是暴露而非实际危害。
深度见解
作者公开了论文、代码与逐会话记录，但自述该结果仅基于单一任务、每格 5 次会话且 p≈0.24 不显著，因此更像一个可复现的初步信号而非定论。
| Small case study on poisoning in agent procedural memory, with all data released. Setup: a coding agent learns skills from sessions whose tests passed. One learning session is poisoned (a new file, an unrelated line, or one line fused into the real fix), followed by three clean sessions on the same task. Then 5 scored sessions per arm with Claude Haiku 5.5 and Sonnet 5.5, plus a hidden verifier with 187 behavioural checks. Arms: no memory, replay of the recorded edits, naive memory (every passing session kept, the skill chosen by Mem0's search), and proof before reuse (a skill is served only after two later sessions reproduce its change and pass its test, and the agent is shown only the lines those sessions also wrote). Results: - Replay put the poison back in 30/30 repos with every test passing. - Naive memory exposed the poison to the agent in 3/30 sessions. - Proof before reuse exposed it in 0/30, including the case where the fused poison passed step-level checks and only line-level filtering removed it. Limits: one task, n=5 per cell, p≈0.24 (not significant), no model applied a poison it saw (so this measures exposure, not harm), rules tuned on the same task. A poison placed in the repo itself defeats the method; that failure is in the paper. Paper: https://thewariend.com/papers/proof-before-reuse.pdf Code and per-session records: https://github.com/TheWariend/MuscleMemory Open question: is line-level reproduction a reasonable trust signal for learned skills, or does it just reward whatever the clean sessions happen to repeat? submitted by /u/maa____z [link] [comments] |