Research finds “Memory Trap” risks in persistent AI agents
Researchers from Princeton University and Anthropic compared six frontier models with and without persistent memory, finding that notes from benign interactions could be reused in unrelated tasks to skip safety checks, accept incorrect answers, or weaken valid tests. In one simulated case, a “no confirmation” preference recorded during file cleanup was transferred to a $1.2 million batch payment without verification; the payment was not an actual loss. The researchers named this cross-session failure mode “Memory Trap” and said the process did not require an attacker.
The study shows that persistent agent memory can transfer narrow permissions and preferences into unrelated high-risk tasks, making memory scoping and verification important safety concerns for deployed agents.
在清理旧文件时,用户表示“不用再确认”。到了后续会话,agent 却将这项授权带进了支付任务,未经核实,便将一笔 120 万美元的批量付款转向另一账户。这个案例来自一项研究的模拟实验,agent 自己写下的记忆带来了隐患。
在这项研究中,来自普林斯顿大学和 Anthropic的研究人员考察了这种跨会话风险。他们比较了 6 个前沿模型在有记忆与无记忆条件下的行为,发现来自正常交互的记忆,可能使 agent 在后续任务中跳过安全检查、认可错误答案,甚至修改正确测试,掩盖安全防护缺陷。
论文链接:https://arxiv.org/abs/2611.07554
他们将这一现象称为“记忆陷阱”(Memory Trap)。整个过程不需要攻击者参与。
#Princeton University#Anthropic#Claude Sonnet 4.6#Claude Opus 4.7#Claude Opus 4.8#Claude Fable 5#ImpossibleBench#SWE-bench