创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

Redwood Research发布智能体监控模拟控制评估

AI 总结

Redwood Research发布论文,提出模拟控制评估(SCE),用于测试监控器能否在源自真实开发环境的条件下,发现追求有害目标的AI智能体。该方法不实际执行工具调用,而是由LLM模拟器生成可信输出,再由独立判定器估计攻击是否会成功。Redwood Research在LinuxArena上的测试显示,模拟与真实评估的差距往往接近两次真实运行之间的差异,而模拟LinuxArena的结果比BashArena等简单控制环境更能预测真实表现。

深度见解

SCE无需攻击实际系统,便可结合部署环境的具体背景评估智能体监控器,使控制评测更贴近真实部署。

We’ve released a paper on simulated control evaluations (SCEs). They’re a method for testing whether the monitors AI developers use to oversee their agents would catch a model deliberately trying to cause harm, under conditions much closer to real deployment than current tests allow.

Developers have started running monitors in their agentic deployments: models that flag or block an agent’s actions when they look dangerous. To know whether a monitor is good enough, you want to know how often it would stop a model that was actively trying to do something harmful, such as quietly inserting a security vulnerability, exfiltrating credentials, or sabotaging an experiment.

The standard way to measure this is a control evaluation: instruct a model to attempt such attacks, and measure how often the monitor catches it before an attack succeeds. Developers typically do this using purpose-built control settings like BashArena or LinuxArena, where attacks can actually be executed. However, these settings only provide indirect evidence for what we really care about: how well monitors will work in the real deployment, where opportunities for harm might depend on particular codebases, services and tasks that aren’t represented in control settings.

#Redwood Research#Simulated Control Evaluations#BashArena#LinuxArena#GPT-5.6 Sol

原文约 18 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。