Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Redwood Research proposes simulated control evaluations for AI agent monitors

AI summary

Redwood Research released a paper introducing simulated control evaluations (SCEs), which test whether monitors can detect an AI agent pursuing a harmful objective under conditions derived from real developer environments. Instead of executing tool calls, an LLM simulator generates plausible outputs and a separate judge estimates whether the attack would have succeeded. In tests conducted by Redwood Research on LinuxArena, the gap between simulated and real evaluations was often comparable to variation between two real runs, while simulated LinuxArena results predicted real LinuxArena performance better than simpler control settings such as BashArena did.

In-depth insight

SCEs offer a way to assess agent-monitor performance using deployment-specific context without attacking live systems, potentially making control evaluations more representative of actual deployments.

We’ve released a paper on simulated control evaluations (SCEs). They’re a method for testing whether the monitors AI developers use to oversee their agents would catch a model deliberately trying to cause harm, under conditions much closer to real deployment than current tests allow.

Developers have started running monitors in their agentic deployments: models that flag or block an agent’s actions when they look dangerous. To know whether a monitor is good enough, you want to know how often it would stop a model that was actively trying to do something harmful, such as quietly inserting a security vulnerability, exfiltrating credentials, or sabotaging an experiment.

The standard way to measure this is a control evaluation: instruct a model to attempt such attacks, and measure how often the monitor catches it before an attack succeeds. Developers typically do this using purpose-built control settings like BashArena or LinuxArena, where attacks can actually be executed. However, these settings only provide indirect evidence for what we really care about: how well monitors will work in the real deployment, where opportunities for harm might depend on particular codebases, services and tasks that aren’t represented in control settings.

#Redwood Research#Simulated Control Evaluations#BashArena#LinuxArena#GPT-5.6 Sol

Original · 18 min readRead the original
Group chat

No comments yet. Start the conversation.