MALT
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 1
- AI auditors detect agent misconduct, but high detection targets bring false alarms
Andy Hall’s pilot study replicated and extended METR’s work by testing whether AI models can identify reward hacking and sandbagging in agent logs. At a 5% false-alarm rate, GPT-5 detected nearly 98% of reward-hacking cases, while the auditors caught 71% to 81% of sandbagging cases. The trade-off became more pronounced at demanding thresholds: in a second test using the R-Judge dataset, nearly all models stayed below 10% false alarms when targeting 95% detection, but the best model’s false-alarm rate rose to 15% at 99% detection. Hall concluded that AI auditors are better suited to estimating incident frequency than to reliably catching every incident, while noting that the analysis reflects his personal views. The reward-hacking results rest on only 47 cases, and most sandbagging cases were synthetic.
The Decoder · 🔥 0
Experience and discussion from the community
Share my MALT experienceAsk about MALT
Nobody has shared their experience with MALT yet.