OpenAI researchers extend monitorability evaluation with paired metrics
Melody Guan, Yuxiang Xie and Bowen Baker published a research post introducing complementary metrics for evaluating AI monitors, including pair-matched comparisons, arm-specific scores and causal-recall bounds for intervention tests. The post distinguishes binary-monitor g-mean² from AUROC and applies the new framework to historical system-card data, including an example using GPT-5.6 Sol and GPT-6 Astra.
The new measures address whether monitorability comparisons are distorted by different task mixes, intervention-arm shortcuts or uncertainty over which outcomes the intervention actually caused.
Beyond g-mean²: complementary metrics for monitorability
Beyond g-mean²: complementary metrics for monitorability
Oct 20, 2026 · Melody Guan, Yuxiang Xie, Bowen Baker
In Monitoring Monitorability, we introduced g-mean² as a principled dataset-level measure of monitor discrimination.
Recap of g-mean²
The precise definition of g-mean² depends on the evaluation archetype.