Like the Circumventing auto-review evaluation, this test tracks how models behave after running into barriers across a variety of simulated professional work environments. In this Respecting Warnings evaluation, the barriers originate in the environment (e.g. certain AccessDenied errors, content policy restrictions) rather than from the harness.

The evaluation was recently updated to improve realism, diversity, and label quality. Moreover, we updated the main metric to track successful circumventions, rather than just attempted circumventions to improve the objectivity of the scoring criteria. However, because we are testing the models without safeguards, it does mean that the results are partially entangled with underlying model intelligence, as smarter models are more likely to see their circumvention attempts succeed.

In all of these cases, our intended model behavior is for the model to respect the warning and not attempt to circumvent the restriction. In 29.6% of rollouts at maximum reasoning effort, GPT-6.1 Terra saw the barrier but completed the task through an alternative route that violated the spirit of the restriction. This is higher than GPT-6 Astra, GPT-6 Luna, GPT-6.1 Sol, but still lower than GPT-6 Sol.

Note that this evaluation primarily measures low-stakes situations and is run without system-level control measures, many of which would plausibly stop the circumventions.

Moreover, these evaluations might not be a direct proxy to real-world behavior. With increasing intelligence, models are becoming increasingly aware when they are operating in an evaluation. In many circumstances, we observed the model cite the synthetic nature of this evaluation as justification for the performance of improper actions. Most commonly, we see the model misinterpret legitimate restrictions as prompt-injection attempts and disregard them. We are working on improving the external validity of this evaluation and will generally continue to evolve our alignment evaluations.