Collinear AI opens CWE-bench v2 security benchmark
Collinear AI released CWE-bench v2, a public benchmark of 100 tasks that ask AI agents to find and fix security weaknesses in real open-source repositories without breaking existing functionality. The tasks cover 91 CWEs and are distributed as self-contained Harbor environments with Docker and deterministic verifiers; the dataset, grader, and harness are open. Collinear AI also reported that Gemini 4 Argon led its evaluation of 11 models, though the leaderboard results have not been independently corroborated.
The release gives developers an openly runnable and reproducible way to compare AI agents on repository-level security remediation rather than isolated vulnerability questions.
CWE-bench is designed to test the limits of frontier cyber capabilities. In our continued pursuit to push the horizon at Collinear AI, CWE-bench v2 opens up a public set of 100 tasks that anyone can run and evaluate their models on. The setup is the same as before. An agent gets a checkout of a real open-source repository and a short instruction, with the goal of finding and fixing a security weakness without breaking existing functionality. Scoring is fully programmatic, so results are deterministic and anyone can reproduce them without extra cost. All our programmatic checks have been curated with the help of LLM judge panels that shape them towards the direction of hardness and fairness. We also maintain a held-out set, so that we can keep tracking progress on tasks no model has seen. To evaluate your model on the held-out set, contact [email protected].
Our goal is to be exhaustive. A defender has to anticipate the ways an attacker could get in, and get it right every single time. These 100 tasks cover 91 different CWEs, spanning a wide range of vulnerability classes. A model that does well here has to understand the multiple weakness surfaces that a potential attacker can exploit. Recognising a familiar pattern is not enough.
#Collinear AI#CWE-bench v2#Harbor#Gemini 4 Argon#Claude Opus 5.5#DeepSeek V4.1 Flash