CR-Bench Exposes Limits in AI Code Review and LLM Scoring
Nutanix researchers' CR-Bench-v2 tested eight code-review agents across 584 real pull requests with known defects; the strongest agent detected only about one-third, while Gemini Code Assist recognized 8.1%. A separate replay study by Masoud Fetanat found that a strong LLM judge awarded high scores to invalid Python, prompting the recommendation to report syntax-pass rates alongside model-based scores.
The findings caution teams evaluating or deploying AI code review to pair model judgments with deterministic checks, because the reported scores and defect-detection results may not reflect whether code runs or whether review agents identify real defects.
摘要:Nutanix 的新评测集让八个 AI 评审去认 584 个真实 PR 里埋的缺陷,最强的也只认出三分之一。而另一项重放研究发现,一个强评委能给 63% 跑不起来的代码打 93.95 分。
你让 agent 改完一段代码,再把 diff 丢回去让它自己审。它回你一句:整体没问题,只有一处命名可以更好。
你信了。三天后,那个它没点出来的空指针在凌晨把一批定时任务全带崩了。
回头翻 diff,那个缺陷明晃晃地躺在那里。于是你得出一个结论:这个评审 agent 不够聪明。
这个结论可能是错的。
这周有两份新数据,搁在一起看,指向的是另一件事——不是它不够聪明,是「审代码」这件事本身有个天花板,而你以为能衡量它的那把尺子,也在骗你。
#Nutanix#CR-Bench-v2#SWE-Bench Verified#Gemini Code Assist#Qwen-2.5-7B-Instruct