Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

CR-Bench Exposes Limits in AI Code Review and LLM Scoring

AI summary

Nutanix researchers' CR-Bench-v2 tested eight code-review agents across 584 real pull requests with known defects; the strongest agent detected only about one-third, while Gemini Code Assist recognized 8.1%. A separate replay study by Masoud Fetanat found that a strong LLM judge awarded high scores to invalid Python, prompting the recommendation to report syntax-pass rates alongside model-based scores.

In-depth insight

The findings caution teams evaluating or deploying AI code review to pair model judgments with deterministic checks, because the reported scores and defect-detection results may not reflect whether code runs or whether review agents identify real defects.

摘要:Nutanix 的新评测集让八个 AI 评审去认 584 个真实 PR 里埋的缺陷,最强的也只认出三分之一。而另一项重放研究发现,一个强评委能给 63% 跑不起来的代码打 93.95 分。

你让 agent 改完一段代码,再把 diff 丢回去让它自己审。它回你一句:整体没问题,只有一处命名可以更好。

你信了。三天后,那个它没点出来的空指针在凌晨把一批定时任务全带崩了。

回头翻 diff,那个缺陷明晃晃地躺在那里。于是你得出一个结论:这个评审 agent 不够聪明。

这个结论可能是错的。

这周有两份新数据,搁在一起看,指向的是另一件事——不是它不够聪明,是「审代码」这件事本身有个天花板,而你以为能衡量它的那把尺子,也在骗你。

#Nutanix#CR-Bench-v2#SWE-Bench Verified#Gemini Code Assist#Qwen-2.5-7B-Instruct

Original · 5 min readRead the original
Group chat

No comments yet. Start the conversation.