Ai2 benchmark exposes brittleness in LLM research-idea novelty judging
Ai2 released the study “Old Ideas, Novel Problems,” along with a benchmark, data, code, and an automated pipeline for building novelty-judge evaluation sets. Across six LLM judges, small changes to prompts and evaluation setups produced sharply different verdicts; in one test, GPT-5.4 accuracy fell from 93.5% to 40.9% after a prompt change, while retrieval and additional reasoning did not consistently improve results. On one Human+Generated test set, a simple Claude Opus 4.6 setup achieved 0.854 macro-F1 at a reported cost of $2.59, outperforming two dedicated evaluators in the study’s reported comparison.
The release gives researchers reusable tools for testing a capability that matters in AI-assisted scientific discovery: determining whether proposed ideas are genuinely novel rather than repackaging prior work.
Can we trust LLMs to judge whether a research idea is novel?
October 27, 2026
Ai2
Today we’re releasing Old Ideas, Novel Problems, a new study of how reliably large language models can evaluate the novelty of scientific research ideas. Alongside the paper, we’re publishing a benchmark, data, and code for testing novelty judges, plus an automated pipeline researchers can use to build new evaluation sets.
#Ai2#Old Ideas, Novel Problems#GPT-5.1#GPT-5.2#GPT-5.4#Claude Sonnet 4.5#Claude Opus 4.5#Claude Opus 4.6#Idea Novelty Checker#Sakana AI#AI Scientist