Old Ideas, Novel Problems
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 1
- Ai2 benchmark exposes brittleness in LLM research-idea novelty judging
Ai2 released the study “Old Ideas, Novel Problems,” along with a benchmark, data, code, and an automated pipeline for building novelty-judge evaluation sets. Across six LLM judges, small changes to prompts and evaluation setups produced sharply different verdicts; in one test, GPT-5.4 accuracy fell from 93.5% to 40.9% after a prompt change, while retrieval and additional reasoning did not consistently improve results. On one Human+Generated test set, a simple Claude Opus 4.6 setup achieved 0.854 macro-F1 at a reported cost of $2.59, outperforming two dedicated evaluators in the study’s reported comparison.
Ai2 Blog · 🔥 0
Experience and discussion from the community
Share my Old Ideas, Novel Problems experienceAsk about Old Ideas, Novel Problems
Nobody has shared their experience with Old Ideas, Novel Problems yet.