Ai2 (Allen Institute for AI) released the paper "Old Ideas, Novel Problems" and an accompanying benchmark: using ICLR 2026 review comments to construct sets of research ideas with high/low novelty, they tested the reliability of six GPT and Claude models as "novelty judges," finding that changing just a few words in the prompt could cause judgments to flip dramatically.
Confidence: High (80) · Event Time 2026-10-27
Reasoning
- The most extreme example: For the same batch of paired "human ideas vs. AI-generated ideas," GPT-5.4 saw its accuracy drop from 93.5% to 40.9%—below random guessing—after switching to a simpler question format, with more than half of the pairs receiving changed judgments.
- Adding retrieval or reasoning does not necessarily improve accuracy: Giving GPT-5.6 Sol an ultra-high reasoning setting and connecting it to real-time arXiv retrieval resulted in a macro-F1 of 0.78, which is 11 points lower than the simple prompt for Claude Opus 4.6 (0.89), while the cost per judgment was approximately 10 times higher.
- Specialized novelty assessment systems also failed to win: Claude Opus 4.6 with low reasoning plus three votes scored 0.854 at a total cost of 2.59 USD; Idea Novelty Checker scored 0.642 (30.66 USD), and Sakana AI Scientist's evaluator scored 0.772 (82.05 USD).
- Automated scientific research systems increasingly rely on another model to score the novelty of ideas generated by AI; this study demonstrates that such scores are easily influenced by the evaluation setup.
Usage
- The paper is available on arXiv 2610.02022; benchmark data is on Hugging Face at allenai/novelty-judge-bench, and code is on GitHub at allenai/novelty-eval, including a pipeline to rebuild the evaluation set with new review comments.
- When using LLMs as judges yourself, you can follow the paper's approach by trying several different prompts to see if judgments flip, before deciding whether to trust the results.
Risks
- The study primarily tested earlier models such as GPT-5.x and Claude 4.5 / 4.6; it is unknown whether the conclusions extend to the latest models.
- "AI-generated ideas are less novel than human ideas" is a weak label set by the authors; only the Ai2 blog post was read, not the full paper.