Paper 2606.24530
NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?
- Published
- Jun 2026
- Research lab
- Independent
- Citations
- 1
- GitHub
- 89 stars
01 In brief
Summary
NatureBench is a benchmark of 90 tasks distilled from peer-reviewed Nature-family publications (2022–2025) to evaluate whether AI coding agents can move beyond reproduction toward scientific discovery.
It is built on NatureGym, an automated pipeline that converts papers into containerized task packages with an information firewall, hidden ground truth, and automated evaluator, addressing environment fragmentation.
Twelve frontier agents were evaluated under a web-search-disabled protocol.
The strongest, Claude Opus 4.7, surpassed published SOTA (g>0.1) on only 17.8% of tasks and matched it on 47.8%.
Analysis of 900 runs showed successes are driven mainly by methodological translation (45.5%) rather than scientific invention; failures are dominated by wrong method choice (45.1%) and insufficient compute budget (24.4%), not task misunderstanding.
Domain difficulty varied, with relational reasoning easiest (60% match) and biomedical modeling hardest (17.9%).
The benchmark uses a SOTA-normalized relative gap metric and a post-hoc validity judge to filter shortcuts.
The authors release NatureBench, NatureGym, and a public leaderboard.
02 From the paper
Abstract
We introduce NatureBench, a cross-discipline benchmark of 90 tasks distilled from peer-reviewed Nature-family publications, designed to evaluate whether AI coding agents can move beyond reproduction toward discovery on real scientific problems. NatureBench is built on NatureGym, an automated pipeline that constructs a standardized, per-task containerized environment from a source paper, addressing the environment-fragmentation problem that has limited the credibility of prior agent-on-research benchmarks. Evaluating ten frontier agent configurations under a strict web-search-disabled protocol, we find that the strongest model surpasses SOTA on only 17.8% of tasks under the g>0.1 criterion. Analysis of method pathways reveals that agents succeed primarily through methodological translation, converting scientific tasks into familiar supervised prediction problems, rather than through genuine scientific invention. Failures are dominated by wrong method choice and insufficient compute budget, not by task misunderstanding. We release the benchmark, the NatureGym pipeline, and a public leaderboard with maintainer-side reproduction. Code: https://github.com/FrontisAI/NatureBench