Paper 2602.06855
AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents
01 In brief
Summary
AIRS-Bench is a benchmark suite of 20 tasks sourced from 17 state-of-the-art machine learning papers, designed to evaluate LLM agents on the full research lifecycle (idea generation, implementation, experimentation, analysis) without providing baseline code.
Tasks span seven categories: language modeling, math, code, molecules/proteins, time series, text classification, and text extraction/matching.
The benchmark uses a task configuration standard (metadata.yaml, project_description.md, prepare.py, evaluate.py) and supports multiple harnesses (AIRA-dojo, MLGym) and scaffolds (One-Shot, Greedy, ReAct).
Agents are evaluated on valid submission rate, normalized score (using a 'march of 9s' transform), and Elo rating.
Results from 14 agents (combinations of LLMs like CWM, GPT-4o, gpt-oss, o3-mini, Devstral with scaffolds) show that agents exceed human SOTA in only 4 tasks (e.g., TextualClassificationSickAccuracy, TextualSimilaritySickSpearmanCorrelation, CoreferenceResolutionWinograndeAccuracy, TimeSeriesForecastingRideshareMAE) but fail to match it in 16 others.
The benchmark is far from saturated, with an overall average normalized score of 24.1% and a valid submission rate of 55.1%.
The authors open-source the benchmark to catalyze development of AI research agents.
02 From the paper
Abstract
LLM agents hold significant promise for advancing scientific research. To accelerate this progress, we introduce AIRS-Bench (the AI Research Science Benchmark), a suite of 20 tasks sourced from state-of-the-art machine learning papers. These tasks span diverse domains, including language modeling, mathematics, bioinformatics, and time series forecasting. AIRS-Bench tasks assess agentic capabilities over the full research lifecycle -- including idea generation, experiment analysis and iterative refinement -- without providing baseline code. The AIRS-Bench task format is versatile, enabling easy integration of new tasks and rigorous comparison across different agentic frameworks. We establish baselines using frontier models paired with both sequential and parallel scaffolds. Our results show that agents exceed human SOTA in four tasks but fail to match it in sixteen others. Even when agents surpass human benchmarks, they do not reach the theoretical performance ceiling for the underlying tasks. These findings indicate that AIRS-Bench is far from saturated and offers substantial room for improvement. We open-source the AIRS-Bench task definitions and evaluation code to catalyze further development in autonomous scientific research.