The Year in AI Papers
The year/Meta AI

Paper 2511.15593

What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversity

Alexis Audran-Reiss, Jordi Armengol-Estapé, Karen Hambardzumyan, Amar Budhiraja, Martin Josifoski, Edan Toledo, Rishi Hazra, Despoina Magka, Michael Shvartsman, Parth Pathak, Justine T Kao, Lucia Cipolina-Kun, et al.

Published
Nov 2025
Research lab
Meta AI
Citations
5
GitHub
Not linked

01 In brief

Summary

This paper investigates the role of ideation diversity in AI research agents' performance on MLE-bench, a benchmark of 75 Kaggle machine learning tasks.

The authors analyzed 11,000 agent trajectories from 6 LLM backbones and 3 scaffolds, measuring ideation diversity via Shannon entropy of model architectures in initial drafts.

They found that higher-performing agents (e.g., o3, GPT-OSS) exhibit greater diversity, and that scaffold choice significantly impacts diversity.

A controlled experiment on 22 tasks (MLE-bench lite) with DeepSeek R1 showed that reducing diversity via prompt modifications decreased medal rates by 6.9 points (AIRA Greedy) and 8.4 points (AIRA MCTS).

This finding held across alternative metrics (valid submission rate, average normalized score, percentile, Elo).

The authors conclude that ideation diversity is crucial for performance, partly because it de-risks implementation failures, and suggest future work on diversity-aware methods.

02 From the paper

Abstract

AI research agents offer the promise to accelerate scientific progress by automating the design, implementation, and training of machine learning models. However, the field is still in its infancy, and the key factors driving the success or failure of agent trajectories are not fully understood. We examine the role that ideation diversity plays in agent performance. First, we analyse agent trajectories on MLE-bench, a well-known benchmark to evaluate AI research agents, across different models and agent scaffolds. Our analysis reveals that different models and agent scaffolds yield varying degrees of ideation diversity, and that higher-performing agents tend to have increased ideation diversity. Further, we run a controlled experiment where we modify the degree of ideation diversity, demonstrating that higher ideation diversity results in stronger performance. Finally, we strengthen our results by examining additional evaluation metrics beyond the standard medal-based scoring of MLE-bench, showing that our findings still hold across other agent performance metrics.