The Year in AI Papers
The year/Independent research

Paper 2508.11987

FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction

Zhiyuan Zeng, Jiashuo Liu, Siyuan Chen, Tianci He, Yali Liao, Yixiao Tian, Jinpeng Wang, Zaiyuan Wang, Yang Yang, Lingyue Yin, Mingren Yin, Zhenwei Zhu, et al.

Published
Aug 2025
Research lab
Independent
Citations
39
GitHub
Not linked

01 In brief

Summary

FutureX is a live benchmark for evaluating LLM agents on future prediction tasks, built to avoid data contamination by using only future events.

It collects questions daily from 195 curated websites across 11 domains, runs 25 models (base LLMs, Think&Search models, open-source and closed-source Deep Research agents), and automatically scores predictions after events resolve.

The benchmark includes four difficulty tiers (Basic, Wide Search, Deep Search, Super Agent) and four event types (single-choice, multi-choice, open-ended ranking, open-ended numerical).

Results from July 20 to August 3, 2025, show Grok-4 and GPT-o4-mini (Think&Search) lead overall, while base LLMs like DouBao-Seed1.6-Thinking excel on simpler levels.

Human experts outperform agents on most tiers except Level 2.

Case studies reveal that Deep Research agents are vulnerable to fake websites (except Gemini), and real-time search capabilities vary, with GPT-o3 Deep Research performing best.

The benchmark aims to push agents toward professional human analyst performance.

02 From the paper

Abstract

Future prediction is a complex task for LLM agents, requiring a high level of analytical thinking, information gathering, contextual understanding, and decision-making under uncertainty. Agents must not only gather and interpret vast amounts of dynamic information but also integrate diverse data sources, weigh uncertainties, and adapt predictions based on emerging trends, just as human experts do in fields like politics, economics, and finance. Despite its importance, no large-scale benchmark exists for evaluating agents on future prediction, largely due to challenges in handling real-time updates and retrieving timely, accurate answers. To address this, we introduce $\textbf{FutureX}$, a dynamic and live evaluation benchmark specifically designed for LLM agents performing future prediction tasks. FutureX is the largest and most diverse live benchmark for future prediction, supporting real-time daily updates and eliminating data contamination through an automated pipeline for question gathering and answer collection. We evaluate 25 LLM/agent models, including those with reasoning, search capabilities, and integration of external tools such as the open-source Deep Research Agent and closed-source Deep Research models. This comprehensive evaluation assesses agents' adaptive reasoning and performance in dynamic environments. Additionally, we provide in-depth analyses of agents' failure modes and performance pitfalls in future-oriented tasks, including the vulnerability to fake web pages and the temporal validity. Our goal is to establish a dynamic, contamination-free evaluation standard that drives the development of LLM agents capable of performing at the level of professional human analysts in complex reasoning and predictive thinking.