Paper 2510.02209
StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets?
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 26
- GitHub
- 177 stars
01 In brief
Summary
STOCKBENCH is a new benchmark for evaluating LLM agents in realistic, multi-month stock trading environments, using data from March to June 2025 to avoid contamination.
Agents receive daily prices, fundamentals, and news, and make sequential buy, sell, or hold decisions.
Performance is measured by cumulative return, maximum drawdown, and Sortino ratio.
Tests on models like GPT-5, Claude-4, Qwen3, and Kimi-K2 show most outperform a passive buy-and-hold baseline, with Kimi-K2 ranking first, but some models underperform.
Reasoning models do not consistently beat instruction-tuned ones.
Larger investment target sizes increase variability and degrade performance, with larger models like Kimi-K2 more robust.
Ablations show news and fundamentals are important, and model rankings shift between market upturns and downturns, with all agents underperforming the baseline in downturns.
The benchmark is open-source to support further research.
02 From the paper
Abstract
Large language models (LLMs) demonstrate strong potential as autonomous agents, with promising capabilities in reasoning, tool use, and sequential decision-making. While prior benchmarks have evaluated LLM agents in various domains, the financial domain remains underexplored, despite its significant economic value and complex reasoning requirements. Most existing financial benchmarks focus on static question-answering, failing to capture the dynamics of real-market trading. To address this gap, we introduce STOCKBENCH, a contamination-free benchmark designed to evaluate LLM agents in realistic, multi-month stock trading environments. Agents receive daily market signals -- including prices, fundamentals, and news -- and make sequential buy, sell, or hold decisions. Performance is measured using financial metrics such as cumulative return, maximum drawdown, and the Sortino ratio, capturing both profitability and risk management. We evaluate a wide range of state-of-the-art proprietary and open-source LLMs. Surprisingly, most models struggle to outperform the simple buy-and-hold baseline, while some models demonstrate the potential to achieve higher returns and stronger risk management. These findings highlight both the challenges and opportunities of LLM-based trading agents, showing that strong performance on static financial question-answering do not necessarily translate into effective trading behavior. We release STOCKBENCH as an open-source benchmark to enable future research on LLM-driven financial agents.