Paper 2508.20453
MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 90
- GitHub
- 496 stars
01 In brief
Summary
MCP-Bench is a benchmark for evaluating LLM agents on realistic, multi-step tool-use tasks via the Model Context Protocol (MCP).
It connects agents to 28 live MCP servers with 250 tools across domains like finance, travel, and science, enabling complex intra- and cross-server workflows.
Tasks are generated via an LLM-based synthesis pipeline that discovers tool dependency chains, filters for solvability and utility, and creates fuzzy, instruction-minimal variants.
Evaluation combines rule-based checks (tool name validity, schema compliance, execution success) with rubric-based LLM-as-a-Judge scoring (task completion, tool usage, planning effectiveness), using prompt shuffling and score averaging for stability.
Experiments on 20 LLMs show that while schema understanding is largely converged (over 95% compliance), planning and reasoning remain key differentiators: gpt-5 (0.749), o3 (0.715), and gpt-oss-120b (0.692) lead, while smaller models like llama-3-1-8b-instruct (0.428) lag, especially in multi-server settings.
The benchmark reveals persistent gaps in dependency awareness, parallelism, and long-horizon planning, highlighting that execution fidelity is no longer the bottleneck.
02 From the paper
Abstract
We introduce MCP-Bench, a benchmark for evaluating large language models (LLMs) on realistic, multi-step tasks that demand tool use, cross-tool coordination, precise parameter control, and planning/reasoning for solving tasks. Built on the Model Context Protocol (MCP), MCP-Bench connects LLMs to 28 representative live MCP servers spanning 250 tools across domains such as finance, traveling, scientific computing, and academic search. Unlike prior API-based benchmarks, each MCP server provides a set of complementary tools designed to work together, enabling the construction of authentic, multi-step tasks with rich input-output coupling. Tasks in MCP-Bench test agents' ability to retrieve relevant tools from fuzzy instructions without explicit tool names, plan multi-hop execution trajectories for complex objectives, ground responses in intermediate tool outputs, and orchestrate cross-domain workflows - capabilities not adequately evaluated by existing benchmarks that rely on explicit tool specifications, shallow few-step workflows, and isolated domain operations. We propose a multi-faceted evaluation framework covering tool-level schema understanding and usage, trajectory-level planning, and task completion. Experiments on 20 advanced LLMs reveal persistent challenges in MCP-Bench. Code and data: https://github.com/Accenture/mcp-bench.