Paper 2509.24002
MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use
- Published
- Sep 2025
- Research lab
- Independent
- Citations
- 25
- GitHub
- 456 stars
01 In brief
Summary
MCPMark is a benchmark designed to evaluate LLM agents' use of the Model Context Protocol (MCP) in realistic, comprehensive workflows.
It comprises 127 tasks across five MCP servers (Filesystem, Notion, Playwright, GitHub, PostgreSQL), each with curated initial states and programmatic verification scripts.
Tasks require diverse CRUD operations and average 16.2 execution turns and 17.4 tool calls, significantly more than prior benchmarks.
Evaluation using a minimal agent framework (MCPMark-Agent) shows that the best model, gpt-5-medium, achieves only 52.56% pass@1 and 33.86% pass^4, while other strong models like claude-sonnet-4 and o3 fall below 30% pass@1 and 15% pass^4.
Performance varies by environment, with local services (PostgreSQL, Filesystem) yielding higher success rates than remote ones (Notion, GitHub).
The benchmark highlights challenges in stability, generalization, and planning, and reveals that more turns do not necessarily improve performance.
Failure analysis shows implicit failures dominate, with explicit errors being model-specific.
The benchmark aims to stress-test current models and guide future improvements in agent reasoning, context efficiency, and execution stability.
02 From the paper
Abstract
MCP standardizes how LLMs interact with external systems, forming the foundation for general agents. However, existing MCP benchmarks remain narrow in scope: they focus on read-heavy tasks or tasks with limited interaction depth, and fail to capture the complexity and realism of real-world workflows. To address this gap, we propose MCPMark, a benchmark designed to evaluate MCP use in a more realistic and comprehensive manner. It consists of $127$ high-quality tasks collaboratively created by domain experts and AI agents. Each task begins with a curated initial state and includes a programmatic script for automatic verification. These tasks demand richer and more diverse interactions with the environment, involving a broad range of create, read, update, and delete (CRUD) operations. We conduct a comprehensive evaluation of cutting-edge LLMs using a minimal agent framework that operates in a tool-calling loop. Empirical results show that the best-performing model, gpt-5-medium, reaches only $52.56$\% pass@1 and $33.86$\% pass^4, while other widely regarded strong models, including claude-sonnet-4 and o3, fall below $30$\% pass@1 and $15$\% pass^4. On average, LLMs require $16.2$ execution turns and $17.4$ tool calls per task, significantly surpassing those in previous MCP benchmarks and highlighting the stress-testing nature of MCPMark.