Paper 2605.16679
CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?
- Published
- May 2026
- Research lab
- Independent
- Citations
- 4
- GitHub
- 54 stars
01 In brief
Summary
χ-Bench is a benchmark for evaluating AI agents on long-horizon, policy-rich healthcare workflows across three domains: provider prior authorization, payer utilization management, and care management.
It uses a high-fidelity simulator of 20 healthcare apps exposed via 87 MCP tools, guided by a 1,279-document managed-care operations handbook.
Across 30 agent harness/model configurations, the best agent (Claude Code + Claude Opus 4.6) resolves only 28.0% of tasks at pass@1, no agent clears 20% on strict pass^3, and executing all tasks in a single session drops performance to 3.8%.
The end-to-end provider–payer arena collapses best prior auth agents to 0%.
Failure analysis attributes most failures to Clinical-Reasoning (35.4%), Workflow-Completion (23.3%), and Policy-Compliance (13.2%), with specific modes like criteria misapplication, skipped required steps, and policy criteria misreading.
The benchmark highlights gaps in current agent capabilities for policy-dense, role-composed, irreversible enterprise domains.
02 From the paper
Abstract
End-to-end automation of realistic healthcare operations stresses three capabilities underrepresented in current benchmarks: policy density, decisions must be grounded in a large library of medical, insurance, and operational rules; Multi-role composition: a single task requires the agent to play multiple roles with handoffs; and multilateral interaction: intermediate workflow steps are multi-turn dialogs, such as peer-to-peer review and patient outreach. We introduce $χ$-Bench, a benchmark of long-horizon healthcare workflows across three domains: provider prior authorization, payer utilization management, and care management. Each task hands the agent a clinical case in a high-fidelity simulator of 20 healthcare apps exposed via 87 MCP tools, which it must drive to a terminal status through tool calls and writing the role's artifacts, guided by a 1,290+ document managed-care operations handbook skill. Across 30 agent harness/models configurations, the best agent resolves only 28.0% of tasks, no agent clears 20% on strict pass^3, and executing all tasks in a single session slumps the performance to 3.8%. These results raise the hypothesis that similar gaps are likely to surface in other policy-dense, role-composed, irreversible enterprise domains.