Paper 2601.22027
CAR-bench: Evaluating the Consistency and Limit-Awareness of LLM Agents under Real-World Uncertainty
- Published
- Jan 2026
- Research lab
- Independent
- Citations
- 6
- GitHub
- 33 stars
01 In brief
Summary
CAR-bench is a benchmark for evaluating LLM agents in real-world, user-facing applications, specifically in-car voice assistants.
It addresses limitations of existing benchmarks that overlook reliability under uncertainty.
The environment includes an LLM-simulated user, 19 domain policies, 58 interconnected tools, and dynamic states/databases.
Beyond standard task completion, it introduces Hallucination tasks (testing limit-awareness when tools/info are missing) and Disambiguation tasks (requiring clarification or internal info gathering).
Baseline results show a large gap between potential (Pass@3) and consistent success (Pass^3).
Even frontier models like GPT-5 achieve less than 50% consistent pass rate on Disambiguation tasks, often due to premature actions.
Models frequently violate policies or fabricate information in Hallucination tasks.
Error analysis reveals a completion-compliance tension, where models prioritize satisfying user requests over following policies.
Thinking models outperform non-thinking ones, but reasoning only partially mitigates issues.
The benchmark provides a detailed error taxonomy and highlights the need for more reliable, self-aware agents.
02 From the paper
Abstract
Existing benchmarks for Large Language Model (LLM) agents focus on task completion under idealistic settings but overlook reliability in real-world, user-facing applications. In domains, such as in-car voice assistants, users often issue incomplete or ambiguous requests, creating intrinsic uncertainty that agents must manage through dialogue, tool use, and policy adherence. We introduce CAR-bench, a benchmark for evaluating consistency, uncertainty handling, and capability awareness in multi-turn, tool-using LLM agents in an in-car assistant domain. The environment features an LLM-simulated user, domain policies, and 58 interconnected tools spanning navigation, productivity, charging, and vehicle control. Beyond standard task completion, CAR-bench introduces Hallucination tasks that test agents' limit-awareness under missing tools or information, and Disambiguation tasks that require resolving uncertainty through clarification or internal information gathering. Baseline results reveal large gaps between occasional and consistent success on all task types. Even frontier reasoning LLMs achieve less than 50% consistent pass rate on Disambiguation tasks due to premature actions, and frequently violate policies or fabricate information to satisfy user requests in Hallucination tasks, underscoring the need for more reliable and self-aware LLM agents in real-world settings.