Paper 2604.10866
OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation
01 In brief
Summary
OCCUBENCH is a benchmark for evaluating AI agents on real-world professional tasks across 100 scenarios, 65 domains, and 10 industry categories.
It uses Language Environment Simulators (LESs), where an LLM simulates domain-specific tool responses, enabling evaluation in domains without public environments.
A multi-agent pipeline synthesizes 382 solvable, verifiable, and diverse task instances.
Agents are evaluated on task completion and environmental robustness under fault injection (explicit errors, implicit data degradation, mixed faults).
Testing 15 frontier models reveals: no single model dominates all industries; implicit faults are harder than explicit or mixed faults; larger models, newer generations, and higher reasoning effort improve performance; and strong agents are not necessarily strong simulators, but with a capable simulator, rankings are consistent (85.7% pairwise agreement).
02 From the paper
Abstract
AI agents are expected to perform professional work across hundreds of occupational domains (from emergency department triage to nuclear reactor safety monitoring to customs import processing), yet existing benchmarks can only evaluate agents in the few domains where public environments exist. We introduce OccuBench, a benchmark covering 100 real-world professional task scenarios across 10 industry categories and 65 specialized domains, enabled by Language Environment Simulators (LESs) that simulate domain-specific environments through LLM-driven tool response generation. Our multi-agent synthesis pipeline automatically produces evaluation instances with guaranteed solvability, calibrated difficulty, and document-grounded diversity. OccuBench evaluates agents along two complementary dimensions: task completion across professional domains and environmental robustness under controlled fault injection (explicit errors, implicit data degradation, and mixed faults). We evaluate 15 frontier models across 8 model families and find that: (1) no single model dominates all industries, as each has a distinct occupational capability profile; (2) implicit faults (truncated data, missing fields) are harder than both explicit errors (timeouts, 500s) and mixed faults, because they lack overt error signals and require the agent to independently detect data degradation; (3) larger models, newer generations, and higher reasoning effort consistently improve performance. GPT-5.2 improves by 27.5 points from minimal to maximum reasoning effort; and (4) strong agents are not necessarily strong environment simulators. Simulator quality is critical for LES-based evaluation reliability. OccuBench provides the first systematic cross-industry evaluation of AI agents on professional occupational tasks.