The Year in AI Papers
The year/Independent research

Paper 2605.14678

$π$-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows

Haoran Zhang, Luxin Xu, Zhilin Wang, Runquan Gui, Shunkai Zhang, Haodi Lei, Zihao He, Bingsu He, Chicheng Qin, Tong Zhu, Xiaoye Qu, Yang Yang, et al.

Published
May 2026
Research lab
Independent
Citations
1
GitHub
58 stars

01 In brief

Summary

The paper introduces π-BENCH, a benchmark for evaluating proactive assistance in long-horizon personal assistant workflows.

It comprises 100 multi-turn tasks across 5 domain-specific user personas (researcher, marketer, law trainee, pharmacist, financier), organized into 20-session episodes with cross-session dependencies.

Each task begins with an underspecified request, and agents must resolve hidden intents (latent requirements) either by direct action (completed), targeted clarification (inferred), or user provision (provided).

The benchmark jointly measures Proactivity (PROC), the fraction of intents resolved proactively, and Completeness (COMP), the fraction of checklist items satisfied.

Experiments on nine frontier models (e.g., GPT-5.4, Claude Opus 4.6, Qwen3.6 Plus) show that proactive assistance remains challenging, with average PROC ranging from 43.1 to 67.0 and COMP from 52.1 to 67.6.

Results reveal a clear distinction between task completion and proactivity, and ablations show that prior interaction history significantly improves proactive intent resolution in later tasks (average PROC decrease of 9.5 points when history is removed).

The benchmark aims to test whether agents can anticipate and address user needs over extended interactions, reducing user burden while completing workflows.

02 From the paper

Abstract

The rise of personal assistant agents, e.g., OpenClaw, highlights the growing potential of large language models to support users across everyday life and work. A core challenge in these settings is proactive assistance, since users often begin with underspecified requests and leave important needs, constraints, or preferences unstated. However, existing benchmarks rarely evaluate whether agents can identify and act on such hidden intents before they are explicitly stated, especially in sustained multi-turn interactions where user needs emerge gradually. To address this gap, we introduce $π$-Bench, a benchmark for proactive assistance comprising 100 multi-turn tasks across 5 domain-specific user personas. By incorporating hidden user intents, inter-task dependencies, and cross-session continuity, $π$-Bench evaluates agents' ability to anticipate and address user needs over extended interactions, jointly measuring proactivity and task completion in long-horizon trajectories that better reflect real-world use. Experiments show (1) proactive assistance remains challenging, (2) a clear distinction between task completion and proactivity, and (3) the value of prior interaction for proactive intent resolution in later tasks.