Paper 2603.23483
SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning
- Published
- Mar 2026
- Research lab
- Independent
- Citations
- 1
- GitHub
- 63 stars
01 In brief
Summary
SpecEyes is an agentic-level speculative acceleration framework for multimodal large language models (MLLMs) that addresses the sequential bottleneck of tool-use loops.
It uses a lightweight, tool-free model to speculatively answer queries that do not require deep agentic reasoning, bypassing the expensive tool-calling pipeline.
The framework consists of four phases: heuristic tool-use judgment, speculative prediction, cognitive gating via an answer separability score, and agentic fallback.
The cognitive gating mechanism uses a scale-invariant metric based on top-K logits to decide when to trust the small model's output.
A heterogeneous parallel funnel exploits the stateless nature of the small model to improve system throughput.
Experiments on V* Bench, HR-Bench, and POPE show that SpecEyes achieves 1.1–3.35× speedup over baselines while preserving or improving accuracy, with the min aggregation variant providing the best trade-off.
The framework is model-agnostic and demonstrates consistent gains across different backbones and draft models.
02 From the paper
Abstract
Agentic multimodal large language models (MLLMs) (e.g., OpenAI o3 and Gemini Agentic Vision) achieve remarkable reasoning capabilities through iterative visual tool invocation. However, the cascaded perception, reasoning, and tool-calling loops introduce significant sequential overhead. This overhead, termed agentic depth, incurs prohibitive latency and seriously limits system-level concurrency. To this end, we propose SpecEyes, an agentic-level speculative acceleration framework that breaks this sequential bottleneck. Our key insight is that a lightweight, tool-free MLLM can serve as a speculative planner to predict the execution trajectory, enabling early termination of expensive tool chains without sacrificing accuracy. To regulate this speculative planning, we introduce a cognitive gating mechanism based on answer separability, which quantifies the model's confidence for self-verification without requiring oracle labels. Furthermore, we design a heterogeneous parallel funnel that exploits the stateless concurrency of the small model to mask the stateful serial execution of the large model, maximizing system throughput. Extensive experiments on V* Bench, HR-Bench, and POPE demonstrate that SpecEyes achieves 1.1-3.35x speedup over the agentic baseline while preserving or even improving accuracy (up to +6.7%), thereby boosting serving throughput under concurrent workloads.