The Year in AI Papers
The year/Independent research

Paper 2510.23763

RoboOmni: Proactive Robot Manipulation in Omni-modal Context

Siyin Wang, Jinlan Fu, Feihong Liu, Xinzhe He, Huangxuan Wu, Junhao Shi, Kexin Huang, Zhaoye Fei, Jingjing Gong, Zuxuan Wu, Yu-Gang Jiang, See-Kiong Ng, et al.

Published
Oct 2025
Research lab
Independent
Citations
8
GitHub
117 stars

01 In brief

Summary

RoboOmni introduces cross-modal contextual instructions, a new setting for robotic manipulation where robots infer user intent from speech, environmental sounds, and visual cues rather than explicit commands.

The authors propose a Perceiver-Thinker-Talker-Executor framework built on end-to-end omni-modal LLMs, which fuses auditory and visual signals for intent recognition, supports direct speech interaction without ASR, and unifies intent recognition, confirmation, and action execution.

To address data scarcity, they construct OmniAction, a dataset with 140k episodes, 5k+ speakers, 2.4k event sounds, 640 backgrounds, and six contextual instruction types, plus OmniAction-LIBERO for simulation.

Experiments show RoboOmni outperforms text- and ASR-based baselines (e.g., OpenVLA, NORA, π0) in success rate (85.6% vs.

25.9% best baseline), inference speed (0.49x latency), proactive assistance, and intention recognition (88.9% accuracy).

Real-world tests on WidowX 250S confirm its effectiveness.

The work highlights the importance of end-to-end audio processing and omni-modal pretraining for robust, proactive robot manipulation.

02 From the paper

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have driven rapid progress in Vision-Language-Action (VLA) models for robotic manipulation. Although effective in many scenarios, current approaches largely rely on explicit instructions, whereas in real-world interactions, humans rarely issue instructions directly. Effective collaboration requires robots to infer user intentions proactively. In this work, we introduce cross-modal contextual instructions, a new setting where intent is derived from spoken dialogue, environmental sounds, and visual cues rather than explicit commands. To address this new setting, we present RoboOmni, a Perceiver-Thinker-Talker-Executor framework based on end-to-end omni-modal LLMs that unifies intention recognition, interaction confirmation, and action execution. RoboOmni fuses auditory and visual signals spatiotemporally for robust intention recognition, while supporting direct speech interaction. To address the absence of training data for proactive intention recognition in robotic manipulation, we build OmniAction, comprising 140k episodes, 5k+ speakers, 2.4k event sounds, 640 backgrounds, and six contextual instruction types. Experiments in simulation and real-world settings show that RoboOmni surpasses text- and ASR-based baselines in success rate, inference speed, intention recognition, and proactive assistance.