Paper 2603.16859
SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models
- Published
- Mar 2026
- Research lab
- Independent
- Citations
- 5
- GitHub
- 46 stars
01 In brief
Summary
SocialOmni is a new benchmark for evaluating social interactivity in omni-modal large language models (OLMs), addressing the gap left by static, accuracy-centric benchmarks.
It evaluates three dimensions: who is speaking (speaker identification), when to interject (turn-taking timing), and how to phrase interruptions (response generation).
The benchmark includes 2,000 perception samples and 209 interaction-generation instances across 15 dialogue domains, with controlled audio-visual inconsistency scenarios.
Twelve OLMs were evaluated, revealing significant variance and a decoupling between perceptual accuracy and generation quality.
For example, Qwen3-Omni leads in speaker identification (69.25% accuracy) but scores low on response quality (45.57), while Gemini 2.5 Flash excels in response quality (85.08) but lags in speaker identification (47.03%).
The benchmark also identifies common failure modes, such as reliance on visual salience over cross-modal binding and premature interruptions based on shallow silence-gap detection.
SocialOmni provides a dual-axis evaluation protocol and robustness probes, demonstrating that understanding-centric metrics alone are insufficient for assessing conversational social competence in OLMs.
02 From the paper
Abstract
Omni-modal large language models (OLMs) redefine human-machine interaction by natively integrating audio, vision, and text. However, existing OLM benchmarks remain anchored to static, accuracy-centric tasks, leaving a critical gap in assessing social interactivity, the fundamental capacity to navigate dynamic cues in natural dialogues. To this end, we propose SocialOmni, a comprehensive benchmark that operationalizes the evaluation of this conversational interactivity across three core dimensions: (i) speaker separation and identification (who is speaking), (ii) interruption timing control (when to interject), and (iii) natural interruption generation (how to phrase the interruption). SocialOmni features 2,000 perception samples and a quality-controlled diagnostic set of 209 interaction-generation instances with strict temporal and contextual constraints, complemented by controlled audio-visual inconsistency scenarios to test model robustness. We benchmarked 12 leading OLMs, which uncovers significant variance in their social-interaction capabilities across models. Furthermore, our analysis reveals a pronounced decoupling between a model's perceptual accuracy and its ability to generate contextually appropriate interruptions, indicating that understanding-centric metrics alone are insufficient to characterize conversational social competence. More encouragingly, these diagnostics from SocialOmni yield actionable signals for bridging the perception-interaction divide in future OLMs.