When Vision Speaks for Sound

The paper identifies a 'Clever Hans effect' in video-capable multimodal LLMs, where models appear to understand audio but actually rely on visual cues to hallucinate or infer sounds without verifying the audio stream.

This is demonstrated across open-source and closed-source models.

To systematically study this, the authors introduce THUD, a diagnostic framework using three counterfactual audio interventions: Shift (temporal displacement), Mute (silence), and Swap (mismatched audio).

Experiments show large accuracy drops under these interventions, with models often defaulting to 'synced' or hallucinating sounds.

The authors propose a two-stage alignment recipe combining intervention-derived preference pairs with general video data, which improves average performance across the three interventions by 28 percentage points while slightly improving general video benchmarks.

The best recipe uses a 10K-sample mixture, enhancing temporal grounding and audio verification without an alignment tax.

The work highlights the need for counterfactual evaluation and training to achieve genuine audio-visual grounding rather than visual-prior shortcuts.

The study uses Qwen3-Omni-30B as the trainable backbone and evaluates on benchmarks like Video-MME, LVBench, DailyOmni, and WorldSense, showing robust improvements in synchronization and localization tasks.

The findings suggest that current models often fail to verify audio presence, timing, and consistency, and that targeted intervention-based training can mitigate these failures while preserving broad video understanding.

The paper contributes a diagnostic protocol, a training recipe, and insights into distinct failure modes of audio-visual grounding, emphasizing the importance of breaking natural correlations in evaluation and training.

The results indicate that temporal synchronization, audio existence, and sound consistency are separate deficiencies…