Paper 2606.05121
Audio Interaction Model
- Published
- Jun 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- 575 stars
01 In brief
Summary
The paper introduces Audio-Interaction, a unified streaming audio language model that operates via an always-on perceive–decide–respond loop, listening to continuous audio and deciding when to respond or remain silent.
It addresses limitations of offline LALMs and task-specific streaming models by unifying capabilities like real-time ASR, translation, dialogue, and proactive assistance.
The SOUNDFLOW framework enables this through streaming-native data construction (TFJP preprocessing, hierarchical event curation), comprehension-aware training (history review, silence training), and asynchronous FIFO inference reducing first-frame latency by 4.5x.
The StreamAudio-2M dataset (2.6M items, 302k hours) covers 7 abilities and 28 sub-tasks, and Proactive-Sound-Bench evaluates proactive intervention.
Experiments on 8 benchmarks show competitive performance (e.g., MMAU 58.15 vs 57.81 baseline) and unlock capabilities like robust audio instruction following and proactive response (61.2/62.8 on Proactive-Sound-Bench).
Ablations confirm the importance of FIFO inference, streaming training, chunk size (0.4s optimal), and dual-loss weight.
The model retains over 91% accuracy on long streams, while baselines degrade by 30%+.
02 From the paper
Abstract
Audio is an inherently interactive modality, yet today's Large Audio Language Models (LALMs) are offline, and streaming audio models each handle only a single task such as streaming ASR or voice chatting. It is time to unify them into one online LALM: a model that, through an always-on perceive-decide-respond loop, listens to sound, environment, and instructions in real time and reacts on the fly. We formalize this regime as the Audio Interaction Model, and realize it with Audio-Interaction, a unified streaming model that retains offline task execution while adding online general audio instruction following, from dialogue to full voice chatting, deciding when to respond from the semantics of the stream. To enable this, we propose SoundFlow, a framework that instantiates the perceive-decide-respond loop end to end, from data to training to deployment, through streaming-native data construction, comprehension-aware training, and asynchronous low-latency inference for stable real-time interaction. We further construct StreamAudio-2M, a 2.6M-item streaming corpus spanning 7 fundamental abilities and 28 sub-tasks, and Proactive-Sound-Bench for evaluating proactive audio intervention. Across 8 benchmarks, Audio-Interaction preserves competitive performance on mainstream audio tasks while unlocking capabilities inaccessible to offline LALMs, including real-time ASR, streaming audio instruction following, and proactive help.