The Year in AI Papers
The year/Topics/Audio, speech, and omni-modal

Research collection

Audio, speech, and omni-modal

Speech recognition and synthesis, music and audio generation, and omni-modal models that jointly handle audio with text or vision.

Papers
11
Research labs
2
Official code
6

1–11 of 11 papers in this collection

01

Independent research

SwanTale: Unified Multi-Speaker Speech and Audio Generation for Instruct and Zero-Shot Tasks

SwanTale is a unified model for multi-speaker speech and audio generation supporting both zero-shot and instruct tasks. It introduces SwanData-Caption, a data pipeline that cleans raw audio, adds targeted synthetic coverage (elderly speech, short utterances, challenging pronunciations), and annotates multi-level captions (environment, speakers, content).…

Yu Zhang, Ruiqi Li, Changhao Pan, Ke Lei, et al.
Published
Aug 2026
Citations
0
Code
Not linked
Read summary Original
02

Independent research

Audio Interaction Model

The paper introduces Audio-Interaction, a unified streaming audio language model that operates via an always-on perceive–decide–respond loop, listening to continuous audio and deciding when to respond or remain silent. It addresses limitations of offline LALMs and task-specific streaming models by unifying capabilities like real-time ASR, translation,…

Zhifei Xie, Zihang Liu, Ze An, Xiaobin Hu, et al.
Published
Jun 2026
Citations
0
Code
575 stars
Read summary Original
03

Independent research

SwanVoice: Expressive Long-Form Zero-Shot Speech Synthesis for Both Monologue and Dialogue

The paper introduces SwanVoice, a zero-shot text-to-speech (TTS) model for expressive long-form monologue and dialogue synthesis with 1–4 speakers. It addresses limitations of stitching monologue outputs for dialogue, which breaks acoustic consistency and affective continuity. The authors build SwanData-Speech, a data pipeline that processes 2.59 million…

Ruiqi Li, Yu Zhang, Changhao Pan, Ke Lei, et al.
Published
May 2026
Citations
2
Code
Not linked
Read summary Original
04

arXiv.org

Mega-ASR: Towards In-the-wild^2 Speech Recognition via Scaling up Real-world Acoustic Simulation

MEGA-ASR is a framework for automatic speech recognition (ASR) in real-world environments, addressing the 'acoustic robustness bottleneck' where models fail under severe, compositional distortions. The authors introduce VOICES-IN-THE-WILD-2M, a large-scale dataset with 7 atomic acoustic effects (noise, far-field, obstructed, echo&reverb, recording,…

Zhifei Xie, Kaiyu Pang, Haobin Zhang, Deheng Ye, et al.
Published
May 2026
Citations
1
Code
1.1K stars
Read summary Original
05

arXiv.org

Qwen3.5-Omni Technical Report

Qwen3.5-Omni is a fully omnimodal large language model that scales to hundreds of billions of parameters and supports a 256k context length. It is pretrained on a massive dataset including over 100 million hours of audio-visual content. The model uses a Thinker-Talker architecture with Hybrid-Attention Mixture-of-Experts (MoE) for both components, enabling…

Qwen Team
Published
Apr 2026
Citations
87
Code
Not linked
Read summary Original
06

Mistral AI

Voxtral TTS

Voxtral TTS is a multilingual zero-shot text-to-speech model that generates natural speech from as little as 3 seconds of reference audio. It uses a hybrid architecture: an autoregressive decoder backbone (based on Ministral 3B) predicts semantic speech tokens, while a flow-matching transformer predicts acoustic tokens. The tokens are produced by Voxtral…

Mistral-AI, :, Alexander H. Liu, Alexis Tacnet, et al.
Published
Mar 2026
Citations
0
Code
Not linked
Read summary Original
07

Qwen

Qwen3-TTS Technical Report

The Qwen3-TTS technical report introduces a family of multilingual, controllable, robust, and streaming text-to-speech models. Trained on over 5 million hours of speech data across 10 languages, Qwen3-TTS supports 3-second voice cloning, description-based voice design, and fine-grained control. It uses a dual-track LM architecture with two tokenizers:…

Hangrui Hu, Xinfa Zhu, Ting He, Dake Guo, et al.
Published
Jan 2026
Citations
88
Code
13K stars
Read summary Original
08

arXiv.org

MOSS Transcribe Diarize Technical Report

MOSS Transcribe Diarize is a unified multimodal large language model for Speaker-Attributed, Time-Stamped Transcription (SATS), jointly performing word recognition, speaker attribution, and timestamp prediction in a single end-to-end pass. It uses a 128k-token context window to process up to 90 minutes of audio without chunking, preserving long-range…

MOSI. AI, :, Donghua Yu, Zhengyuan Lin, et al.
Published
Jan 2026
Citations
5
Code
Not linked
Read summary Original
09

arXiv.org

StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs

StableToken is a semantic speech tokenizer designed to be robust to acoustic noise, addressing the fragility of existing VQ-based tokenizers that produce unstable token sequences under perturbations. The paper identifies two flaws: a brittle single-path quantization architecture and a distant ASR training signal that ignores intermediate token stability.…

Yuhan Song, Linhao Zhang, Chuhan Wu, Aiwei Liu, et al.
Published
Sep 2025
Citations
4
Code
33 stars
Read summary Original
10

arXiv.org

EchoX: Towards Mitigating Acoustic-Semantic Gap via Echo Training for Speech-to-Speech LLMs

EchoX is a framework for speech-to-speech large language models (SLLMs) that addresses the acoustic-semantic gap, which causes knowledge and reasoning degradation compared to text-based LLMs. The authors propose a three-stage training process: (1) speech-to-text training, (2) text-to-codec training, and (3) Echo training, where a frozen text-to-codec…

Yuhao Zhang, Yuhao Du, Zhanchen Dai, Xiangnan Ma, et al.
Published
Sep 2025
Citations
3
Code
47 stars
Read summary Original
11

arXiv.org

VibeVoice Technical Report

VIBEVOICE is a novel model from Microsoft Research for synthesizing long-form, multi-speaker speech (up to 90 minutes, 4 speakers) using next-token diffusion. It introduces a causal speech tokenizer with a 3200x compression rate (7.5 Hz frame rate), which is 80 times more efficient than Encodec while maintaining comparable performance. The model uses a…

Zhiliang Peng, Jianwei Yu, Wenhui Wang, Yaoyao Chang, et al.
Published
Aug 2025
Citations
36
Code
52K stars
Read summary Original