The Year in AI Papers
The year/Independent research

Paper 2605.30993

SwanVoice: Expressive Long-Form Zero-Shot Speech Synthesis for Both Monologue and Dialogue

Ruiqi Li, Yu Zhang, Changhao Pan, Ke Lei, Xiang Yin, Cheng Yang

Published
May 2026
Research lab
Independent
Citations
2
GitHub
Not linked

01 In brief

Summary

The paper introduces SwanVoice, a zero-shot text-to-speech (TTS) model for expressive long-form monologue and dialogue synthesis with 1–4 speakers.

It addresses limitations of stitching monologue outputs for dialogue, which breaks acoustic consistency and affective continuity.

The authors build SwanData-Speech, a data pipeline that processes 2.59 million hours of in-the-wild audio (2.24M Chinese, 0.35M English) using speech enhancement, speaker diarization, ASR, pause-aware punctuation correction, and quality/emotion filtering.

SwanVoice uses a 25 Hz VAE, raw-text conditioning with pause symbols and pinyin substitution, and a flow-matching DiT with speaker-turn conditioning.

Training follows a curriculum: monologue pretraining (2M hours), mixed conversational training, and SFT on real dialogue, followed by DiffusionNFT post-training with phone-level and speaker-similarity rewards.

On SwanBench-Speech, SwanVoice achieves higher richness and hierarchy scores than all evaluated open-source baselines in both monologue (3.81/3.62) and dialogue (3.62/3.71) settings, though content accuracy remains a limitation.

The model has 2B parameters and was trained on A100 GPUs.

02 From the paper

Abstract

Zero-shot text-to-speech (TTS) has improved substantially for single-speaker synthesis, yet expressive long-form multi-speaker dialogue remains difficult. A common workaround is to synthesize each turn with a monologue TTS model and stitch the outputs together. This adds inference cost and often breaks acoustic consistency, conversational coherence, and affective continuity across turns. Recent dialogue TTS systems have begun to address this setting, but they still struggle to keep expressive coherence, controllable speaker switching, and monologue quality at the same time. We present SwanData-Speech and SwanVoice. SwanData-Speech builds monologue and dialogue corpora from in-the-wild audio, using Swan Forced Aligner for pause-aware word-level alignment and RobustMegaTTS3 for pronunciation-hard cases. Built on these data, SwanVoice is a zero-shot TTS model for 1--4 speakers, combining a 25 Hz VAE, raw-text conditioning with pause-aware symbols and pinyin substitution, and a flow-matching DiT with speaker-turn conditioning. Training starts from monologue speech, moves through mixed and real dialogue data, and then uses DiffusionNFT post-training with phone-level and speaker-similarity rewards. On SwanBench-Speech, SwanVoice obtains higher richness and hierarchy scores than all evaluated open-source baselines in both monologue and dialogue settings, while content accuracy remains the main limitation. Audio demos are available at https://swanaigc.github.io//#swanvoice.