Paper 2508.19205
VibeVoice Technical Report
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 36
- GitHub
- 52K stars
01 In brief
Summary
VIBEVOICE is a novel model from Microsoft Research for synthesizing long-form, multi-speaker speech (up to 90 minutes, 4 speakers) using next-token diffusion.
It introduces a causal speech tokenizer with a 3200x compression rate (7.5 Hz frame rate), which is 80 times more efficient than Encodec while maintaining comparable performance.
The model uses a pre-trained LLM (Qwen2.5 1.5B or 7B) to process concatenated voice and text inputs, and a token-level diffusion head to predict acoustic VAE features.
In evaluations, VIBEVOICE-7B outperformed open-source and proprietary systems (e.g., Gemini 2.5 Pro, ElevenLabs) on subjective metrics (realism, richness, preference) and achieved lower WER (1.29% Whisper) and higher speaker similarity (0.692) on long-form podcast generation.
On short utterances (SEED test sets), it showed competitive performance with lower frame rate.
The tokenizer achieved leading PESQ (3.068) and UTMOS (4.181) on LibriTTS test-clean.
Limitations include support for English and Chinese only, no non-speech audio, no overlapping speech, and potential misuse for deepfakes; the model is intended for research only.
02 From the paper
Abstract
This report presents VibeVoice, a novel model designed to synthesize long-form speech with multiple speakers by employing next-token diffusion, which is a unified method for modeling continuous data by autoregressively generating latent vectors via diffusion. To enable this, we introduce a novel continuous speech tokenizer that, when compared to the popular Encodec model, improves data compression by 80 times while maintaining comparable performance. The tokenizer effectively preserves audio fidelity while significantly boosting computational efficiency for processing long sequences. Thus, VibeVoice can synthesize long-form speech for up to 90 minutes (in a 64K context window length) with a maximum of 4 speakers, capturing the authentic conversational ``vibe'' and surpassing open-source and proprietary dialogue models.