The Year in AI Papers
The year/Mistral AI

Paper 2603.25551

Voxtral TTS

Mistral-AI, :, Alexander H. Liu, Alexis Tacnet, Andy Ehrenberg, Andy Lo, Chen-Yo Sun, Guillaume Lample, Henry Lagarde, Jean-Malo Delignon, Jaeyoung Kim, John Harvill, et al.

Published
Mar 2026
Research lab
Mistral AI
Citations
0
GitHub
Not linked

01 In brief

Summary

Voxtral TTS is a multilingual zero-shot text-to-speech model that generates natural speech from as little as 3 seconds of reference audio.

It uses a hybrid architecture: an autoregressive decoder backbone (based on Ministral 3B) predicts semantic speech tokens, while a flow-matching transformer predicts acoustic tokens.

The tokens are produced by Voxtral Codec, a speech tokenizer with a hybrid VQ-FSQ quantization scheme, compressing 24 kHz audio into 12.5 Hz frames of 1 semantic and 36 acoustic tokens at 2.14 kbps.

The codec uses an ASR-distilled semantic token from Whisper and adversarial training.

The model supports 9 languages and is trained with a two-part loss (cross-entropy for semantic tokens, flow-matching for acoustic tokens), followed by Direct Preference Optimization (DPO) to improve WER and speaker similarity.

In human evaluations, Voxtral TTS achieves a 68.4% win rate over ElevenLabs Flash v2.5 for voice cloning and 58.3% for flagship voices.

It also outperforms ElevenLabs v3 on speaker similarity in automatic evaluations.

The model is served via vLLM-Omni with CUDA graph acceleration and asynchronous chunked streaming, achieving sub-second latency and up to 1,431 characters per second per GPU on a single H200.

Weights are released under CC BY-NC license.

02 From the paper

Abstract

We introduce Voxtral TTS, an expressive multilingual text-to-speech model that generates natural speech from as little as 3 seconds of reference audio. Voxtral TTS adopts a hybrid architecture that combines auto-regressive generation of semantic speech tokens with flow-matching for acoustic tokens. These tokens are encoded and decoded with Voxtral Codec, a speech tokenizer trained from scratch with a hybrid VQ-FSQ quantization scheme. In human evaluations conducted by native speakers, Voxtral TTS is preferred for multilingual voice cloning due to its naturalness and expressivity, achieving a 68.4\% win rate over ElevenLabs Flash v2.5. We release the model weights under a CC BY-NC license.