The Year in AI Papers
The year/Independent research

Paper 2509.22220

StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs

Yuhan Song, Linhao Zhang, Chuhan Wu, Aiwei Liu, Wei Jia, Houfeng Wang, Xiao Zhou

Published
Sep 2025
Research lab
Independent
Citations
4
GitHub
33 stars

01 In brief

Summary

StableToken is a semantic speech tokenizer designed to be robust to acoustic noise, addressing the fragility of existing VQ-based tokenizers that produce unstable token sequences under perturbations.

The paper identifies two flaws: a brittle single-path quantization architecture and a distant ASR training signal that ignores intermediate token stability.

StableToken introduces a multi-branch Voting-LFQ module with bit-wise majority voting, which provides error correction at inference and fine-grained fusion during training.

A Noise-Aware Consensus Training strategy feeds clean inputs to a majority of branches and perturbed inputs to a minority, using a consensus loss to enforce invariance.

Experiments show StableToken achieves state-of-the-art token stability, reducing Unit Edit Distance (UED) from 26.17% to 10.17% on average, while maintaining reconstruction quality.

Downstream SpeechLLMs using StableToken show significant robustness gains in ASR, SER, and TTS tasks, especially under severe noise.

Ablations confirm each component contributes to performance, and the optimal number of voters is N=5.

02 From the paper

Abstract

Prevalent semantic speech tokenizers, designed to capture linguistic content, are surprisingly fragile. We find they are not robust to meaning-irrelevant acoustic perturbations; even at high Signal-to-Noise Ratios (SNRs) where speech is perfectly intelligible, their output token sequences can change drastically, increasing the learning burden for downstream LLMs. This instability stems from two flaws: a brittle single-path quantization architecture and a distant training signal indifferent to intermediate token stability. To address this, we introduce StableToken, a tokenizer that achieves stability through a consensus-driven mechanism. Its multi-branch architecture processes audio in parallel, and these representations are merged via a powerful bit-wise voting mechanism to form a single, stable token sequence. StableToken sets a new state-of-the-art in token stability, drastically reducing Unit Edit Distance (UED) under diverse noise conditions. This foundational stability translates directly to downstream benefits, significantly improving the robustness of SpeechLLMs on a variety of tasks. Our code and model are publicly available at https://github.com/Tencent/StableToken.