Paper 2509.09174
EchoX: Towards Mitigating Acoustic-Semantic Gap via Echo Training for Speech-to-Speech LLMs
- Published
- Sep 2025
- Research lab
- Independent
- Citations
- 3
- GitHub
- 47 stars
01 In brief
Summary
EchoX is a framework for speech-to-speech large language models (SLLMs) that addresses the acoustic-semantic gap, which causes knowledge and reasoning degradation compared to text-based LLMs.
The authors propose a three-stage training process: (1) speech-to-text training, (2) text-to-codec training, and (3) Echo training, where a frozen text-to-codec module generates pseudo speech targets from the LLM's hidden states, aligning acoustic and semantic learning.
EchoX uses unit language as speech tokens and a streaming inference mechanism to handle long sequences.
Experiments show that EchoX-3B and EchoX-8B, trained on about 6,000 hours of data, achieve competitive performance on knowledge-based QA benchmarks (Llama Questions, Web Questions, TriviaQA) compared to models trained on millions of hours.
Analysis shows Echo training reduces knowledge degradation, and unit language improves compression and accuracy over raw units.
Human evaluation indicates EchoX wins on helpfulness but lags on naturalness.
02 From the paper
Abstract
Speech-to-speech large language models (SLLMs) are attracting increasing attention. Derived from text-based large language models (LLMs), SLLMs often exhibit degradation in knowledge and reasoning capabilities. We hypothesize that this limitation arises because current training paradigms for SLLMs fail to bridge the acoustic-semantic gap in the feature representation space. To address this issue, we propose EchoX, which leverages semantic representations and dynamically generates speech training targets. This approach integrates both acoustic and semantic learning, enabling EchoX to preserve strong reasoning abilities as a speech LLM. Experimental results demonstrate that EchoX, with about six thousand hours of training data, achieves advanced performance on multiple knowledge-based question-answering benchmarks. The project is available at https://github.com/FreedomIntelligence/EchoX.