Paper 2508.15882
Beyond Transcription: Mechanistic Interpretability in ASR
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 11
- GitHub
- Not linked
01 In brief
Summary
This paper adapts interpretability methods from LLMs—logit lens, linear probing, and activation patching—to analyze the internal mechanisms of ASR models, specifically Whisper-large-v3 and Qwen2-Audio.
The authors find that acoustic and semantic attributes (e.g., speaker gender, noise, accent) are linearly decodable from encoder layers, with peak accuracies of 94.6% for gender, 90.0% for clean/noisy, and 97.0% for accent.
Hallucination-related signals are linearly separable in the decoder's residual stream, enabling quality prediction with up to 93.4% accuracy.
They also show that the encoder encodes contextual and semantic information, which can override acoustic evidence, and that disrupting encoder components can paradoxically improve acoustic accuracy.
Repetition hallucinations are localized to specific cross-attention components, with head 13 in layer 18 suppressing repetitions by 78.1%.
The Encoder Lens reveals that mid-encoder layers produce fluent but ungrounded text, and Qwen2-Audio reverts to memorized phrases when uncertain.
These insights demonstrate the value of interpretability for improving ASR transparency and robustness.
02 From the paper
Abstract
Interpretability methods have recently gained significant attention, particularly in the context of large language models, enabling insights into linguistic representations, error detection, and model behaviors such as hallucinations and repetitions. However, these techniques remain underexplored in automatic speech recognition (ASR), despite their potential to advance both the performance and interpretability of ASR systems. In this work, we adapt and systematically apply established interpretability methods such as logit lens, linear probing, and activation patching, to examine how acoustic and semantic information evolves across layers in ASR systems. Our experiments reveal previously unknown internal dynamics, including specific encoder-decoder interactions responsible for repetition hallucinations and semantic biases encoded deep within acoustic representations. These insights demonstrate the benefits of extending and applying interpretability techniques to speech recognition, opening promising directions for future research on improving model transparency and robustness.