Paper 2510.19779
AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoders
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 7
- GitHub
- 39 stars
01 In brief
Summary
AdaSPEC is a novel knowledge distillation method for speculative decoding (SD) that improves draft model alignment with the target model by selectively filtering tokens during training.
Unlike conventional KD that minimizes KL divergence across all tokens, AdaSPEC identifies and excludes 'hard' tokens that are difficult for the small draft model to learn, focusing its limited capacity on easier tokens.
The method uses a reference model (distilled from the target) to compute token-wise loss differences, selecting the top k% of tokens with the largest gaps for distillation.
Experiments on GSM8K, Alpaca, MBPP, CNN/Daily Mail, and XSUM with Pythia-31M/1.4B and CodeGen-350M/Phi-2 configurations show AdaSPEC consistently outperforms DistillSpec, achieving up to 15% higher acceptance rates.
Ablations confirm the importance of token selection, with top 40% tokens outperforming bottom 40%, and forward KL divergence being superior to RKL and TVD.
AdaSPEC also demonstrates 10-20% wall-clock speed-ups, integrates with EAGLE, scales to larger models (Qwen2.5-0.5B/32B), and works on mixed datasets.
The approach is particularly effective when the size gap between draft and target models is large.
02 From the paper
Abstract
Speculative Decoding (SD) accelerates large language model inference by employing a small draft model to generate predictions, which are then verified by a larger target model. The effectiveness of SD hinges on the alignment between these models, which is typically enhanced by Knowledge Distillation (KD). However, conventional KD methods aim to minimize the KL divergence between the draft and target models across all tokens, a goal that is misaligned with the true objective of SD, which is to maximize token acceptance rate. Therefore, draft models often struggle to fully assimilate the target model's knowledge due to capacity constraints, leading to suboptimal performance. To address this challenge, we propose AdaSPEC, a novel method that incorporates selective token filtering into the KD process. AdaSPEC utilizes a reference model to identify and filter out difficult-to-fit tokens, enabling the distillation of a draft model that better aligns with the target model on simpler tokens. This approach improves the overall token acceptance rate without compromising generation quality. We evaluate AdaSPEC across diverse tasks, including arithmetic reasoning, instruction-following, coding, and summarization, using model configurations of 31M/1.4B and 350M/2.7B parameters. Our results demonstrate that AdaSPEC consistently outperforms the state-of-the-art DistillSpec method, achieving higher acceptance rates across all tasks (up to 15\%). The code is publicly available at https://github.com/yuezhouhu/adaspec.