Paper 2601.09088
Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoning
- Published
- Jan 2026
- Research lab
- Independent
- Citations
- 13
- GitHub
- 105 stars
01 In brief
Summary
This report introduces DASD-4B-Thinking, a lightweight open-source reasoning model that achieves state-of-the-art performance on math, science, and code benchmarks, outperforming larger models (e.g., 32B) using only 448K training samples.
The authors critique the common sequence-level distillation paradigm (SFT on teacher-generated responses) for three limitations: inadequate coverage of the teacher's output distribution, misalignment between teacher distribution and student capacity, and exposure bias from teacher forcing.
They propose three innovations: temperature-scheduled learning (training first on low-temperature, then high-temperature samples), divergence-aware sampling (prioritizing responses where teacher confidence is high but student probability is low), and mixed-policy distillation (using student-generated prefixes completed by the teacher).
Using Qwen3-4B-Instruct-2507 as student and gpt-oss-120b as teacher, they achieve 88.5 on AIME24, 83.3 on AIME25, 69.3 on LiveCodeBench v5, and 68.4 on GPQA-Diamond.
They also release a MoE variant, DASD-30B-A3B-Thinking-Preview, which shows competitive results.
The models and dataset are open-sourced.
02 From the paper
Abstract
In this report, we introduce DASD-4B-Thinking, a lightweight yet highly capable, fully open-source reasoning model. It achieves SOTA performance among open-source models of comparable scale across challenging benchmarks in mathematics, scientific reasoning, and code generation -- even outperforming several larger models. We begin by critically reexamining a widely adopted distillation paradigm in the community: SFT on teacher-generated responses, also known as sequence-level distillation. Although a series of recent works following this scheme have demonstrated remarkable efficiency and strong empirical performance, they are primarily grounded in the SFT perspective. Consequently, these approaches focus predominantly on designing heuristic rules for SFT data filtering, while largely overlooking the core principle of distillation itself -- enabling the student model to learn the teacher's full output distribution so as to inherit its generalization capability. Specifically, we identify three critical limitations in current practice: i) Inadequate representation of the teacher's sequence-level distribution; ii) Misalignment between the teacher's output distribution and the student's learning capacity; and iii) Exposure bias arising from teacher-forced training versus autoregressive inference. In summary, these shortcomings reflect a systemic absence of explicit teacher-student interaction throughout the distillation process, leaving the essence of distillation underexploited. To address these issues, we propose several methodological innovations that collectively form an enhanced sequence-level distillation training pipeline. Remarkably, DASD-4B-Thinking obtains competitive results using only 448K training samples -- an order of magnitude fewer than those employed by most existing open-source efforts. To support community research, we publicly release our models and the training dataset.