Paper 2510.13998
BitNet Distillation
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 2
- GitHub
- Not linked
01 In brief
Summary
BitDistill is a lightweight pipeline for fine-tuning full-precision LLMs (e.g., Qwen3) into 1.58-bit ternary precision for specific downstream tasks, achieving performance comparable to full-precision models while reducing memory by 10× and speeding up CPU inference by 2.65×.
The method comprises three stages: (1) modeling refinement with SubLN modules to stabilize optimization, (2) continual pre-training on 10B tokens from the FALCON corpus as a warm-up to mitigate scalability issues, and (3) distillation-based fine-tuning combining logits distillation and MiniLM-based multi-head attention distillation.
Experiments on classification (MNLI, QNLI, SST-2) and summarization (CNNDM) across 0.6B, 1.7B, and 4B models show BitDistill matches FP16 baselines (e.g., MNLI 88.17 vs 88.01 for 0.6B) and outperforms direct 1.58-bit fine-tuning (BitNet-SFT).
The method is robust across different base models (Qwen2.5, Gemma) and compatible with various quantization techniques (Block-Quant, GPTQ, AWQ).
Ablations confirm each stage contributes, and attention distillation on a single later layer yields best results.
Using larger FP16 teachers further improves student performance.
02 From the paper
Abstract
In this paper, we present BitNet Distillation (BitDistill), a lightweight pipeline that fine-tunes off-the-shelf full-precision LLMs (e.g., Qwen) into 1.58-bit precision (i.e., ternary weights {-1, 0, 1}) for specific downstream tasks, achieving strong task-specific performance with minimal computational cost. Specifically, BitDistill incorporates three key techniques: the SubLN module, as introduced in BitNet; multi-head attention distillation, based on MiniLM; and continual pre-training, which serves as a crucial warm-up step to mitigate the scalability issue of the performance gap between finetuned full-precision and 1.58-bit LLMs on specific tasks. Experimental results show that BitDistill achieves performance comparable to the full-precision counterpart models across model size, while enabling up to 10x memory savings and 2.65x faster inference on CPUs. Code is available at https://github.com/microsoft/BitNet.