Embarrassingly Simple Self-Distillation Improves Code Generation

The paper introduces Simple Self-Distillation (SSD), a method that improves LLM code generation by sampling solutions from the model itself with specific temperature and truncation settings, then fine-tuning on those raw, unverified outputs via standard supervised fine-tuning.

SSD requires no verifier, teacher model, or reinforcement learning.

It improves Qwen3-30B-Instruct from 42.4% to 55.3% pass@1 on LiveCodeBench v6, with gains concentrated on harder problems, and generalizes across six models (Qwen, Llama, GPT-OSS) at 4B–30B scale, including instruct and thinking variants.

The authors attribute the gains to a 'precision-exploration conflict' in decoding: 'lock' positions demand precision, while 'fork' positions demand exploration.

SSD reshapes token distributions context-dependently, suppressing distractor tails at locks while preserving useful diversity at forks.

Theoretical analysis decomposes the SSD objective into support compression, within-support reshaping, and alignment terms.

A stress test with gibberish training data still yields improvements, indicating the benefit comes from distributional reshaping rather than sample quality.

Decode-only tuning cannot match SSD's gains, as it cannot alter the model's distribution itself.