CDLM: Consistency Diffusion Language Models for Faster Sampling

CDLM (Consistency Diffusion Language Models) is a training-based method to accelerate inference in Diffusion Language Models (DLMs) by addressing two bottlenecks: excessive refinement steps and incompatibility with KV caching.

It integrates consistency modeling to reduce sampling steps via multi-token finalization and enforces a block-wise causal attention mask during fine-tuning to enable exact KV caching.

The method involves distilling a block-causal student from a bidirectional teacher, using a three-objective loss (distillation, consistency, and DLM loss).

Experiments on Dream-7B-Instruct and LLaDA-8B-Instruct show 3.6×–14.5× lower latency and 3.4×–7.9× fewer steps while maintaining competitive accuracy on math and coding benchmarks.

CDLM also surpasses equal-size autoregressive models in tokens-per-second.

Training takes 8-16 hours on 4× A100 GPUs.

The approach is orthogonal to inference-only acceleration techniques and can be combined with them.

Limitations include reliance on offline, static trajectories and being bounded by the teacher's knowledge.

Code is available at https://github.com/SqueezeAILab/CDLM.

The paper includes ablations on loss weights, step reduction, confidence thresholds, and block size, plus a roofline analysis showing block-wise DLMs balance memory-bound and compute-bound regimes.

Future directions include scaling the distillation corpus, on-policy learning, and distilling from stronger teachers.

The work is published at MLSys 2026.

The method is validated on GSM8K, MATH, HumanEval, and MBPP benchmarks.

The authors are from Seoul National University, UC Berkeley, and Together AI.

The paper is arXiv:2511.19269v2.

The method reduces refinement steps by 3.4×–7.9× and latency by 3.6×–14.5×.

The student model uses block-wise causal attention, enabling exact KV caching.

The training objective includes distillation, consistency, and DLM losses…