Set Block Decoding is a Language Model Inference Accelerator

Set Block Decoding (SBD) is a new inference acceleration paradigm for autoregressive language models that integrates next token prediction (NTP) with masked token prediction (MATP) in a single architecture.

SBD allows sampling multiple, non-consecutive future tokens in parallel, using solvers from discrete diffusion literature like the Entropy Bounded (EB) Sampler.

It requires no architectural changes, maintains exact KV-caching, and can be implemented by fine-tuning existing NTP models.

The training loss combines NTP and masked token prediction losses, with a variable block size sampled uniformly from [2,16].

Experiments fine-tuning Llama-3.1 8B and Qwen-3 8B on 70B tokens show SBD preserves performance while reducing the number of forward passes by 3-5x.

Ablations on 3B models show the NTP loss term is crucial for maintaining autoregressive capabilities, and SBD requires more SFT steps to match NTP performance but closes the gap after ~34k iterations.

A roofline analysis on H100 GPUs indicates that NFE speedups translate almost directly to wall-clock speedups for block sizes up to 16, with diminishing returns at larger batch or block sizes.

SBD is simpler than speculative decoding and hybrid models, offering a practical way to accelerate LLM inference without compromising accuracy.