Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference

Seed Diffusion Preview is a large-scale discrete-state diffusion language model developed by ByteDance Seed and Tsinghua AIR, focused on code generation.

It achieves an inference speed of 2,146 tokens per second on H20 GPUs, significantly faster than contemporary models like Mercury Coder and Gemini Diffusion, while maintaining competitive performance on standard code benchmarks.

The model addresses two key challenges: the inductive bias of token-order modeling and inference inefficiency.

It introduces a two-stage curriculum (TSC) combining mask-based and edit-based forward corruption processes, a constrained-order diffusion training using distilled high-quality trajectories, and an on-policy diffusion learning paradigm to reduce generation steps.

Inference uses block-level parallel sampling with KV-caching.

Evaluations on benchmarks like HumanEval, MBPP, BigCodeBench, LiveCodeBench, MBXP, NaturalCodeBench, Aider, and CanItEdit show performance comparable to autoregressive models, with notable strengths in code editing tasks.

The model establishes a new state of the art on the speed-quality Pareto frontier for code models, demonstrating the potential of discrete diffusion for fast inference in large language models.

A demo is available at https://studio.seed.ai/exp/seed_diffusion/.