Paper 2602.04705
ERNIE 5.0 Technical Report
- Published
- Feb 2026
- Research lab
- Independent
- Citations
- 8
- GitHub
- Not linked
01 In brief
Summary
ERNIE 5.0 is a natively autoregressive foundation model from Baidu for unified multimodal understanding and generation across text, image, video, and audio.
All modalities are trained from scratch under a unified next-group-of-tokens prediction objective, using an ultra-sparse mixture-of-experts (MoE) architecture with modality-agnostic expert routing.
A novel elastic training paradigm enables a single pre-training run to produce a family of sub-models with varying depth, expert capacity, and routing sparsity, allowing flexible trade-offs between performance, model size, and inference latency.
The report also addresses scaling reinforcement learning to unified models, ensuring stable post-training under ultra-sparse MoE and multimodal settings.
Extensive experiments show strong, balanced performance across modalities, and it is claimed to be the first production-scale trillion-parameter unified autoregressive model supporting both multimodal understanding and generation.
Key techniques include visual modeling with Next-Frame-and-Scale Prediction, audio modeling with Next-Codec Prediction, and infrastructure innovations like FlashMask and a disaggregated RL system.
Ablations show that reducing routing top-k to 25% yields over 15% decoding speedup with minor accuracy loss, and elastic training preserves near-full performance using only 53.7% activated parameters and 35.8% total parameters.
The report includes detailed visualizations of expert routing and empirical analysis of elastic training, offering insights for the community.
The model is evaluated on language, vision, and audio benchmarks, consistently matching or outperforming specialized baselines.
The work represents a significant step toward scalable, general-purpose unified multimodal models.
The report is dated February 4, 2026, and authored by the ERNIE Team at Baidu.
The architecture integrates text, image, video, and audio in…
02 From the paper
Abstract
In this report, we introduce ERNIE 5.0, a natively autoregressive foundation model desinged for unified multimodal understanding and generation across text, image, video, and audio. All modalities are trained from scratch under a unified next-group-of-tokens prediction objective, based on an ultra-sparse mixture-of-experts (MoE) architecture with modality-agnostic expert routing. To address practical challenges in large-scale deployment under diverse resource constraints, ERNIE 5.0 adopts a novel elastic training paradigm. Within a single pre-training run, the model learns a family of sub-models with varying depths, expert capacities, and routing sparsity, enabling flexible trade-offs among performance, model size, and inference latency in memory- or time-constrained scenarios. Moreover, we systematically address the challenges of scaling reinforcement learning to unified foundation models, thereby guaranteeing efficient and stable post-training under ultra-sparse MoE architectures and diverse multimodal settings. Extensive experiments demonstrate that ERNIE 5.0 achieves strong and balanced performance across multiple modalities. To the best of our knowledge, among publicly disclosed models, ERNIE 5.0 represents the first production-scale realization of a trillion-parameter unified autoregressive model that supports both multimodal understanding and generation. To facilitate further research, we present detailed visualizations of modality-agnostic expert routing in the unified model, alongside comprehensive empirical analysis of elastic training, aiming to offer profound insights to the community.