Paper 2511.12609
Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data
- Published
- Nov 2025
- Research lab
- Independent
- Citations
- 21
- GitHub
- 1.1K stars
01 In brief
Summary
Uni-MoE-2.0-Omni is a fully open-source omnimodal large model (OLM) built from the dense Qwen2.5-7B LLM, designed for unified understanding, reasoning, and generation across text, image, audio, and video.
Its architecture introduces a dynamic-capacity Mixture-of-Experts (MoE) with shared, routed, and null experts for efficient computation and modality specialization, plus an Omni-Modality 3D RoPE for spatio-temporal cross-modal alignment.
Training uses a progressive strategy: cross-modal pretraining on ~75B tokens, expert warm-up, MoE fine-tuning with balanced data, annealing, and iterative GSPO-DPO reinforcement learning.
Generation is enabled via a Context-Aware MoE-TTS for speech and a Task-Aware Diffusion Transformer for image generation/editing.
Evaluated on 85 benchmarks, it achieves state-of-the-art or competitive results, surpassing Qwen2.5-Omni (trained on 1.2T tokens) on over 50 of 76 benchmarks, with strengths in video understanding (+7% avg), omnimodal understanding (+7% avg), audiovisual reasoning (+4%), long-form speech (WER reduced up to 4.2%), and low-level image processing.
The model is released with code, checkpoints, and data lists.
02 From the paper
Abstract
We present Uni-MoE 2.0 from the Lychee family. As a fully open-source omnimodal large model (OLM), it substantially advances Lychee's Uni-MoE series in language-centric multimodal understanding, reasoning, and generating. Based on the dense LLM, we build Uni-MoE-2.0-Omni from scratch through three core contributions: dynamic-capacity Mixture-of-Experts (MoE) design, a progressive training strategy enhanced with an iterative reinforcement strategy, and a carefully curated multimodal data matching technique. It is capable of omnimodal understanding, as well as generating images, text, and speech. Architecturally, our new MoE framework balances computational efficiency and capability for 10 cross-modal inputs using shared, routed, and null experts, while our Omni-Modality 3D RoPE ensures spatio-temporal cross-modality alignment in the self-attention layer. For training, following cross-modal pretraining, we use a progressive supervised fine-tuning strategy that activates modality-specific experts and is enhanced by balanced data composition and an iterative GSPO-DPO method to stabilise RL training and improve reasoning. Data-wise, the base model, trained on approximately 75B tokens of open-source multimodal data, is equipped with special speech and image generation tokens, allowing it to learn these generative tasks by conditioning its outputs on linguistic cues. Extensive evaluation across 85 benchmarks demonstrates that our model achieves SOTA or highly competitive performance against leading OLMs, surpassing Qwen2.5-Omni (trained with 1.2T tokens) on over 50 of 76 benchmarks. Key strengths include video understanding (+7% avg. of 8), omnimodallity understanding (+7% avg. of 4), and audiovisual reasoning (+4%). It also advances long-form speech processing (reducing WER by 4.2%) and leads in low-level image processing and controllable generation across 5 metrics.