The Year in AI Papers
The year/Independent research

Paper 2510.13344

UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity MoE

Zhenyu Liu, Yunxin Li, Xuanyu Zhang, Qixun Teng, Shenyuan Jiang, Xinyu Chen, Haoyuan Shi, Jinchao Li, Qi Wang, Haolan Chen, Fanbo Meng, Mingjun Zhao, et al.

Published
Oct 2025
Research lab
Independent
Citations
6
GitHub
Not linked

01 In brief

Summary

UniMoE-Audio is a unified speech and music generation model that addresses task conflict and data imbalance through a Dynamic-Capacity Mixture-of-Experts (MoE) framework.

The architecture introduces a Top-P routing strategy for dynamic expert allocation and a hybrid expert design with routed, shared, and null experts.

To handle data imbalance, a three-stage training curriculum is used: 1) Independent Specialist Training on raw imbalanced data, 2) MoE Integration and Warmup on a balanced subset, and 3) Synergistic Joint Training on the full balanced dataset.

Experiments show UniMoE-Audio achieves state-of-the-art performance on speech benchmarks (e.g., UTMOS 4.36 on SeedTTS-EN) and superior aesthetic quality in music generation, outperforming a dense baseline (Unify-Baseline) and mitigating performance degradation seen in naive joint training.

The model uses 280K hours of speech data, achieving data efficiency rivaling models trained on millions of hours.

Analysis reveals balanced expert utilization, clear task specialization (Experts 1-4 for speech, 5-8 for music), and dynamic computation allocation with a rise-and-fall pattern across layers.

02 From the paper

Abstract

Recent advances in unified multimodal models indicate a clear trend towards comprehensive content generation. However, the auditory domain remains a significant challenge, with music and speech often developed in isolation, hindering progress towards universal audio synthesis. This separation stems from inherent task conflicts and severe data imbalances, which impede the development of a truly unified audio generation model. To address this challenge, we propose UniMoE-Audio, a unified speech and music generation model within a novel Dynamic-Capacity Mixture-of-Experts (MoE) framework. Architecturally, UniMoE-Audio introduces a Top-P routing strategy for dynamic expert number allocation, and a hybrid expert design comprising routed experts for domain-specific knowledge, shared experts for domain-agnostic features, and null experts for adaptive computation skipping. To tackle data imbalance, we introduce a three-stage training curriculum: 1) Independent Specialist Training leverages original datasets to instill domain-specific knowledge into each "proto-expert" without interference; 2) MoE Integration and Warmup incorporates these specialists into the UniMoE-Audio architecture, warming up the gate module and shared expert using a subset of balanced dataset; and 3) Synergistic Joint Training trains the entire model end-to-end on the fully balanced dataset, fostering enhanced cross-domain synergy. Extensive experiments show that UniMoE-Audio not only achieves state-of-the-art performance on major speech and music generation benchmarks, but also demonstrates superior synergistic learning, mitigating the performance degradation typically seen in naive joint training. Our findings highlight the substantial potential of specialized MoE architecture and curated training strategies in advancing the field of universal audio generation. Homepage: https://mukioxun.github.io/Uni-MoE-site/home.html