The Year in AI Papers
The year/Independent research

Paper 2601.21204

Scaling Embeddings Outperforms Scaling Experts in Language Models

Hong Liu, Jiaqi Zhang, Chao Wang, Xing Hu, Linkun Lyu, Jiaqi Sun, Xurui Yang, Bo Wang, Fengcun Li, Yulei Qian, Lingtong Si, Yerui Sun, et al.

Published
Jan 2026
Research lab
Independent
Citations
11
GitHub
Not linked

01 In brief

Summary

This technical report from Meituan's LongCat team investigates scaling embeddings as an alternative to scaling experts in Mixture-of-Experts (MoE) language models.

The authors find that N-gram Embedding, which augments token embeddings with hashed n-gram sub-tables, achieves a superior Pareto frontier compared to increasing expert count in specific regimes, particularly when the base model is highly sparse.

Key design principles include introducing N-gram Embedding only after experts reach a 'sweet spot', allocating no more than 50% of parameters to embeddings, and avoiding vocabulary sizes that are integer multiples of the base vocabulary to reduce hash collisions.

The advantage of embedding scaling grows with model width but diminishes with depth.

System optimizations like an N-gram Cache and kernel fusion, combined with speculative decoding, convert parameter sparsity into inference speedups.

The authors introduce LongCat-Flash-Lite, a 68.5B parameter model with ~3B activated parameters and 31.4B N-gram Embedding parameters, trained from scratch.

It outperforms a parameter-equivalent MoE baseline and shows competitive results against models like Qwen3-Next-80B-A3B and Gemini 2.5 Flash-Lite, especially in agentic tool use and coding tasks.

02 From the paper

Abstract

While Mixture-of-Experts (MoE) architectures have become the standard for sparsity scaling in large language models, they increasingly face diminishing returns and system-level bottlenecks. In this work, we explore embedding scaling as a potent, orthogonal dimension for scaling sparsity. Through a comprehensive analysis and experiments, we identify specific regimes where embedding scaling achieves a superior Pareto frontier compared to expert scaling. We systematically characterize the critical architectural factors governing this efficacy -- ranging from parameter budgeting to the interplay with model width and depth. Moreover, by integrating tailored system optimizations and speculative decoding, we effectively convert this sparsity into tangible inference speedups. Guided by these insights, we introduce LongCat-Flash-Lite, a 68.5B parameter model with ~3B activated trained from scratch. Despite allocating over 30B parameters to embeddings, LongCat-Flash-Lite not only surpasses parameter-equivalent MoE baselines but also exhibits exceptional competitiveness against existing models of comparable scale, particularly in agentic and coding domains.