The Year in AI Papers
The year/Independent research

Paper 2605.18678

Lance: Unified Multimodal Modeling by Multi-Task Synergy

Fengyi Fu, Mengqi Huang, Shaojin Wu, Yunsheng Jiang, Yufei Huo, Hao Li, Yinghang Song, Fei Ding, Jianzhu Guo, Qian He, Zheren Fu, Zhendong Mao, et al.

Published
May 2026
Research lab
Independent
Citations
3
GitHub
1.3K stars

01 In brief

Summary

Lance is a lightweight native unified multimodal model from ByteDance that supports understanding, generation, and editing for both images and videos.

It uses a dual-stream mixture-of-experts architecture on shared interleaved multimodal sequences, combining autoregressive language modeling for understanding with flow matching for generation.

A modality-aware rotary positional encoding (MaPE) reduces interference among heterogeneous visual tokens.

Training follows a staged multi-task paradigm (PT, CT, SFT, RL) with capability-oriented objectives and adaptive data scheduling.

With only 3B activated parameters and a 128-GPU budget, Lance outperforms existing open-source unified models on image and video generation benchmarks (e.g., GenEval 0.90, VBench 85.11) while maintaining strong understanding (MVBench 62.0) and editing (GEdit 7.30).

Ablations show that multi-task data synergy and MaPE improve performance across tasks.

02 From the paper

Abstract

We present Lance, a lightweight native unified model supporting multimodal understanding, generation, and editing for both images and videos. Rather than relying on model capacity scaling or text-image-dominant designs, Lance explores a practical paradigm for unified multimodal modeling via collaborative multi-task training. It is grounded in two core principles: unified context modeling and decoupled capability pathways. Specifically, Lance is trained from scratch and employs a dual-stream mixture-of-experts architecture on shared interleaved multimodal sequences, enabling joint context learning while decoupling the pathways for understanding and generation. We further introduce modality-aware rotary positional encoding to mitigate interference among heterogeneous visual tokens and boost cross-task alignment. During training, Lance adopts a staged multi-task training paradigm with capability-oriented objectives and adaptive data scheduling to strengthen both semantic comprehension and visual generation performance. Experimental results demonstrate that Lance substantially outperforms existing open-source unified models in image and video generation, while retaining strong multimodal understanding capabilities. The homepage is available at https://lance-project.github.io.