The year/Topics/Multimodal

Topic area

Multimodal

Every collection across multimodal.

Papers
144
Research labs
9
Official code
109

101144 of 144 papers in this topic area

101

arXiv.org

A Style is Worth One Code: Unlocking Code-to-Style Image Generation with Discrete Style Space

The paper introduces CoTyle, the first open-source framework for code-to-style image generation, where a numerical style code controls the visual style of generated images. Existing methods rely on textual prompts, reference images, or LoRA fine-tuning, which suffer from poor consistency, limited creativity, or complex representations. CoTyle trains a…

Huijie Liu, Shuhao Cui, Haoxiang Cao, Shuai Ma, et al.
Published
Nov 2025
Citations
0
Code
222 stars
102

arXiv.org

VCode: a Multimodal Coding Benchmark with SVG as Symbolic Visual Representation

The paper introduces VCode, a benchmark that reframes multimodal understanding as visual-centric code generation: given an image, a model must produce SVG code that preserves symbolic meaning for downstream reasoning. VCode covers three domains: general commonsense (MM-Vet), professional disciplines (MMMU), and visual-centric perception (CV-Bench),…

Kevin Qinghong Lin, Yuhao Zheng, Hangyu Ran, Dantong Zhu, et al.
Published
Nov 2025
Citations
12
Code
135 stars
103

arXiv.org

Emu3.5: Native Multimodal Models are World Learners

Emu3.5 is a large-scale multimodal world model from BAAI that natively predicts the next state across vision and language. It is pre-trained end-to-end with a unified next-token prediction objective on over 10 trillion tokens from interleaved vision-language data, primarily derived from internet videos. The model accepts and generates interleaved…

Yufeng Cui, Honghao Chen, Haoge Deng, Xu Huang, et al.
Published
Oct 2025
Citations
109
Code
1.5K stars
104

arXiv.org

Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations

Concerto is a self-supervised learning framework that combines 3D intra-modal self-distillation with 2D-3D cross-modal joint embedding prediction to learn spatial representations from point clouds and images. It outperforms standalone 2D (DINOv2) and 3D (Sonata) self-supervised models by 14.2% and 4.8% respectively in linear probing for 3D scene…

Yujia Zhang, Xiaoyang Wu, Yixing Lao, Chengyao Wang, et al.
Published
Oct 2025
Citations
29
Code
532 stars
105

arXiv.org

FARMER: Flow AutoRegressive Transformer over Pixels

FARMER is a novel end-to-end generative framework that unifies Normalizing Flows (NF) and Autoregressive (AR) models for tractable likelihood estimation and high-quality image synthesis directly from raw pixels. It employs an invertible autoregressive flow to transform images into latent sequences, whose distribution is modeled implicitly by an AR model.…

Guangting Zheng, Qinyu Zhao, Tao Yang, Fei Xiao, et al.
Published
Oct 2025
Citations
13
Code
Not linked
106

arXiv.org

UniGenBench++: A Unified Semantic Evaluation Benchmark for Text-to-Image Generation

UniGenBench++ is a unified semantic evaluation benchmark for text-to-image (T2I) generation, addressing limitations in existing benchmarks such as coarse evaluation dimensions, lack of prompt diversity, and missing multilingual support. It comprises 600 prompts hierarchically organized into 5 primary themes and 20 subthemes, covering 10 primary and 27…

Yibin Wang, Zhimin Li, Yuhang Zang, Jiazi Bu, et al.
Published
Oct 2025
Citations
20
Code
139 stars
107

DeepSeek

DeepSeek-OCR: Contexts Optical Compression

DeepSeek-OCR is a vision-language model that explores optical compression of long contexts by mapping text to images and decoding them back to text. It consists of DeepEncoder, a novel vision encoder with low activation and high compression, and a DeepSeek3B-MoE decoder. On the Fox benchmark, it achieves 97% OCR precision at compression ratios under 10x,…

Haoran Wei, Yaofeng Sun, Yukun Li
Published
Oct 2025
Citations
159
Code
24K stars
108

arXiv.org

PICABench: How Far Are We from Physically Realistic Image Editing?

PICABench is a benchmark for evaluating physical realism in image editing, addressing the gap where existing models and benchmarks focus on semantic fidelity but overlook physical effects like shadows, reflections, and state transitions. It categorizes physical consistency into three dimensions—Optics, Mechanics, and State Transition—spanning eight…

Yuandong Pu, Le Zhuo, Songhao Han, Jinbo Xing, et al.
Published
Oct 2025
Citations
10
Code
39 stars
109

NVIDIA

OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM

OmniVinci is an open-source omni-modal LLM from NVIDIA that jointly understands vision, audio, and text. The paper introduces three architectural innovations: OmniAlignNet, which aligns vision and audio embeddings in a shared latent space via contrastive learning; Temporal Embedding Grouping (TEG), which organizes embeddings by timestamps to capture…

Hanrong Ye, Chao-Han Huck Yang, Arushi Goel, Wei Huang, et al.
Published
Oct 2025
Citations
50
Code
675 stars
110

arXiv.org

From Pixels to Words -- Towards Native Vision-Language Primitives at Scale

This paper introduces NEO, a family of native Vision-Language Models (VLMs) that unify vision and language processing in a single monolithic architecture, contrasting with modular VLMs that combine separate visual encoders and language models. NEO is built from first principles using three key primitives: flexible position encoding, Multi-Head Native…

Haiwen Diao, Mingxuan Li, Silei Wu, Linjun Dai, et al.
Published
Oct 2025
Citations
12
Code
881 stars
111

arXiv.org

WithAnyone: Towards Controllable and ID Consistent Image Generation

The paper introduces WithAnyone, a diffusion-based model for controllable and identity-consistent image generation, addressing the 'copy-paste' artifact where models replicate reference faces instead of preserving identity across variations. The authors construct MultiID-2M, a large-scale dataset with 500k group photos and paired references for 3k…

Hengyuan Xu, Wei Cheng, Peng Xing, Yixiao Fang, et al.
Published
Oct 2025
Citations
14
Code
573 stars
112

arXiv.org

PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model

PaddleOCR-VL is a state-of-the-art, resource-efficient document parsing solution from Baidu's PaddlePaddle team. It uses a two-stage architecture: PP-DocLayoutV2 for layout analysis (detection, classification, reading order) and PaddleOCR-VL-0.9B, a compact vision-language model combining a NaViT-style dynamic resolution visual encoder with the…

Cheng Cui, Ting Sun, Suyin Liang, Tingquan Gao, et al.
Published
Oct 2025
Citations
69
Code
Not linked
113

arXiv.org

Detect Anything via Next Point Prediction

Rex-Omni is a 3B-parameter multimodal large language model (MLLM) that unifies object detection and other visual perception tasks under a next-point prediction framework. It uses quantized relative coordinates represented by 1,000 special tokens, reducing token usage per bounding box to four. Training combines supervised fine-tuning on 22 million samples…

Qing Jiang, Junan Huo, Xingyu Chen, Yuda Xiong, et al.
Published
Oct 2025
Citations
60
Code
1.5K stars
114

Independent research

There is No VAE: End-to-End Pixel-Space Generative Modeling via Self-Supervised Pre-training

The paper introduces EPG, a two-stage training framework for pixel-space diffusion and consistency models that closes the performance gap with latent-space methods. Stage one pre-trains an encoder using a contrastive loss and a representation consistency loss, aligning features of noisy images along deterministic sampling trajectories. Stage two fine-tunes…

Jiachen Lei, Keli Liu, Julius Berner, Haiming Yu, et al.
Published
Oct 2025
Citations
15
Code
153 stars
115

Neural Information Processing Systems

Scaling Language-Centric Omnimodal Representation Learning

This paper introduces LCO-EMB, a language-centric framework for multimodal representation learning that leverages the implicit cross-modal alignment in multimodal large language models (MLLMs). The authors show that text-only contrastive fine-tuning of the language decoder reduces anisotropy and improves kernel-level similarity across image, audio, and…

Chenghao Xiao, Hou Pong Chan, Hao Zhang, Weiwen Xu, et al.
Published
Oct 2025
Citations
15
Code
48 stars
116

arXiv.org

Diffusion Transformers with Representation Autoencoders

The paper introduces Representation Autoencoders (RAEs), which replace the traditional VAE in latent diffusion models with a frozen pretrained representation encoder (e.g., DINOv2, SigLIP2, MAE) paired with a trained decoder. RAEs provide both high-quality reconstructions and semantically rich latent spaces, outperforming SD-VAE in reconstruction FID…

Boyang Zheng, Nanye Ma, Shengbang Tong, Saining Xie
Published
Oct 2025
Citations
221
Code
2K stars
117

arXiv.org

Thinking with Camera: A Unified Multimodal Model for Camera-Centric Understanding and Generation

The paper introduces Puffin, a unified multimodal model that jointly performs camera-centric understanding and generation. It treats camera parameters as a first-class modality, introducing 'thinking with camera' to align spatially grounded visual cues with professional photographic terms. Puffin integrates language regression and diffusion-based…

Kang Liao, Size Wu, Zhonghua Wu, Linyi Jin, et al.
Published
Oct 2025
Citations
15
Code
420 stars
118

arXiv.org

UniVideo: Unified Understanding, Generation, and Editing for Videos

UniVideo is a unified multimodal framework for video understanding, generation, and editing, presented at ICLR 2026. It uses a dual-stream architecture combining a frozen Multimodal Large Language Model (MLLM, Qwen2.5VL-7B) for understanding and a Multimodal DiT (MMDiT, HunyuanVideo-T2V-13B) for generation, connected via a trainable MLP connector. This…

Cong Wei, Quande Liu, Zixuan Ye, Qiulin Wang, et al.
Published
Oct 2025
Citations
68
Code
546 stars
119

arXiv.org

DreamOmni2: Multimodal Instruction-based Editing and Generation

DreamOmni2 introduces two new tasks: multimodal instruction-based editing and generation, which accept both text and image instructions and support concrete objects as well as abstract attributes (e.g., texture, pose, style). The authors propose a three-stage data synthesis pipeline: (1) a feature mixing scheme in attention to generate paired images…

Bin Xia, Bohao Peng, Yuechen Zhang, Junjia Huang, et al.
Published
Oct 2025
Citations
34
Code
2K stars
120

arXiv.org

Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizer

Ming-UniVision introduces MingTok, a continuous visual tokenizer with a three-stage architecture (low-level encoder, semantic decoder, pixel decoder) that unifies image understanding and generation in a single autoregressive framework. Unlike discrete tokenizers, MingTok avoids quantization errors, balancing compact latents for generation with…

Ziyuan Huang, DanDan Zheng, Cheng Zou, Rui Liu, et al.
Published
Oct 2025
Citations
26
Code
143 stars
121

arXiv.org

Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understanding

Lumina-DiMOO is an open-source, 8B-parameter unified multi-modal model based on a fully discrete diffusion architecture, built on the LLaDA-Base dLLM. It supports text-to-image generation at arbitrary resolutions, image-to-image tasks (editing, style transfer, subject-driven, controllable, multi-view), and image understanding. The model uses a unified…

Yi Xin, Qi Qin, Siqi Luo, Kaiwen Zhu, et al.
Published
Oct 2025
Citations
87
Code
1K stars
122

arXiv.org

Apriel-1.5-15b-Thinker

Apriel-1.5-15B-Thinker is a 15-billion-parameter open-weights multimodal reasoning model developed by ServiceNow's SLAM Lab. It achieves frontier-level performance through a data-centric mid-training approach rather than scale, starting from Pixtral-12B. The methodology involves three stages: depth upscaling to expand reasoning capacity, staged continual…

Shruthan Radhakrishna, Aman Tiwari, Aanjaneya Shukla, Masoud Hashemi, et al.
Published
Oct 2025
Citations
2
Code
Not linked
123

arXiv.org

OpenGPT-4o-Image: A Comprehensive Dataset for Advanced Image Generation and Editing

OpenGPT-4o-Image is a large-scale dataset for training unified multimodal models in image generation and editing. It contains 80,000 instruction-image pairs across 11 major domains and 51 subtasks, built using a hierarchical taxonomy and an automated pipeline that leverages GPT-4o. The taxonomy covers five generation modules (Style Control, Complex…

Zhihong Chen, Xuehai Bai, Yang Shi, Chaoyou Fu, et al.
Published
Sep 2025
Citations
24
Code
33 stars
124

arXiv.org

StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs

StableToken is a semantic speech tokenizer designed to be robust to acoustic noise, addressing the fragility of existing VQ-based tokenizers that produce unstable token sequences under perturbations. The paper identifies two flaws: a brittle single-path quantization architecture and a distant ASR training signal that ignores intermediate token stability.…

Yuhan Song, Linhao Zhang, Chuhan Wu, Aiwei Liu, et al.
Published
Sep 2025
Citations
4
Code
33 stars
125

arXiv.org

MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing

MinerU2.5 is a 1.2B-parameter vision-language model for document parsing that achieves state-of-the-art accuracy with high efficiency. It uses a decoupled, two-stage strategy: first, global layout analysis on a downsampled 1036x1036 image; second, targeted content recognition on native-resolution crops guided by the layout. The model uses a 675M NaViT…

Junbo Niu, Zheng Liu, Zhuangcheng Gu, Bin Wang, et al.
Published
Sep 2025
Citations
86
Code
77K stars
126

arXiv.org

Seedream 4.0: Toward Next-generation Multimodal Image Generation

Seedream 4.0 is a multimodal image generation system by ByteDance Seed that unifies text-to-image (T2I) synthesis, image editing, and multi-image composition in a single framework. It uses an efficient diffusion transformer (DiT) with a high-compression VAE, reducing image tokens and enabling native 1K-4K resolution generation. The model is pretrained on…

Team Seedream, :, Yunpeng Chen, Yu Gao, et al.
Published
Sep 2025
Citations
226
Code
Not linked
127

Qwen

Qwen3-Omni Technical Report

Qwen3-Omni is a single multimodal model that achieves state-of-the-art performance across text, image, audio, and video without degradation relative to single-modal counterparts. It matches the performance of same-sized Qwen single-modal models and excels on audio tasks, achieving open-source SOTA on 32 of 36 audio benchmarks and overall SOTA on 22,…

Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, et al.
Published
Sep 2025
Citations
405
Code
3.9K stars
128

arXiv.org

MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer

Manzano is a unified multimodal LLM that integrates image understanding and generation using a hybrid vision tokenizer. A single shared vision encoder feeds two lightweight adapters: a continuous adapter for understanding tasks and a discrete adapter (using FSQ with a 64K codebook) for generation, producing tokens in a common semantic space. A unified…

Yanghao Li, Rui Qian, Bowen Pan, Haotian Zhang, et al.
Published
Sep 2025
Citations
15
Code
Not linked
129

arXiv.org

Baseer: A Vision-Language Model for Arabic Document-to-Markdown OCR

The paper introduces Baseer, a vision-language model fine-tuned for Arabic document OCR, and Misraj-DocOCR, a benchmark for evaluation. Baseer is built on Qwen2.5-VL-3B-Instruct and trained on 500,000 image-text pairs (300k synthetic, 200k real-world) using a decoder-only fine-tuning strategy that freezes the vision encoder. The authors also corrected the…

Khalil Hennara, Muhammad Hreden, Mohamed Motasim Hamed, Ahmad Bastati, et al.
Published
Sep 2025
Citations
2
Code
Not linked
130

arXiv.org

MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe

MiniCPM-V 4.5 is an 8B-parameter multimodal large language model (MLLM) designed for high efficiency and strong performance. It introduces three core improvements: a unified 3D-Resampler architecture that compresses images and videos into compact token sequences (up to 16x for images and an additional 6x for videos), a unified learning paradigm for…

Tianyu Yu, Zefan Wang, Chongyi Wang, Fuwei Huang, et al.
Published
Sep 2025
Citations
124
Code
26K stars
131

arXiv.org

EchoX: Towards Mitigating Acoustic-Semantic Gap via Echo Training for Speech-to-Speech LLMs

EchoX is a framework for speech-to-speech large language models (SLLMs) that addresses the acoustic-semantic gap, which causes knowledge and reasoning degradation compared to text-based LLMs. The authors propose a three-stage training process: (1) speech-to-text training, (2) text-to-codec training, and (3) Echo training, where a frozen text-to-codec…

Yuhao Zhang, Yuhao Du, Zhanchen Dai, Xiangnan Ma, et al.
Published
Sep 2025
Citations
3
Code
47 stars
132

arXiv.org

Visual Representation Alignment for Multimodal Large Language Models

Multimodal large language models (MLLMs) trained with visual instruction tuning underperform on vision-centric tasks like object counting and spatial reasoning. The authors attribute this to text-only supervision, which provides indirect guidance for the visual pathway, causing MLLMs to discard fine-grained visual details from the vision encoder. They…

Heeji Yoon, Jaewoo Jung, Junwan Kim, Hyungyu Choi, et al.
Published
Sep 2025
Citations
35
Code
162 stars
133

arXiv.org

VibeVoice Technical Report

VIBEVOICE is a novel model from Microsoft Research for synthesizing long-form, multi-speaker speech (up to 90 minutes, 4 speakers) using next-token diffusion. It introduces a causal speech tokenizer with a 3200x compression rate (7.5 Hz frame rate), which is 80 times more efficient than Encodec while maintaining comparable performance. The model uses a…

Zhiliang Peng, Jianwei Yu, Wenhui Wang, Yaoyao Chang, et al.
Published
Aug 2025
Citations
36
Code
52K stars
134

arXiv.org

USO: Unified Style and Subject-Driven Generation via Disentangled and Reward Learning

The paper introduces USO, a unified framework for style-driven and subject-driven image generation, arguing these tasks are complementary and can be jointly optimized through cross-task co-disentanglement. The authors construct a large-scale triplet dataset (content, style, stylized images) using a subject-for-style curation framework with stylization and…

Shaojin Wu, Mengqi Huang, Yufeng Cheng, Wenxu Wu, et al.
Published
Aug 2025
Citations
29
Code
1.2K stars
135

arXiv.org

InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency

InternVL3.5 is a new family of open-source multimodal models that improves versatility, reasoning, and efficiency over its predecessor InternVL3. Key innovations include Cascade Reinforcement Learning (Cascade RL), which combines offline RL (MPO) for stable convergence and online RL (GSPO) for refined alignment, boosting reasoning performance by up to…

Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, et al.
Published
Aug 2025
Citations
1.2K
Code
Not linked
136

Independent research

Intern-S1: A Scientific Multimodal Foundation Model

Intern-S1 is a multimodal Mixture-of-Experts (MoE) foundation model with 28 billion activated parameters (241B total), designed to bridge the gap between open-source and closed-source models in scientific domains. It was continually pre-trained on 5T tokens, including over 2.5T scientific tokens, using pipelines for PDF parsing, domain-centric web data,…

Lei Bai, Zhongrui Cai, Yuhang Cao, Maosong Cao, et al.
Published
Aug 2025
Citations
13
Code
842 stars
137

arXiv.org

Thyme: Think Beyond Images

The paper introduces Thyme (Think Beyond Images), a paradigm for multimodal large language models (MLLMs) that autonomously generates and executes code for image manipulations (cropping, rotation, contrast enhancement) and complex computations, going beyond existing 'think with images' approaches. Thyme uses a two-stage training: Supervised Fine-Tuning…

Yi-Fan Zhang, Xingyu Lu, Shukang Yin, Chaoyou Fu, et al.
Published
Aug 2025
Citations
119
Code
586 stars
138

arXiv.org

Ovis2.5 Technical Report

Ovis2.5 is a multimodal large language model from Alibaba Group, succeeding Ovis2. It introduces native-resolution visual perception using a NaViT vision transformer, which processes images at their original resolutions to preserve fine details and global layout, crucial for complex charts. The model also adds an optional 'thinking mode' for deep…

Shiyin Lu, Yang Li, Yu Xia, Yuwei Hu, et al.
Published
Aug 2025
Citations
81
Code
1.5K stars
139

arXiv.org

NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scale

NextStep-1 is a 14B autoregressive model for text-to-image generation that uses continuous image tokens instead of discrete VQ tokens, paired with a 157M flow matching head. It is initialized from Qwen2.5-14B and trained with next-token prediction, combining cross-entropy loss for text and flow matching loss for images. The image tokenizer, fine-tuned from…

NextStep Team, Chunrui Han, Guopeng Li, Jingwei Wu, et al.
Published
Aug 2025
Citations
52
Code
694 stars
140

Meta AI

DINOv3

DINOv3 is a self-supervised vision foundation model that scales to 7B parameters, trained on a curated dataset of 1.689 billion images (LVD-1689M) from Instagram, combined with ImageNet-1k. The model uses a ViT architecture with RoPE embeddings and constant hyperparameter schedules. A key contribution is Gram anchoring, a regularization technique that…

Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, et al.
Published
Aug 2025
Citations
1.2K
Code
11K stars
141

arXiv.org

Story2Board: A Training-Free Approach for Expressive Storyboard Generation

Story2Board is a training-free framework for generating expressive storyboards from natural language. It introduces two mechanisms: Latent Panel Anchoring (LPA), which pairs each panel with a shared reference prompt to maintain character consistency, and Reciprocal Attention Value Mixing (RAVM), which blends value vectors between semantically aligned…

David Dinkevich, Matan Levy, Omri Avrahami, Dvir Samuel, et al.
Published
Aug 2025
Citations
12
Code
266 stars
142

ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia

Voost: A Unified and Scalable Diffusion Transformer for Bidirectional Virtual Try-On and Try-Off

Voost is a unified and scalable diffusion transformer framework that jointly learns virtual try-on and try-off using a single model. It uses a token-level concatenation of garment and person images, with a task token encoding generation direction and garment category, enabling bidirectional supervision without task-specific networks or auxiliary losses.…

Seungyong Lee, Jeong-gi Kwak
Published
Aug 2025
Citations
12
Code
340 stars
143

arXiv.org

Skywork UniPic: Unified Autoregressive Modeling for Visual Understanding and Generation

Skywork UniPic is a 1.5B-parameter autoregressive model that unifies image understanding, text-to-image generation, and image editing in a single architecture without task-specific adapters. It achieves a GenEval score of 0.86, a DPG-Bench score of 85.5, and GEditBench-EN and ImgEdit-Bench scores of 5.83 and 3.49, respectively, while generating 1024x1024…

Peiyu Wang, Yi Peng, Yimeng Gan, Liang Hu, et al.
Published
Aug 2025
Citations
17
Code
871 stars
144

Qwen

Qwen-Image Technical Report

Qwen-Image is a new image generation foundation model from the Qwen team, designed to excel in complex text rendering and precise image editing. It uses a comprehensive data pipeline and a progressive curriculum learning strategy to enhance text rendering, especially for Chinese. For editing, it employs a multi-task training paradigm with dual-encoding…

Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, et al.
Published
Aug 2025
Citations
875
Code
8.2K stars