The year/Topics/Multimodal

Topic area

Multimodal

Every collection across multimodal.

Papers
144
Research labs
9
Official code
109

1–50 of 144 papers in this topic area

01

Meta AI

DINOv3

DINOv3 is a self-supervised vision foundation model that scales to 7B parameters, trained on a curated dataset of 1.689 billion images (LVD-1689M) from Instagram, combined with ImageNet-1k. The model uses a ViT architecture with RoPE embeddings and constant hyperparameter schedules. A key contribution is Gram anchoring, a regularization technique that…

Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, et al.
Published
Aug 2025
Citations
1.2K
Code
11K stars
02

Qwen

Qwen-Image Technical Report

Qwen-Image is a new image generation foundation model from the Qwen team, designed to excel in complex text rendering and precise image editing. It uses a comprehensive data pipeline and a progressive curriculum learning strategy to enhance text rendering, especially for Chinese. For editing, it employs a multi-task training paradigm with dual-encoding…

Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, et al.
Published
Aug 2025
Citations
875
Code
8.2K stars
03

Independent research

Intern-S1: A Scientific Multimodal Foundation Model

Intern-S1 is a multimodal Mixture-of-Experts (MoE) foundation model with 28 billion activated parameters (241B total), designed to bridge the gap between open-source and closed-source models in scientific domains. It was continually pre-trained on 5T tokens, including over 2.5T scientific tokens, using pipelines for PDF parsing, domain-centric web data,…

Lei Bai, Zhongrui Cai, Yuhang Cao, Maosong Cao, et al.
Published
Aug 2025
Citations
13
Code
842 stars
04

arXiv.org

CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence

CiteVQA is a benchmark for evaluating Multimodal Large Language Models (MLLMs) on document visual question answering, requiring both correct answers and element-level bounding-box citations. It includes 1,897 questions from 711 PDFs across seven domains and two languages, with an average of 40.6 pages per document. Ground-truth citations are generated via…

Dongsheng Ma, Jiayu Li, Zhengren Wang, Yijie Wang, et al.
Published
May 2026
Citations
3
Code
69 stars
05

arXiv.org

ERNIE 5.0 Technical Report

ERNIE 5.0 is a natively autoregressive foundation model from Baidu for unified multimodal understanding and generation across text, image, video, and audio. All modalities are trained from scratch under a unified next-group-of-tokens prediction objective, using an ultra-sparse mixture-of-experts (MoE) architecture with modality-agnostic expert routing. A…

Haifeng Wang, Hua Wu, Tian Wu, Yu Sun, et al.
Published
Feb 2026
Citations
8
Code
Not linked
06

arXiv.org

Tstars-Tryon 1.0: Robust and Realistic Virtual Try-On for Diverse Fashion Items

Tstars-Tryon 1.0, developed by the Pailitao Team at Alibaba Group, is a commercial-scale virtual try-on system designed for robustness, realism, versatility, and efficiency. It handles challenging real-world cases like extreme poses, lighting variations, and motion blur, while preserving garment details and avoiding synthetic artifacts. The system supports…

Mengting Chen, Zhengrui Chen, Yongchao Du, Zuan Gao, et al.
Published
Apr 2026
Citations
2
Code
Not linked
07

arXiv.org

Crafter: A Multi-Agent Harness for Editable Scientific Figure Generation from Diverse Inputs

The paper introduces CRAFTER, a multi-agent harness for generating scientific figures from diverse inputs, and CRAFTEDITOR, which converts raster outputs into editable SVGs. CRAFTER uses five agents (intent reasoner, plan generator, critic, specification refiner, convergence judge) sharing an evolving specification, with mechanisms for diversity-driven…

Haozhe Zhao, Shuzheng Si, Zhenhailong Wang, Zheng Wang, et al.
Published
May 2026
Citations
1
Code
149 stars
08

arXiv.org

Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer

Z-Image is a 6B-parameter image generation foundation model from Alibaba Group, built on a Scalable Single-Stream Diffusion Transformer (S3-DiT). It challenges the 'scale-at-all-costs' paradigm by optimizing data infrastructure, architecture, training, and inference. The full training workflow costs 314K H800 GPU hours (~$628K). Z-Image-Turbo, a distilled…

Z-Image Team, Huanqia Cai, Sihan Cao, Ruoyi Du, et al.
Published
Nov 2025
Citations
206
Code
12K stars
09

arXiv.org

Kandinsky 5.0: A Family of Foundation Models for Image and Video Generation

Kandinsky 5.0 is a family of foundation models for high-resolution image and video synthesis, comprising three model line-ups: Image Lite (6B parameters), Video Lite (2B), and Video Pro (19B). The models are built on a unified latent diffusion architecture with a CrossDiT backbone and trained using flow matching. Key innovations include the NABLA sparse…

Vladimir Arkhipkin, Vladimir Korviakov, Nikolai Gerasimenko, Denis Parkhomenko, et al.
Published
Nov 2025
Citations
9
Code
805 stars
10

arXiv.org

InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency

InternVL3.5 is a new family of open-source multimodal models that improves versatility, reasoning, and efficiency over its predecessor InternVL3. Key innovations include Cascade Reinforcement Learning (Cascade RL), which combines offline RL (MPO) for stable convergence and online RL (GSPO) for refined alignment, boosting reasoning performance by up to…

Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, et al.
Published
Aug 2025
Citations
1.2K
Code
Not linked
11

arXiv.org

BabyVision: Visual Reasoning Beyond Language

The paper introduces BABYVISION, a benchmark to evaluate core visual abilities in Multimodal LLMs (MLLMs) that are independent of linguistic knowledge, targeting skills humans develop before language. It contains 388 questions across 22 subtypes in four categories: Fine-grained Discrimination, Visual Tracking, Spatial Perception, and Visual Pattern…

Liang Chen, Weichu Xie, Yiyan Liang, Hongfeng He, et al.
Published
Jan 2026
Citations
24
Code
238 stars
12

arXiv.org

STEP3-VL-10B Technical Report

Step3-VL-10B is a 10B-parameter open-source multimodal foundation model that rivals or surpasses models 10-20x larger, such as GLM-4.6V-106B and Qwen3-VL-235B, and proprietary systems like Gemini 2.5 Pro. It achieves 92.2% on MMBench, 80.11% on MMMU, 94.43% on AIME2025, and 75.95% on MathVision. The model uses a unified pre-training strategy on 1.2T…

Ailin Huang, Chengyuan Yao, Chunrui Han, Fanqi Wan, et al.
Published
Jan 2026
Citations
26
Code
412 stars
13

arXiv.org

SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecture

SenseNova-U1 is a native unified multimodal model built on the NEO-unify architecture, designed to overcome the traditional divide between understanding and generation. It operates directly on raw pixels and text, eliminating the need for pretrained vision encoders (VEs) and variational autoencoders (VAEs). The model uses a near-lossless visual interface…

Haiwen Diao, Penghao Wu, Hanming Deng, Jiahao Wang, et al.
Published
May 2026
Citations
12
Code
4.5K stars
14

Independent research

Kwai Keye-VL-2.0 Technical Report

The report introduces Kwai Keye-VL-2.0-30B-A3B, an open-source Mixture-of-Experts (MoE) multimodal foundation model with 30B total parameters and 3B active, designed for long-video understanding and agentic intelligence. It is the first to adapt DeepSeek Sparse Attention (DSA) to GQA-based architectures, enabling lossless 256K context processing. The model…

Kwai Keye Team, Bin Wen, Changyi Liu, Chengru Song, et al.
Published
Jun 2026
Citations
0
Code
809 stars
15

arXiv.org

Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations

Concerto is a self-supervised learning framework that combines 3D intra-modal self-distillation with 2D-3D cross-modal joint embedding prediction to learn spatial representations from point clouds and images. It outperforms standalone 2D (DINOv2) and 3D (Sonata) self-supervised models by 14.2% and 4.8% respectively in linear probing for 3D scene…

Yujia Zhang, Xiaoyang Wu, Yixing Lao, Chengyao Wang, et al.
Published
Oct 2025
Citations
29
Code
532 stars
16

arXiv.org

HY-Embodied-0.5: Embodied Foundation Models for Real-World Agents

Tencent's HY-Embodied-0.5 is a family of vision-language foundation models designed for real-world embodied agents, bridging the gap between general VLMs and physical-world tasks. The suite includes an efficient 2B-activated-parameter model (MoT-2B) for edge deployment and a powerful 32B-activated-parameter model (MoE-A32B) for complex reasoning. Key…

Tencent Robotics X, HY Vision Team, :, Xumin Yu, et al.
Published
Apr 2026
Citations
12
Code
843 stars
17

arXiv.org

VibeVoice Technical Report

VIBEVOICE is a novel model from Microsoft Research for synthesizing long-form, multi-speaker speech (up to 90 minutes, 4 speakers) using next-token diffusion. It introduces a causal speech tokenizer with a 3200x compression rate (7.5 Hz frame rate), which is 80 times more efficient than Encodec while maintaining comparable performance. The model uses a…

Zhiliang Peng, Jianwei Yu, Wenhui Wang, Yaoyao Chang, et al.
Published
Aug 2025
Citations
36
Code
52K stars
18

arXiv.org

MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing

MinerU2.5 is a 1.2B-parameter vision-language model for document parsing that achieves state-of-the-art accuracy with high efficiency. It uses a decoupled, two-stage strategy: first, global layout analysis on a downsampled 1036x1036 image; second, targeted content recognition on native-resolution crops guided by the layout. The model uses a 675M NaViT…

Junbo Niu, Zheng Liu, Zhuangcheng Gu, Bin Wang, et al.
Published
Sep 2025
Citations
86
Code
77K stars
19

arXiv.org

Diffusion Transformers with Representation Autoencoders

The paper introduces Representation Autoencoders (RAEs), which replace the traditional VAE in latent diffusion models with a frozen pretrained representation encoder (e.g., DINOv2, SigLIP2, MAE) paired with a trained decoder. RAEs provide both high-quality reconstructions and semantically rich latent spaces, outperforming SD-VAE in reconstruction FID…

Boyang Zheng, Nanye Ma, Shengbang Tong, Saining Xie
Published
Oct 2025
Citations
221
Code
2K stars
20

Qwen

Qwen3-VL Technical Report

Qwen3-VL is a state-of-the-art vision-language model family from the Qwen team, released on December 1, 2025. It supports interleaved contexts up to 256K tokens and comes in dense (2B/4B/8B/32B) and MoE (30B-A3B/235B-A22B) variants. Key architectural innovations include interleaved-MRoPE for balanced spatial-temporal encoding, DeepStack for multi-level ViT…

Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, et al.
Published
Nov 2025
Citations
1.8K
Code
20K stars
21

arXiv.org

Urban Socio-Semantic Segmentation with Vision-Language Reasoning

The paper introduces urban socio-semantic segmentation, targeting entities defined by social attributes (e.g., schools, parks) rather than physical ones. The authors present SocioSeg, a benchmark with over 13,000 samples, organizing labels into three hierarchical tasks: socio-name, socio-class, and socio-function. A key innovation is representing…

Yu Wang, Yi Wang, Rui Dai, Yujie Wang, et al.
Published
Jan 2026
Citations
2
Code
175 stars
22

arXiv.org

Qianfan-OCR: A Unified End-to-End Model for Document Intelligence

Qianfan-OCR is a 4B-parameter end-to-end vision-language model that unifies document parsing, layout analysis, and understanding, outperforming all end-to-end models on OmniDocBench v1.5 (93.12) and OlmOCR Bench (79.8). It introduces Layout-as-Thought, an optional thinking phase triggered by ⟨think⟩ tokens that generates structured layout representations…

Daxiang Dong, Mingming Zheng, Dong Xu, Chunhua Luo, et al.
Published
Mar 2026
Citations
11
Code
421 stars
23

Qwen

Qwen3-Omni Technical Report

Qwen3-Omni is a single multimodal model that achieves state-of-the-art performance across text, image, audio, and video without degradation relative to single-modal counterparts. It matches the performance of same-sized Qwen single-modal models and excels on audio tasks, achieving open-source SOTA on 32 of 36 audio benchmarks and overall SOTA on 22,…

Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, et al.
Published
Sep 2025
Citations
405
Code
3.9K stars
24

arXiv.org

LongCat-Next: Lexicalizing Modalities as Discrete Tokens

LongCat-Next, developed by Meituan's LongCat team, introduces the Discrete Native Autoregressive (DiNA) paradigm, which unifies text, vision, and audio into a shared discrete token space, enabling a single autoregressive model to handle all modalities. A key innovation is the Discrete Native Any-resolution Visual Transformer (dNaViT), which uses…

Meituan LongCat Team, Bin Xiao, Chao Wang, Chengjiang Li, et al.
Published
Mar 2026
Citations
18
Code
467 stars
25

arXiv.org

NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scale

NextStep-1 is a 14B autoregressive model for text-to-image generation that uses continuous image tokens instead of discrete VQ tokens, paired with a 157M flow matching head. It is initialized from Qwen2.5-14B and trained with next-token prediction, combining cross-entropy loss for text and flow matching loss for images. The image tokenizer, fine-tuned from…

NextStep Team, Chunrui Han, Guopeng Li, Jingwei Wu, et al.
Published
Aug 2025
Citations
52
Code
694 stars
26

NVIDIA

LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding

LocateAnything is a unified vision-language model for visual grounding and detection that introduces Parallel Box Decoding (PBD). Unlike standard next-token prediction (NTP) which serializes bounding box coordinates into 1D token streams, PBD treats each bounding box as an atomic unit, predicting all its coordinates in a single forward pass. This…

Shihao Wang, Shilong Liu, Yuanguo Kuang, Xinyu Wei, et al.
Published
May 2026
Citations
7
Code
Not linked
27

arXiv.org

Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models

The paper addresses the Modality Gap in multimodal contrastive learning, where embeddings of different modalities for the same semantics occupy offset regions. Prior methods rely on isotropic assumptions, which are flawed. The authors propose the Fixed-frame Modality Gap Theory, decomposing the gap into stable biases (PMB, POB) and anisotropic residuals,…

Xiaomin Yu, Yi Xin, Yuhui Zhang, Wenjie Zhang, et al.
Published
Feb 2026
Citations
9
Code
76 stars
28

Independent research

Moebius: 0.2B Lightweight Image Inpainting Framework with 10B-Level Performance

Moebius is a highly efficient lightweight image inpainting framework that rivals the generation quality of 10B-level industrial models like FLUX.1-Fill-Dev while using only 0.22B parameters (less than 2% of FLUX's 11.9B) and delivering over 15× faster total inference time. To overcome the representation bottleneck from extreme structural compression, the…

Kangsheng Duan, Ziyang Xu, Wenyu Liu, Xiaohu Ruan, et al.
Published
Jun 2026
Citations
0
Code
508 stars
29

arXiv.org

MinerU-Diffusion: Rethinking Document OCR as Inverse Rendering via Diffusion Decoding

MinerU-Diffusion is a 2.5B-parameter diffusion-based framework for document OCR that replaces autoregressive (AR) decoding with block-wise parallel diffusion denoising under visual conditioning. The authors argue that left-to-right causal generation is an artifact of serialization, not intrinsic to OCR, and propose inverse rendering via diffusion. The…

Hejun Dong, Junbo Niu, Bin Wang, Weijun Zeng, et al.
Published
Mar 2026
Citations
7
Code
628 stars
30

Meta AI

SAM 3: Segment Anything with Concepts

SAM 3 is a unified model for promptable concept segmentation (PCS) in images and videos, accepting noun phrases, image exemplars, or both as prompts to detect, segment, and track all matching instances. It decouples recognition and localization via a presence head, improving detection accuracy. A data engine with human and AI verifiers produced 4M unique…

Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, et al.
Published
Nov 2025
Citations
711
Code
11K stars
31

Independent research

Boogu-Image-0.1: Boosting Open Agentic Multimodal Generation via Understanding under a Minimal Budget

Boogu-Image-0.1 is an open-source family of unified multimodal understanding and generation models (Base, Turbo, Edit, Edit-Turbo) that achieves competitive performance in text-to-image generation, fast inference, instruction-based editing, and bilingual text rendering. The authors argue that strengthening understanding—via a stronger multimodal encoder…

Guoxuan Chen, Chufeng Xiao, Haoran Yang, Siyue Xie, et al.
Published
Jul 2026
Citations
0
Code
Not linked
32

arXiv.org

Mega-ASR: Towards In-the-wild^2 Speech Recognition via Scaling up Real-world Acoustic Simulation

MEGA-ASR is a framework for automatic speech recognition (ASR) in real-world environments, addressing the 'acoustic robustness bottleneck' where models fail under severe, compositional distortions. The authors introduce VOICES-IN-THE-WILD-2M, a large-scale dataset with 7 atomic acoustic effects (noise, far-field, obstructed, echo&reverb, recording,…

Zhifei Xie, Kaiyu Pang, Haobin Zhang, Deheng Ye, et al.
Published
May 2026
Citations
1
Code
1.1K stars
33

arXiv.org

Baseer: A Vision-Language Model for Arabic Document-to-Markdown OCR

The paper introduces Baseer, a vision-language model fine-tuned for Arabic document OCR, and Misraj-DocOCR, a benchmark for evaluation. Baseer is built on Qwen2.5-VL-3B-Instruct and trained on 500,000 image-text pairs (300k synthetic, 200k real-world) using a decoder-only fine-tuning strategy that freezes the vision encoder. The authors also corrected the…

Khalil Hennara, Muhammad Hreden, Mohamed Motasim Hamed, Ahmad Bastati, et al.
Published
Sep 2025
Citations
2
Code
Not linked
34

arXiv.org

One Small Step in Latent, One Giant Leap for Pixels: Fast Latent Upscale Adapter for Your Diffusion Models

The paper introduces the Latent Upscaler Adapter (LUA), a lightweight module that performs super-resolution directly on a diffusion model's latent code before VAE decoding, enabling high-resolution image synthesis without retraining the generator or adding diffusion stages. LUA uses a shared SwinIR-style backbone with scale-specific pixel-shuffle heads for…

Aleksandr Razin, Danil Kazantsev, Ilya Makarov
Published
Nov 2025
Citations
1
Code
34 stars
35

arXiv.org

PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model

PaddleOCR-VL is a state-of-the-art, resource-efficient document parsing solution from Baidu's PaddlePaddle team. It uses a two-stage architecture: PP-DocLayoutV2 for layout analysis (detection, classification, reading order) and PaddleOCR-VL-0.9B, a compact vision-language model combining a NaViT-style dynamic resolution visual encoder with the…

Cheng Cui, Ting Sun, Suyin Liang, Tingquan Gao, et al.
Published
Oct 2025
Citations
69
Code
Not linked
36

arXiv.org

Thinking with Camera: A Unified Multimodal Model for Camera-Centric Understanding and Generation

The paper introduces Puffin, a unified multimodal model that jointly performs camera-centric understanding and generation. It treats camera parameters as a first-class modality, introducing 'thinking with camera' to align spatially grounded visual cues with professional photographic terms. Puffin integrates language regression and diffusion-based…

Kang Liao, Size Wu, Zhonghua Wu, Linyi Jin, et al.
Published
Oct 2025
Citations
15
Code
420 stars
37

Independent research

Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models

The paper introduces Imaginative Perception Tokens (IPTs), intermediate visual representations that externalize what a VLM would perceive under an alternative spatial configuration, to improve spatial reasoning. Three tasks requiring imaginative perception are formulated: Perspective Taking (PET), Path Tracing (PT), and Multiview Counting (MVC), with…

Mahtab Bigverdi, Linjie Li, Weikai Huang, Yiming Liu, et al.
Published
Jun 2026
Citations
0
Code
102 stars
38

arXiv.org

Apriel-1.5-15b-Thinker

Apriel-1.5-15B-Thinker is a 15-billion-parameter open-weights multimodal reasoning model developed by ServiceNow's SLAM Lab. It achieves frontier-level performance through a data-centric mid-training approach rather than scale, starting from Pixtral-12B. The methodology involves three stages: depth upscaling to expand reasoning capacity, staged continual…

Shruthan Radhakrishna, Aman Tiwari, Aanjaneya Shukla, Masoud Hashemi, et al.
Published
Oct 2025
Citations
2
Code
Not linked
39

arXiv.org

MinerU2.5-Pro: Pushing the Limits of Data-Centric Document Parsing at Scale

MinerU2.5-Pro improves document parsing purely through data engineering and training strategy, keeping the 1.2B-parameter architecture of MinerU2.5 unchanged. The authors identify that state-of-the-art models share failure patterns on hard samples, indicating a data bottleneck rather than an architectural one. They build a Data Engine with three…

Bin Wang, Tianyao He, Linke Ouyang, Fan Wu, et al.
Published
Apr 2026
Citations
14
Code
Not linked
40

Independent research

Audio Interaction Model

The paper introduces Audio-Interaction, a unified streaming audio language model that operates via an always-on perceive–decide–respond loop, listening to continuous audio and deciding when to respond or remain silent. It addresses limitations of offline LALMs and task-specific streaming models by unifying capabilities like real-time ASR, translation,…

Zhifei Xie, Zihang Liu, Ze An, Xiaobin Hu, et al.
Published
Jun 2026
Citations
0
Code
575 stars
41

arXiv.org

Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders

Penguin-VL, developed by Tencent AI Lab, introduces compact 2B and 8B vision-language models that challenge the reliance on contrastive pretraining (e.g., CLIP/SigLIP) for vision encoders. The authors argue that contrastive learning suppresses fine-grained visual cues needed for reasoning. Instead, they propose Penguin-Encoder, initialized from a text-only…

Boqiang Zhang, Lei Ke, Ruihan Yang, Qi Gao, et al.
Published
Mar 2026
Citations
4
Code
206 stars
42

Independent research

Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models

Wan-Streamer is a native-streaming, end-to-end interactive foundation model from Alibaba Group designed for real-time, low-latency, full-duplex audio-visual interaction. It models language, audio, and video as both input and output within a single Transformer, using block-causal attention for incremental streaming. Unlike cascaded systems, it does not rely…

Lianghua Huang, Zhi-Fan Wu, Wei Wang, Yupeng Shi, et al.
Published
Jun 2026
Citations
4
Code
Not linked
43

arXiv.org

PixelSmile: Toward Fine-Grained Facial Expression Editing

PixelSmile is a diffusion-based framework for fine-grained facial expression editing, addressing semantic overlap between expressions like fear-surprise and anger-disgust. The authors construct the Flex Facial Expression (FFE) dataset with 60,000 images (real and anime) annotated with continuous 12-dimensional affective scores, and establish FFE-Bench to…

Jiabin Hua, Hengyuan Xu, Aojie Li, Wei Cheng, et al.
Published
Mar 2026
Citations
0
Code
Not linked
44

Qwen

Qwen-Image-2.0 Technical Report

Qwen-Image-2.0 is an image generation foundation model that unifies text-to-image (T2I) generation and instruction-based image editing in a single framework. It addresses challenges in ultra-long text rendering (up to 1K tokens), multilingual typography, high-resolution photorealism (native 2K), complex instruction following, and inference efficiency. The…

Bing Zhao, Chenfei Wu, Deqing Li, Hao Meng, et al.
Published
May 2026
Citations
8
Code
Not linked
45

arXiv.org

Emu3.5: Native Multimodal Models are World Learners

Emu3.5 is a large-scale multimodal world model from BAAI that natively predicts the next state across vision and language. It is pre-trained end-to-end with a unified next-token prediction objective on over 10 trillion tokens from interleaved vision-language data, primarily derived from internet videos. The model accepts and generates interleaved…

Yufeng Cui, Honghao Chen, Haoge Deng, Xu Huang, et al.
Published
Oct 2025
Citations
109
Code
1.5K stars
46

arXiv.org

Ovis2.5 Technical Report

Ovis2.5 is a multimodal large language model from Alibaba Group, succeeding Ovis2. It introduces native-resolution visual perception using a NaViT vision transformer, which processes images at their original resolutions to preserve fine details and global layout, crucial for complex charts. The model also adds an optional 'thinking mode' for deep…

Shiyin Lu, Yang Li, Yu Xia, Yuwei Hu, et al.
Published
Aug 2025
Citations
81
Code
1.5K stars
47

Independent research

SwanTale: Unified Multi-Speaker Speech and Audio Generation for Instruct and Zero-Shot Tasks

SwanTale is a unified model for multi-speaker speech and audio generation supporting both zero-shot and instruct tasks. It introduces SwanData-Caption, a data pipeline that cleans raw audio, adds targeted synthetic coverage (elderly speech, short utterances, challenging pronunciations), and annotates multi-level captions (environment, speakers, content).…

Yu Zhang, Ruiqi Li, Changhao Pan, Ke Lei, et al.
Published
Aug 2026
Citations
0
Code
Not linked
48

Independent research

There is No VAE: End-to-End Pixel-Space Generative Modeling via Self-Supervised Pre-training

The paper introduces EPG, a two-stage training framework for pixel-space diffusion and consistency models that closes the performance gap with latent-space methods. Stage one pre-trains an encoder using a contrastive loss and a representation consistency loss, aligning features of noisy images along deterministic sampling trajectories. Stage two fine-tunes…

Jiachen Lei, Keli Liu, Julius Berner, Haiming Yu, et al.
Published
Oct 2025
Citations
15
Code
153 stars
49

Z.ai / GLM

GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents

GLM-5V-Turbo is a native multimodal foundation model for agentic tasks, integrating perception, reasoning, planning, and execution. It introduces CogViT, a vision encoder trained via distillation and contrastive learning, and Multimodal Multi-Token Prediction (MMTP) using a shared <|image|> token for efficiency. The model undergoes joint RL over 30+ task…

GLM-V Team, :, Wenyi Hong, Xiaotao Gu, et al.
Published
Apr 2026
Citations
11
Code
Not linked
50

arXiv.org

Rethinking Cross-Layer Information Routing in Diffusion Transformers

This paper investigates cross-layer information routing in Diffusion Transformers (DiTs), identifying three symptoms of standard residual connections: forward magnitude inflation, backward gradient decay, and block-wise redundancy. The authors propose Diffusion-Adaptive Routing (DAR), a drop-in replacement that uses learnable, timestep-adaptive,…

Chao Xu, Maohua Li, Qirui Li, Yixuan Xu, et al.
Published
May 2026
Citations
2
Code
Not linked