Paper 2510.15870
OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM
01 In brief
Summary
OmniVinci is an open-source omni-modal LLM from NVIDIA that jointly understands vision, audio, and text.
The paper introduces three architectural innovations: OmniAlignNet, which aligns vision and audio embeddings in a shared latent space via contrastive learning; Temporal Embedding Grouping (TEG), which organizes embeddings by timestamps to capture relative temporal order; and Constrained Rotary Time Embedding (CRTE), which encodes absolute timestamps with a constrained frequency scheme.
The authors curate a 24M-sample dataset, including implicit learning from video QA and explicit learning from a synthetic data engine that corrects modality-specific hallucinations.
OmniVinci outperforms Qwen2.5-Omni by +19.05 on DailyOmni, +1.7 on MMAR, and +3.9 on Video-MME, using only 0.2T training tokens (6x fewer).
Ablations show each component improves performance, and GRPO post-training with audio input yields faster convergence.
The model also demonstrates gains in downstream tasks like robotics navigation, sports video understanding, speech translation, medical AI, and smart factory monitoring.
02 From the paper
Abstract
Advancing machine intelligence requires developing the ability to perceive across multiple modalities, much as humans sense the world. We introduce OmniVinci, an initiative to build a strong, open-source, omni-modal LLM. We carefully study the design choices across model architecture and data curation. For model architecture, we present three key innovations: (i) OmniAlignNet for strengthening alignment between vision and audio embeddings in a shared omni-modal latent space; (ii) Temporal Embedding Grouping for capturing relative temporal alignment between vision and audio signals; and (iii) Constrained Rotary Time Embedding for encoding absolute temporal information in omni-modal embeddings. We introduce a curation and synthesis pipeline that generates 24M single-modal and omni-modal conversations. We find that modalities reinforce one another in both perception and reasoning. Our model, OmniVinci, outperforms Qwen2.5-Omni with +19.05 on DailyOmni (cross-modal understanding), +1.7 on MMAR (audio), and +3.9 on Video-MME (vision), while using just 0.2T training tokens - a 6 times reduction compared to Qwen2.5-Omni's 1.2T. We finally demonstrate omni-modal advantages in downstream applications spanning robotics, medical AI, and smart factory.