The Year in AI Papers
The year/NVIDIA

Paper 2606.02800

Cosmos 3: Omnimodal World Models for Physical AI

NVIDIA, :, Aditi, Niket Agarwal, Arslan Ali, Jon Allen, Martin Antolini, Adeline Aubame, Alisson Azzolini, Junjie Bai, Maciej Bala, Yogesh Balaji, et al.

Published
Jun 2026
Research lab
NVIDIA
Citations
36
GitHub
Not linked

01 In brief

Summary

NVIDIA introduces Cosmos 3, a family of omnimodal world models that jointly process and generate language, image, video, audio, and action sequences within a unified mixture-of-transformers architecture.

It subsumes vision-language models, video generators, world simulators, and world-action models into a single framework, supporting flexible input-output configurations.

Evaluations show state-of-the-art results across understanding and generation tasks, with post-trained models ranking as best open-source Text-to-Image and Image-to-Video models by Artificial Analysis, and best policy model by RoboArena.

The models are available in Edge (4B), Nano (16B), and Super (64B) variants, initialized from pre-trained VLMs.

Training uses a reasoner-generator dual-tower design with autoregressive and diffusion pathways, and a progressive curriculum including pre-training, mid-training, and post-training stages.

The release includes code, checkpoints, synthetic datasets (SDG-PhyxSim, SDG-RobotSim, SDG-DriveSim, SDG-SynHuman, SDG-Warehouse), and the Cosmos-HUE benchmark under the OpenMDW-1.1 License.

Cosmos 3 aims to accelerate Physical AI research by providing a general-purpose backbone for embodied agents, enabling synthetic data generation, task-specific specialization, and training environments.

The paper details architecture, data, training, infrastructure, and results across robotics, autonomous driving, smart infrastructure, and general domains, demonstrating strong performance and adaptability for downstream applications.

02 From the paper

Abstract

We introduce Cosmos 3, a family of omnimodal world models designed to jointly process and generate language, image, video, audio, and action sequences within a unified mixture-of-transformers architecture. By supporting highly flexible input-output configurations, Cosmos 3 seamlessly unifies critical modalities for Physical AI -- effectively subsuming vision-language models, video generators, world simulators, and world-action models into a single framework. Our evaluation demonstrates that Cosmos 3 establishes a new state-of-the-art across a diverse suite of understanding and generation tasks, demonstrating omnimodal world models as scalable, general-purpose backbones for embodied agents. Our post-trained Cosmos 3 models were ranked as the best open-source Text-to-Image and Image-to-Video models by Artificial Analysis, and the best policy model by RoboArena at the time the technical report was written. To accelerate open research and deployment in Physical AI, we make our code, model checkpoints, curated synthetic datasets, and evaluation benchmark available under the Linux Foundation's OpenMDW-1.1 License at https://github.com/nvidia/cosmos and https://huggingface.co/collections/nvidia/cosmos3. The project website is available at https://research.nvidia.com/labs/cosmos-lab/cosmos3.