Paper 2606.30534
Orca: The World is in Your Mind
- Published
- Jun 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- 906 stars
01 In brief
Summary
Orca, developed by the Beijing Academy of Artificial Intelligence, is a general world foundation model that learns a unified world latent space from multimodal signals (vision and language) using Next-State-Prediction modeling.
It employs two complementary learning paradigms: unconscious learning captures dense natural state transitions from continuous videos, while conscious learning models sparse meaningful transitions guided by language-described events and VQA supervision.
Pre-training uses 125K hours of video data and 160M event annotations (only one-tenth used in this version).
After pre-training, the backbone is frozen, and lightweight modality-specific decoders are trained for text generation, image prediction, and embodied action generation.
Experiments show scalability: losses decrease with model size and data, and stronger latents improve downstream performance.
Orca-4B outperforms similar-sized baselines on text benchmarks (avg 51.8), image prediction (PRICE-V0.1 avg 59.8), and real-robot action generation (overall rule-based 32.4), surpassing specialized models like Qwen3.5 and V-JEPA 2.1.
Ablations confirm all three objectives are needed for balanced performance.
Limitations include limited modalities, ViT-space supervision, model size, and short-horizon transitions.
Future work includes more modalities, native world-state modeling, and self-evolutionary loops.
02 From the paper
Abstract
We introduce Orca, an initial instantiation of a general world foundation model. Orca learns a unified world latent space from multimodal world signals and exposes it through multimodal readout interfaces. Rather than optimizing isolated next-token, next-frame, or next-action prediction, we are centered on Next-State-Prediction modeling, offering a unified state-transition modeling route toward understanding, predicting, and acting upon the world. Orca learns through two complementary paradigms: unconscious learning captures dense natural state transitions from continuous videos, and conscious learning models sparse meaningful state transitions by language-described events and VQA supervision. For pre-training, we construct a large-scale world-learning inventory data, including 125K hours of video data and 160M event annotations. After pre-training, Orca learns a unified world latent space. To examine whether the learned latent supports downstream, we evaluate it by three representative downstream readouts: text generation, image prediction, and embodied action generation. Orca's backbone is frozen, and only the lightweight modality-specific decoders are trainable. Experiments show the scalability of the proposed paradigm and verify that stronger world latent enables stronger downstream readouts. Orca outperforms similar-sized specialized baselines. These results show that Orca, as a general world foundation model, presents a promising approach to understanding, predicting, and acting upon the world. Finally, we discuss the current limitations, aiming to provide useful insights and inspiration for the community.