Paper 2606.26025
In-Context World Modeling for Robotic Control
- Published
- Jun 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- Not linked
01 In brief
Summary
The paper introduces In-Context World Modeling (ICWM), a framework that enables Vision-Language-Action (VLA) models to adapt to novel system configurations (e.g., camera viewpoints, robot morphologies) without parameter updates.
Standard VLA models condition only on current observations and language instructions, implicitly assuming a fixed execution context, which leads to performance degradation under novel setups.
ICWM treats system identification as an in-context adaptation problem: before task execution, the robot performs a short sequence of task-agnostic random exploratory movements, and the resulting visual transitions are prepended as context.
The model then implicitly infers the system's dynamics from this interaction history, enabling test-time adaptation.
Experiments on the LIBERO benchmark and a real UR5e platform show that ICWM significantly outperforms standard VLA baselines on unseen viewpoints, with improvements of 13.0% over a multi-view baseline in simulation and substantial gains in real-world tasks.
Ablations confirm that the interaction context is essential, and the model generalizes to semantic scene variations and morphological changes.
The approach adds minimal inference latency and can be optimized via KV caching.
02 From the paper
Abstract
Modern Vision-Language-Action (VLA) models often fail to generalize to novel setups, such as altered camera viewpoints or robot morphologies, because they are typically conditioned only on current observations and language instructions. By ignoring the underlying system configuration as a variable, these models implicitly assume a fixed execution context encountered during training, necessitating data-intensive fine-tuning for any new environment. In this work, we introduce In-Context World Modeling (ICWM), a framework that treats system identification as an in-context adaptation problem. ICWM enables robot policies to autonomously infer essential system variables from a short history of self-generated, task-agnostic interactions. Unlike traditional In-Context Learning that uses demonstrations to specify what task to perform, ICWM leverages the context window to understand how the system operates. By processing these interactions before task execution, the model implicitly captures the world dynamics of the current system, enabling adaptation to novel configurations without parameter updates. Extensive experiments in simulation and on real-world robot platforms demonstrate that ICWM significantly outperforms standard VLA baselines on novel camera viewpoints.