Paper 2602.09856
Code2World: A GUI World Model via Renderable Code Generation
- Published
- Feb 2026
- Research lab
- Independent
- Citations
- 14
- GitHub
- 323 stars
01 In brief
Summary
Code2World is a vision-language coder that predicts the next GUI state by generating renderable HTML code, combining high visual fidelity with structural controllability.
To address data scarcity, the authors constructed AndroidCode, a corpus of over 80K screen-action pairs, by translating GUI trajectories from AndroidControl into HTML using GPT-5 and refining them via a visual-feedback revision loop (SigLIP score > 0.9).
The model, based on Qwen3-VL-8B, is trained in two stages: Supervised Fine-Tuning (SFT) for format following, then Render-Aware Reinforcement Learning (RARL) with GRPO, using dual rewards—visual semantic fidelity (R_sem) and action consistency (R_act)—derived from rendered outcomes.
Experiments show Code2World-8B achieves top performance in next UI prediction, rivaling GPT-5 and Gemini-3-Pro-Image, and significantly enhances downstream GUI agents.
As a plug-and-play simulator, it boosts Gemini-2.5-Flash's success rate by +9.5% on AndroidWorld navigation, and improves offline grounding accuracy by up to +5.23 for Qwen2.5-VL-7B.
Ablations confirm that both rewards are crucial, as R_sem alone risks reward hacking and R_act alone lacks visual detail.
The work demonstrates that renderable code generation is a viable paradigm for GUI world modeling, enabling agents to simulate and evaluate actions before execution, thereby improving planning and safety.
02 From the paper
Abstract
Autonomous GUI agents interact with environments by perceiving interfaces and executing actions. As a virtual sandbox, the GUI World model empowers agents with human-like foresight by enabling action-conditioned prediction. However, existing text- and pixel-based approaches struggle to simultaneously achieve high visual fidelity and fine-grained structural controllability. To this end, we propose Code2World, a vision-language coder that simulates the next visual state via renderable code generation. Specifically, to address the data scarcity problem, we construct AndroidCode by translating GUI trajectories into high-fidelity HTML and refining synthesized code through a visual-feedback revision mechanism, yielding a corpus of over 80K high-quality screen-action pairs. To adapt existing VLMs into code prediction, we first perform SFT as a cold start for format layout following, then further apply Render-Aware Reinforcement Learning which uses rendered outcome as the reward signal by enforcing visual semantic fidelity and action consistency. Extensive experiments demonstrate that Code2World-8B achieves the top-performing next UI prediction, rivaling the competitive GPT-5 and Gemini-3-Pro-Image. Notably, Code2World significantly enhances downstream navigation success rates in a flexible manner, boosting Gemini-2.5-Flash by +9.5% on AndroidWorld navigation. The code is available at https://github.com/AMAP-ML/Code2World.