Paper 2603.23497
WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPG
- Published
- Mar 2026
- Research lab
- Independent
- Citations
- 6
- GitHub
- 420 stars
01 In brief
Summary
WildWorld is a large-scale, action-conditioned world modeling dataset automatically collected from the AAA game Monster Hunter: Wilds.
It contains over 108 million frames with more than 450 actions (movement, attacks, skill casting) and per-frame annotations including character skeletons, world states, camera poses, and depth maps.
The dataset addresses the lack of semantically meaningful action spaces and explicit state annotations in existing datasets, which often entangle actions with pixel-level changes.
WildBench, derived from WildWorld, evaluates models via Action Following (agreement with ground-truth sub-actions) and State Alignment (tracking skeletal keypoints).
Experiments with baseline models (CamCtrl, SkelCtrl, StateCtrl) show that while all improve over the baseline, VBench metrics saturate, and models still struggle with long-horizon state consistency.
The dataset supports diverse entities (29 monster species, 4 weapons) and scenes (5 stages, 66% combat).
The authors highlight the need for state-aware video generation and invite researchers to collaborate.
02 From the paper
Abstract
Dynamical systems theory and reinforcement learning view world evolution as latent-state dynamics driven by actions, with visual observations providing partial information about the state. Recent video world models attempt to learn this action-conditioned dynamics from data. However, existing datasets rarely match the requirement: they typically lack diverse and semantically meaningful action spaces, and actions are directly tied to visual observations rather than mediated by underlying states. As a result, actions are often entangled with pixel-level changes, making it difficult for models to learn structured world dynamics and maintain consistent evolution over long horizons. In this paper, we propose WildWorld, a large-scale action-conditioned world modeling dataset with explicit state annotations, automatically collected from a photorealistic AAA action role-playing game (Monster Hunter: Wilds). WildWorld contains over 108 million frames and features more than 450 actions, including movement, attacks, and skill casting, together with synchronized per-frame annotations of character skeletons, world states, camera poses, and depth maps. We further derive WildBench to evaluate models through Action Following and State Alignment. Extensive experiments reveal persistent challenges in modeling semantically rich actions and maintaining long-horizon state consistency, highlighting the need for state-aware video generation. The project page is https://shandaai.github.io/wildworld-project/.