Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining

The paper introduces Video2GUI, a fully automated framework that extracts grounded GUI interaction trajectories from unlabeled internet videos to address the scarcity of large-scale training data for GUI agents.

The pipeline uses a coarse-to-fine filtering strategy: first, metadata-based classification (using a fine-tuned Qwen2.5-7B) reduces 500 million videos to ~20 million candidates; then, a video quality scorer (fine-tuned Qwen2.5-Omni) retains 4.16 million high-quality tutorial videos (~300,000 hours).

Trajectory extraction uses Gemini-3-Pro with a sliding-window approach to generate task instructions, action timestamps, and low-level instructions.

Action spatial grounding maps actions to precise screen coordinates using multi-frame high-resolution screenshots.

The resulting dataset, WildGUI, contains 12.7 million trajectories and 124.5 million screenshots across 1,500+ applications and websites.

Pre-training Qwen2.5-VL and Mimo-VL on WildGUI yields consistent improvements of 5–20% across GUI grounding and agent benchmarks, matching or surpassing state-of-the-art performance.

Scaling studies show performance increases with pretraining tokens up to 200 billion.

Ablations confirm the importance of grounding, action, and trajectory losses, and the two-stage training paradigm.

Human evaluation rates WildGUI trajectories higher than baselines (4.62 vs.

3.35 and 4.05).

The dataset and pipeline will be released to facilitate future research.