The Year in AI Papers
The year/Independent research

Paper 2511.08892

Lumine: An Open Recipe for Building Generalist Agents in 3D Open Worlds

Weihao Tan, Xiangyang Li, Yunhao Fang, Heyuan Yao, Shi Yan, Hao Luo, Tenglong Ao, Huihui Li, Hongbin Ren, Bairen Yi, Yujia Qin, Bo An, et al.

Published
Nov 2025
Research lab
Independent
Citations
13
GitHub
Not linked

01 In brief

Summary

Lumine, developed by ByteDance Seed, is the first open recipe for building generalist agents that complete hours-long missions in real time within 3D open-world environments.

Built on Qwen2-VL-7B-Base, it processes raw pixels at 5 Hz and generates keyboard-mouse actions at 30 Hz using action chunking, with a hybrid thinking mode that invokes reasoning only when necessary.

Trained in Genshin Impact via a three-stage curriculum (1731 hours of pre-training, 200 hours of instruction following, and 15 hours of reasoning data), Lumine completed the five-hour Mondstadt main storyline in 56 minutes, matching expert human performance.

It achieved over 80% success on short-horizon tasks and demonstrated zero-shot generalization, completing a 100-minute mission in Wuthering Waves and the five-hour first chapter of Honkai: Star Rail without fine-tuning.

Real-time inference was achieved through optimizations including StreamingLLM, tensor parallelism, quantization, and speculative decoding, yielding a 25.3× latency reduction.

The paper also introduces a 141-task benchmark and discusses limitations such as long-term memory and online learning, proposing future directions for scaling and broader applications in game development and interactive entertainment.

02 From the paper

Abstract

We introduce Lumine, the first open recipe for developing generalist agents capable of completing hours-long complex missions in real time within challenging 3D open-world environments. Lumine adopts a human-like interaction paradigm that unifies perception, reasoning, and action in an end-to-end manner, powered by a vision-language model. It processes raw pixels at 5 Hz to produce precise 30 Hz keyboard-mouse actions and adaptively invokes reasoning only when necessary. Trained in Genshin Impact, Lumine successfully completes the entire five-hour Mondstadt main storyline on par with human-level efficiency and follows natural language instructions to perform a broad spectrum of tasks in both 3D open-world exploration and 2D GUI manipulation across collection, combat, puzzle-solving, and NPC interaction. In addition to its in-domain performance, Lumine demonstrates strong zero-shot cross-game generalization. Without any fine-tuning, it accomplishes 100-minute missions in Wuthering Waves and the full five-hour first chapter of Honkai: Star Rail. These promising results highlight Lumine's effectiveness across distinct worlds and interaction dynamics, marking a concrete step toward generalist agents in open-ended environments.