Paper 2510.12798
Detect Anything via Next Point Prediction
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 60
- GitHub
- 1.5K stars
01 In brief
Summary
Rex-Omni is a 3B-parameter multimodal large language model (MLLM) that unifies object detection and other visual perception tasks under a next-point prediction framework.
It uses quantized relative coordinates represented by 1,000 special tokens, reducing token usage per bounding box to four.
Training combines supervised fine-tuning on 22 million samples with GRPO-based reinforcement post-training using geometry-aware rewards (box IoU, point-in-mask, point-in-box) to correct SFT-induced issues like duplicate and large-box predictions.
On COCO, Rex-Omni achieves zero-shot F1@IoU0.5 of 72.0, outperforming Grounding DINO and SEED1.5-VL.
It also excels on LVIS, dense detection (Dense200, VisDrone), referring expression comprehension, visual prompting, GUI grounding, layout grounding, OCR, spatial pointing, and keypoint estimation.
Analysis shows GRPO primarily corrects behavioral deficiencies rather than improving raw coordinate precision, and special-token coordinates improve inference efficiency.
Limitations include slower inference than traditional detectors.
02 From the paper
Abstract
Object detection has long been dominated by traditional coordinate regression-based models, such as YOLO, DETR, and Grounding DINO. Although recent efforts have attempted to leverage MLLMs to tackle this task, they face challenges like low recall rate, duplicate predictions, coordinate misalignment, etc. In this work, we bridge this gap and propose Rex-Omni, a 3B-scale MLLM that achieves state-of-the-art object perception performance. On benchmarks like COCO and LVIS, Rex-Omni attains performance comparable to or exceeding regression-based models (e.g., DINO, Grounding DINO) in a zero-shot setting. This is enabled by three key designs: 1) Task Formulation: we use special tokens to represent quantized coordinates from 0 to 999, reducing the model's learning difficulty and improving token efficiency for coordinate prediction; 2) Data Engines: we construct multiple data engines to generate high-quality grounding, referring, and pointing data, providing semantically rich supervision for training; \3) Training Pipelines: we employ a two-stage training process, combining supervised fine-tuning on 22 million data with GRPO-based reinforcement post-training. This RL post-training leverages geometry-aware rewards to effectively bridge the discrete-to-continuous coordinate prediction gap, improve box accuracy, and mitigate undesirable behaviors like duplicate predictions that stem from the teacher-guided nature of the initial SFT stage. Beyond conventional detection, Rex-Omni's inherent language understanding enables versatile capabilities such as object referring, pointing, visual prompting, GUI grounding, spatial referring, OCR and key-pointing, all systematically evaluated on dedicated benchmarks. We believe that Rex-Omni paves the way for more versatile and language-aware visual perception systems.