Paper 2604.08626
WildDet3D: Scaling Promptable 3D Detection in the Wild
- Published
- Apr 2026
- Research lab
- Independent
- Citations
- 7
- GitHub
- 606 stars
01 In brief
Summary
WildDet3D introduces a unified, geometry-aware architecture for open-vocabulary monocular 3D object detection that accepts text, point, and box prompts, and optionally incorporates depth cues at inference time.
The model uses dual-vision encoders (image and RGBD) with a depth fusion module, a promptable detector, and a 3D detection head with unambiguous rotation normalization.
To support open-world generalization, the authors present WildDet3D-Data, the largest open 3D detection dataset, with over 1M images and 13.5K categories, built by generating candidate 3D boxes from 2D annotations and filtering them via rule-based checks, VLM scoring, and human verification.
WildDet3D achieves state-of-the-art results: 22.6/24.8 AP3D on WildDet3D-Bench (text/box prompts), 34.2/36.4 AP3D on Omni3D, and 40.3/48.9 ODS zero-shot on Argoverse 2 and ScanNet.
Adding depth at inference yields substantial gains (+20.7 AP on average).
The model is demonstrated in applications including iPhone, Meta Quest 3, robotics, and VLM-based spatial reasoning.
Limitations include camera intrinsics accuracy, single-image depth ambiguity, rotation estimation, computational cost, and long-tail category performance.
The work is not intended for safety-critical applications.
02 From the paper
Abstract
Understanding objects in 3D from a single image is a cornerstone of spatial intelligence. A key step toward this goal is monocular 3D object detection--recovering the extent, location, and orientation of objects from an input RGB image. To be practical in the open world, such a detector must generalize beyond closed-set categories, support diverse prompt modalities, and leverage geometric cues when available. Progress is hampered by two bottlenecks: existing methods are designed for a single prompt type and lack a mechanism to incorporate additional geometric cues, and current 3D datasets cover only narrow categories in controlled environments, limiting open-world transfer. In this work we address both gaps. First, we introduce WildDet3D, a unified geometry-aware architecture that natively accepts text, point, and box prompts and can incorporate auxiliary depth signals at inference time. Second, we present WildDet3D-Data, the largest open 3D detection dataset to date, constructed by generating candidate 3D boxes from existing 2D annotations and retaining only human-verified ones, yielding over 1M images across 13.5K categories in diverse real-world scenes. WildDet3D establishes a new state-of-the-art across multiple benchmarks and settings. In the open-world setting, it achieves 22.6/24.8 AP3D on our newly introduced WildDet3D-Bench with text and box prompts. On Omni3D, it reaches 34.2/36.4 AP3D with text and box prompts, respectively. In zero-shot evaluation, it achieves 40.3/48.9 ODS on Argoverse 2 and ScanNet. Notably, incorporating depth cues at inference time yields substantial additional gains (+20.7 AP on average across settings).