Paper 2604.26067
RADIO-ViPE: Online Tightly Coupled Multi-Modal Fusion for Open-Vocabulary Semantic SLAM in Dynamic Environments
- Published
- Apr 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- 138 stars
01 In brief
Summary
RADIO-ViPE is an online, calibration-free semantic SLAM system that processes raw monocular RGB video to produce geometry-aware, open-vocabulary 3D grounding.
It tightly couples multi-modal embeddings from agglomerative foundation models (RADIO/RADSeg) with geometric information within a dense bundle adjustment framework, using a factor graph with embedding-based co-visibility.
The system introduces a temporally consistent adaptive robust kernel based on Barron's loss, which classifies pixels into static, movable, and dynamic categories to suppress disturbances from moving objects and displaced scene elements.
RADIO-ViPE achieves state-of-the-art average ATE on the dynamic TUM-RGBD benchmark (1.63 cm with the adaptive kernel) and ranks top-3 on Replica for 3D semantic segmentation, despite using PCA-compressed features (D=256) and no ground-truth depth, pose, or calibration.
The system operates at 8–10 FPS and supports real-time open-vocabulary grounding by projecting RADIO features into SigLIP space for text matching.
Key contributions include a vision-language-geometric fusion via dense bundle adjustment, a dynamic-aware robust optimization scheme, and a calibration-free online open-vocabulary semantic SLAM pipeline.
02 From the paper
Abstract
We present RADIO-ViPE (Reduce All Domains Into One -- Video Pose Engine), an online semantic SLAM system that enables geometry-aware open-vocabulary grounding, associating arbitrary natural language queries with localized 3D regions and objects in dynamic environments. Unlike existing approaches that require calibrated, posed RGB-D input, RADIO-ViPE operates directly on raw monocular RGB video streams, requiring no prior camera intrinsics, depth sensors, or pose initialization. The system tightly couples multi-modal embeddings -- spanning vision and language -- derived from agglomerative foundation models (e.g., RADIO) with geometric scene information. This coupling takes place in initialization, optimization and factor graph connections to improve the consistency of the map from multiple modalities. The optimization is wrapped within adaptive robust kernels, designed to handle both actively moving objects and agent-displaced scene elements (e.g., furniture rearranged during ego-centric session). Experiments demonstrate that RADIO-ViPE achieves state-of-the-art results on the dynamic TUM-RGBD benchmark while maintaining competitive performance against offline open-vocabulary methods that rely on calibrated data and static scene assumptions. RADIO-ViPE bridges a critical gap in real-world deployment, enabling robust open-vocabulary semantic grounding for autonomous robotics and unconstrained in-the-wild video streams. Project page: https://be2rlab.github.io/radio_vipe