Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations

Concerto is a self-supervised learning framework that combines 3D intra-modal self-distillation with 2D-3D cross-modal joint embedding prediction to learn spatial representations from point clouds and images.

It outperforms standalone 2D (DINOv2) and 3D (Sonata) self-supervised models by 14.2% and 4.8% respectively in linear probing for 3D scene perception, and also surpasses their feature concatenation.

With full fine-tuning, Concerto achieves state-of-the-art results on multiple benchmarks, including 80.7% mIoU on ScanNet semantic segmentation.

The framework also includes a variant for video-lifted point clouds and a translator that linearly projects representations into CLIP's language space for open-world perception.

Ablation studies show that cosine similarity for cross-modal loss, balanced loss weights, and weak image augmentations improve performance.

Concerto demonstrates superior data efficiency, with linear probing outperforming full fine-tuning in data-limited scenarios, and shows strong parameter efficiency with LoRA fine-tuning.

The work highlights the emergence of coherent, modality-agnostic spatial representations through multisensory synergy, inspired by human concept learning.