Paper 2511.16719
SAM 3: Segment Anything with Concepts
01 In brief
Summary
SAM 3 is a unified model for promptable concept segmentation (PCS) in images and videos, accepting noun phrases, image exemplars, or both as prompts to detect, segment, and track all matching instances.
It decouples recognition and localization via a presence head, improving detection accuracy.
A data engine with human and AI verifiers produced 4M unique concept labels and 52M masks, plus a synthetic dataset with 38M phrases.
The SA-Co benchmark includes 207K unique concepts across 120K images and 1.7K videos.
SAM 3 doubles prior PCS accuracy, achieving 48.8 mask AP on LVIS (vs.
38.5 best prior) and 54.1 cgF1 on SA-Co/Gold (vs.
24.6 for OWLv2*).
It also improves PVS over SAM 2 and supports interactive refinement.
Ablations show the presence head, hard negatives, and high-quality data boost performance.
SAM 3 runs in 30 ms per image on H200 and near real-time for ~5 objects in video.
The model is open-sourced with code and demo links provided.
Limitations include difficulty with out-of-domain fine-grained concepts and linear inference scaling with object count in video, mitigated by multiplexing and MLLM integration for complex queries.
The SAM 3 Agent, combining SAM 3 with an MLLM, surpasses prior zero-shot results on ReasonSeg and OmniLabel without training on referring expression data.
Training used 172k A100 and 86k H200 GPU hours, with estimated emissions of 65-78 metric tons CO2e.
The paper includes extensive appendices on ablations, data engine details, and additional experiments.
Overall, SAM 3 sets a new state-of-the-art in promptable segmentation, with the SA-Co benchmark enabling…
02 From the paper
Abstract
We present Segment Anything Model (SAM) 3, a unified model that detects, segments, and tracks objects in images and videos based on concept prompts, which we define as either short noun phrases (e.g., "yellow school bus"), image exemplars, or a combination of both. Promptable Concept Segmentation (PCS) takes such prompts and returns segmentation masks and unique identities for all matching object instances. To advance PCS, we build a scalable data engine that produces a high-quality dataset with 4M unique concept labels, including hard negatives, across images and videos. Our model consists of an image-level detector and a memory-based video tracker that share a single backbone. Recognition and localization are decoupled with a presence head, which boosts detection accuracy. SAM 3 doubles the accuracy of existing systems in both image and video PCS, and improves previous SAM capabilities on visual segmentation tasks. We open source SAM 3 along with our new Segment Anything with Concepts (SA-Co) benchmark for promptable concept segmentation.