Paper 2511.19046
MedSAM3: Delving into Segment Anything with Medical Concepts
- Published
- Nov 2025
- Research lab
- Independent
- Citations
- 28
- GitHub
- 315 stars
01 In brief
Summary
MedSAM-3 adapts the SAM 3 architecture for medical image and video segmentation, enabling Promptable Concept Segmentation (PCS) via open-vocabulary text prompts.
The model is fine-tuned on medical images paired with concise concept phrases (≤3 words) across modalities like X-ray, MRI, Ultrasound, CT, and video.
The MedSAM-3 Agent integrates a Multimodal Large Language Model (MLLM) for iterative refinement, improving accuracy.
Experiments show MedSAM-3 outperforms existing specialist and foundation models, with text-plus-bounding-box prompting yielding the best results.
The agentic framework, using Gemini 3 Pro, further boosts Dice scores (e.g., from 0.7772 to 0.8064 on BUSI).
Key findings include SAM 3's poor baseline performance, misalignment between concepts and targets, and limited semantic discrimination, which fine-tuning and agentic workflows mitigate.
02 From the paper
Abstract
Medical image segmentation is fundamental for biomedical discovery. Existing methods lack generalizability and demand extensive, time-consuming manual annotation for new clinical application. Here, we propose MedSAM-3, a text promptable medical segmentation model for medical image and video segmentation. By fine-tuning the Segment Anything Model (SAM) 3 architecture on medical images paired with semantic conceptual labels, our MedSAM-3 enables medical Promptable Concept Segmentation (PCS), allowing precise targeting of anatomical structures via open-vocabulary text descriptions rather than solely geometric prompts. We further introduce the MedSAM-3 Agent, a framework that integrates Multimodal Large Language Models (MLLMs) to perform complex reasoning and iterative refinement in an agent-in-the-loop workflow. Comprehensive experiments across diverse medical imaging modalities, including X-ray, MRI, Ultrasound, CT, and video, demonstrate that our approach significantly outperforms existing specialist and foundation models. We will release our code and model at https://github.com/Joey-S-Liu/MedSAM3.