SAM 3 detects, segments and tracks every instance of a concept in images and video, where the concept is a short noun phrase, an image exemplar, or both. The authors call this promptable concept segmentation. An image-level detector and a memory-based video tracker share one backbone, and a separate presence head decides whether the concept is in the image at all, decoupling recognition from localisation. A data engine produced labels for 4 million unique concepts, including hard negatives.
Findings
- The abstract says SAM 3 doubles the accuracy of existing systems on image and video concept segmentation, and improves on earlier SAM models for visual-prompt segmentation.
- The README puts it at 75 to 80% of human performance on the new SA-Co benchmark, which has 270K unique concepts.
- SAM 3.1 (March 2026, release notes) adds Object Multiplex, which tracks objects jointly in fixed-size buckets, for about 7x speed at 128 objects on one H100.
Limits
- I read the abstract, README and release notes, not the paper body.
- Needs a recent CUDA GPU; this is not an onboard model for a small vehicle.
- Concepts come from internet imagery, so rare or specialist categories are the ones to check before relying on it.
Relevance
Text-prompted instance tracking with persistent identities is the front end many open-vocabulary mapping systems now assume. For long missions the interesting parts are the tracker’s memory and the presence head: how often does an identity swap or a false “present” survive into the map? See open-vocabulary-mapping, dynamic-scenes and foundation models for robotics.
Links
- arXiv · code
- Subfield: Perception