SAM 2 extends promptable segmentation from images to video. The model is a transformer with streaming memory, so it can carry a mask forward frame by frame in real time, and a data engine that improves through user interaction produced what the authors call the largest video segmentation dataset to date. Model, dataset, training code and a demo are released.
Findings
- Video: better accuracy with 3x fewer user interactions than earlier approaches.
- Images: more accurate than SAM and 6x faster.
Limits
- Abstract-level reading only. The abstract gives no numbers on long videos, heavy occlusion or objects that leave and re-enter the frame.
Relevance
Streaming memory gives robots a small form of object permanence, since a mask persists across frames, useful for dynamic-scenes and long-term autonomy. Open question: how does that memory behave when the camera is on a rolling boat or a drone in gusts? Cited in GoGet; the successor is SAM 3.
Links
- arXiv
- Subfield: Perception