SAM 3D is a generative model that predicts an object’s geometry, texture and pose in the scene (layout) from a single image, aimed at cluttered natural photos where occlusion is common and context helps recognition. The training data comes from a human- and model-in-the-loop annotation pipeline for object shape, texture and pose, at a scale the authors describe as unprecedented. Training is multi-stage: synthetic pretraining first, then alignment on real-world data.
Findings
- At least a 5:1 win rate in human preference tests against recent work, on real-world objects and scenes (abstract).
- The authors release code, weights, a demo and a new benchmark for in-the-wild 3D object reconstruction.
Limits
- Abstract only so far.
- Preference tests measure what looks right, not metric accuracy. The unseen side of each object is generated.
- Scale and layout from one view rest on learned priors.
Relevance
The base model behind Fast-SAM3D. It gives objects a full shape from a glimpse, which is useful for planning and dangerous for anything that treats the shape as measured. See foundation models for robotics.
Links
- arXiv · project page
- Subfield: Mapping and localisation, Perception