Fast-SAM3D speeds up SAM 3D without retraining. The authors find that generic diffusion speed-ups break this pipeline because its stages behave differently, and add one fix per stage. Modality-aware step caching extrapolates the smoothly evolving shape tokens and smooths the jittery layout tokens. Joint spatiotemporal token carving spends refinement only on high-entropy regions of the structured latent. Spectral-aware token aggregation picks the mesh decoder’s resolution from a high-frequency energy ratio, compressing simple shapes hard and keeping detail on complex ones.
Findings
- Up to 2.67x end-to-end speed-up for single objects and 2.01x for scene generation (README).
- Geometric fidelity holds: F-score 92.59 against 92.34 for the original (README).
Limits
- I read the abstract and README only.
- It inherits everything about SAM 3D: one image in, a plausible full object out, with the hidden side generated from a prior.
- Still a large GPU model; 2.67x faster does not make it onboard-ready for a small vehicle.
Relevance
Object-level 3D from a single masked image gives a robot a full shape for an object it has only seen from one side. The risk is that generated geometry looks like measured geometry; anything downstream would need to know which parts were seen and which were imagined. See foundation models for robotics.
Links
- arXiv · code
- Subfield: Mapping and localisation, Perception