VGGT is a feed-forward transformer that takes one, a few or hundreds of images and directly predicts camera parameters, point maps, depth maps and 3D point tracks. There is no separate optimisation stage: the network replaces the structure-from-motion pipeline for many uses.
Findings
- Reconstructs in under one second and beats alternatives that need geometric optimisation afterwards, on camera estimation, multi-view depth, dense point clouds and point tracking (abstract).
- Its pretrained features also help non-rigid tracking and feed-forward view synthesis.
Limits
- Abstract only so far. Memory grows with the number of frames, which is why long-video variants (LoGeR) and chunked SLAM wrappers exist.
- Trained in air. Wat3R starts from its checkpoint because it does not transfer to water as is.
Relevance
A backbone that underwater work now fine-tunes (Wat3R). See foundation models for robotics.
Links
- arXiv · code
- Subfield: Mapping and localisation, Perception