Large pretrained models that a robot uses without training them itself: for segmentation, depth, language, and increasingly for choosing actions. They change what a cheap robot can perceive, and they bring failure modes nobody wrote down.
Models
| Job | Models | Notes |
|---|---|---|
| Segment anything | SAM, SAM 2, SAM 3, SAM 3D, Fast-SAM3D | SAM 3 takes text prompts; SAM 3D lifts one object to a mesh |
| Depth and geometry | MoGe (2, 3), UniDepth (V2), Depth Anything 3, InfiniDepth, Fast-FoundationStereo | dense geometry from one camera, some with uncertainty |
| Multi-view reconstruction | VGGT, MapAnything, MASt3R-SLAM, LoGeR, Pow3R-SLAM | poses and points in one forward pass |
| Language and vision-language | Transformer, Gemma, Whisper, OS-Atlas | the parts of a local assistant |
| Acting | RT-2, PaLM-E, OpenVLA, π0, SmolVLA, ACT, Diffusion Policy | vision-language-action models and the policies they are compared with |
Uses
- A robot with one cheap camera can now get dense depth, segment objects it was never trained on, and answer questions about what it sees. Fast-FoundationStereo runs in 14 to 49 ms on a desktop GPU, within about 650 MB.
- Models with optional inputs (MapAnything, Pow3R, InfiniDepth) take whatever the robot has: a depth sensor, a pressure reading, a sparse lidar scan.
- A segmenter and a vision-language model can check each other. GoGet does this when it counts objects; see disagreement between models.
Failure modes
- Domain shift. These models are trained on internet images: daylight, air, familiar objects. On RoboTwin 2.0, changing only lighting, texture, clutter and table height takes π0 from 46.4% to 16.3% success (see benchmarks). Depth models degrade badly in water until fine-tuned (underwater perception).
- Confident and wrong. Dense wrong depth is a worse prompt than sparse true depth (Pow3R-SLAM). A VLM states a wrong count with full confidence. Brain2Qwerty’s language model writes fluent wrong sentences when its encoder fails.
- Attacks. A printed sign can hijack a VLM planner, and backdoors in vision-language-action models survive fine-tuning. See robot security and the to-read list.
- The link. Many systems call a hosted model. A robot that loses its connection loses its perception or its vocabulary.
Questions
- Which of these can run on a robot that has a Jetson and a battery, and what does it lose?
- How should a robot discount agreement between two teammates that run the same model?
- When a foundation model is wrong, how would the robot find out?
Related: open-vocabulary mapping, language-grounded planning, evaluating language models, multi-robot multimodal perception.