How a robot turns pixels and points into objects and distances. Most of the papers here are foundation models a robot uses without training them: segmenters, depth and stereo models, speech recognition. The questions are where they break, and how a robot would know.
Concepts
- Foundation models for robotics: what is on my disk, what it changes, how it fails.
- Disagreement between models: using agreement between models, or robots, to decide when to trust an answer.
- Open-vocabulary mapping, filed under semantic mapping.
- Underwater perception, filed under field robotics.
Entities
Datasets, platforms, tools and groups that come up in this subfield: CLIP, DINO, FastSAM, RealSense, lidar. The entity index has the rest.
Open questions
- How does a robot notice that its depth model is confidently wrong?
- When two robots run the same segmenter, is their agreement worth anything?
- What runs on a Jetson-class computer, and what does it lose?
Papers
| Paper | Year | Venue | Domain | Threads |
|---|---|---|---|---|
| Wat3R | 2026 | ECCV 2026 | sea | perception |
| Fast-SAM3D | 2026 | ICML 2026 | general | perception |
| Pow3R-SLAM | 2026 | arXiv 2026 | land | perception, robustness |
| MoGe-3 | 2026 | arXiv 2026 | general | perception |
| QuerySplat | 2026 | arXiv preprint | general | perception |
| Fast-FoundationStereo | 2026 | CVPR 2026 | general | perception |
| InfiniDepth | 2026 | CVPR 2026 | general | perception |
| LOST-3DSG | 2026 | arXiv preprint | land | long-term, perception |
| DAAAM | 2026 | CVPR 2026 (per the code README; arXiv lists no venue) | land | long-term, perception |
| TrackGraph | 2026 | arXiv 2026 | land, air | perception, long-term |
| Hydra++ | 2026 | IROS 2026 (accepted, per arXiv comments) | land | perception |
| SceneGraphVLM | 2026 | arXiv preprint | general | perception |
| Picasso | 2026 | arXiv preprint | general | perception |
| MERLION | 2025 | ICRA 2025 | sea | perception, long-term |
| MoGe | 2025 | CVPR 2025 | general | perception |
| SAM 3 | 2025 | arXiv 2025 | general | perception, long-term |
| UniDepthV2 | 2025 | arXiv 2025 | general | perception, robustness |
| MoGe-2 | 2025 | arXiv 2025 | general | perception |
| FunGraph | 2025 | IROS 2025 | land | perception |
| Sea-thru distance accuracy | 2024 | ACRS 2024 | sea | perception |
| Gemma | 2024 | arXiv 2024 | general | perception |
| UniDepth | 2024 | CVPR 2024 | general | perception |
| SAM 2 | 2024 | arXiv 2024 | general | perception, long-term |
| OS-Atlas | 2024 | arXiv 2024 (cited as ICLR 2025) | general | perception |
| OpenGraph | 2024 | arXiv preprint | land | perception |
| Open3DSG | 2024 | CVPR 2024 | land | perception |
| DM_underwater | 2023 | arXiv 2023 | sea | perception |
| RT-2 | 2023 | CoRL 2023 (PMLR 229) | land | perception |
| SAM | 2023 | ICCV 2023 | general | perception |
| Whisper | 2023 | ICML 2023 (arXiv 2022) | general | perception |
| PaLM-E | 2023 | ICML 2023 | land | perception |
| ConceptGraphs | 2023 | ICRA 2024 | land | perception |
| Common-sense spatial knowledge (workshop) | 2023 | RSS 2023 Workshop on Robot Representations for Scene Understanding, Reasoning and Planning | land | perception |
| Language-enabled spatial ontologies | 2023 | IEEE RA-L 2024 | land | perception |
| Mono-Hydra | 2023 | ISPRS Annals (GSW 2023) | land, air | perception |
| Verified AI | 2022 | Communications of the ACM 65(7) | general | robustness |
| 3D Scene Graph (Armeni) | 2019 | ICCV 2019 | land | perception |
| Active perception | 2018 | Autonomous Robots 42(2), 2018 (arXiv 2016) | general | perception, long-term |
| Paper | Year | Venue | Domain | Threads |
|---|---|---|---|---|
| Swimm3R | 2026 | arXiv 2026 | sea | perception |
| MAG-VLAQ | 2026 | arXiv preprint | air, land | perception |
| Sonar VFM adaptation | 2026 | arXiv preprint | sea | perception |
| LeRobot | 2026 | arXiv 2026 | land | perception |
| MapAnything | 2026 | 3DV 2026 | general | perception |
| CERPE | 2026 | arXiv preprint | land | perception |
| CoSMO | 2026 | CoRL 2026 | land | robustness, perception |
| RIWM | 2026 | arXiv preprint (perspective) | general | long-term, robustness |
| FailBench | 2026 | arXiv preprint | general | robustness |
| Hide-and-Seek | 2026 | NeurIPS 2026 | general | robustness |
| Detect and Suppress | 2026 | arXiv preprint | general | robustness |
| Embodied agent security survey | 2026 | arXiv preprint (survey) | general | robustness, long-term |
| Physical prompt injection | 2026 | arXiv preprint | general | robustness |
| Worth Remembering | 2026 | arXiv preprint | land | long-term |
| OASIS-Map | 2026 | arXiv preprint | land | long-term |
| Mono-Hydra++ | 2026 | arXiv preprint | land | perception |
| ChronoGraph | 2026 | arXiv preprint | land | perception, long-term |
| UQ-DAAAM | 2026 | arXiv preprint | land | long-term, perception |
| Underwater metric depth benchmark | 2025 | arXiv 2025 | sea | perception, robustness |
| SmolVLA | 2025 | arXiv 2025 | land | perception |
| SAM 3D | 2025 | arXiv 2025 | general | perception |
| MASt3R-SLAM | 2025 | CVPR 2025 | land | perception |
| CRISP | 2025 | CVPR 2025 | general | robustness, perception |
| VGGT | 2025 | CVPR 2025 | general | perception |
| Depth Anything 3 | 2025 | arXiv 2025 | general | perception |
| SAFE | 2025 | NeurIPS 2025 | general | robustness |
| BadVLA | 2025 | arXiv preprint | general | robustness |
| π0 | 2024 | RSS 2025 | land | perception, robustness |
| OpenVLA | 2024 | arXiv 2024 | land | perception |
| Sea-thru | 2019 | CVPR 2019 | sea | perception |
Other subfields: mapping and localisation, semantic mapping, planning and navigation, control, robot learning, multi-robot systems, safety and verification, security, Human-robot interaction, field robotics.