A map whose objects can be found by any phrase, including ones nobody listed when the map was built. “The red toolbox”, “something to sit on” and “the thing leaking oil” should all work. Closed-set maps use a fixed label list (a detector trained on 80 or 150 classes) and can’t answer questions outside it.
Language features
Three recipes, often mixed:
- A CLIP vector per segment, averaged over views. Segment each frame (SAM, FastSAM), embed each segment with CLIP, and keep a running mean per object. ConceptGraphs, Clio and ROMAN do this. HOV-SG blends whole-frame, crop and masked-crop embeddings and takes a majority vote per segment instead of a mean, which scores better on Replica (mIoU 0.231 against 0.18).
- A caption from a vision-language model. ConceptGraphs adds LLaVA captions from each object’s ten best views and has GPT-4 condense them. OpenGraph uses the caption’s sentence embedding as the feature. FunGraph asks GPT-4o to name the parts of an object (handles, knobs) with the parent object in view.
- Features learned in 3D. Open3DSG trains a 3D network to output CLIP-space node features and BLIP-space edge features, then asks an LLM what each relation is.
CLIP alone fails on negation, relations and small parts. ConceptGraphs’ negation queries get the right object first 26% of the time from CLIP and 80% from GPT-4 reading captions. CLIP relation queries in Open3DSG score a recall@50 of 0.02. CLIP names FunGraph’s parts correctly 30% of the time, GPT-4o 78%.
They disagree on when to commit. ConceptGraphs and OpenGraph fix the text at mapping time. Open3DSG stores features and prompts later. Clio fixes a task list before the robot moves and keeps only what is informative about those tasks. ASHiTA lets the tasks and the graph refine each other.
Cost
Most of the compute goes into the language layer. Planning over the finished graph is cheap.
| System | Cost reported |
|---|---|
| Clio | 0.30 s per frame on an RTX 3090, 92 objects kept |
| ConceptGraphs | 2.0 to 8.1 s per frame for the CLIP stage, 181 objects |
| HOV-SG | 1.5 GB over 8 scenes, against 6 GB for VLMaps |
| ROMAN | under 250 KB per 40-object submap; 100 times smaller without CLIP, for 12% less alignment success |
| BiMoSG | 0.11 s per frame by staying closed-set until the task needs a VLM |
| USS-Nav | 15 Hz on a Jetson Orin NX by embedding label text instead of image crops |
Most of the others report no runtime at all, and those that do are not comparable.
Teams
- Two robots that caption the same chair differently, or carve a shelf into different objects because they were given different tasks, need more than geometry to agree. See 3D scene graphs and semantic coordination.
- CLIP vectors are 512 to 1024 numbers per object. Over a weak radio link at tens of kilobits per second that is the whole budget. ROMAN’s maps are 100 times smaller without CLIP, for 12% less alignment success.
- Thermal cameras, smoke, rain and night give images that CLIP was never trained on, so the language layer degrades in the conditions field robots work in.
- Several systems call a hosted model (GPT-4, GPT-4o). A robot that loses its link loses its vocabulary.
- There is no agreed protocol for evaluating open-vocabulary graphs, which Open3DSG’s authors point out themselves.
Questions
- Can disagreement between two models, or two robots, tell a robot when not to trust an answer? GoGet uses agreement between a segmenter and a VLM to decide when to trust a count.
- What is the cheapest feature that still supports the queries a team actually makes?
- How should a robot mark a node as “seen but not understood”?
Related: foundation models for robotics, language-grounded planning, multi-robot multimodal perception.