ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning

3 min read

ConceptGraphs builds an object-level 3D scene graph from posed RGB-D frames using only off-the-shelf 2D models, with no 3D training. In each frame SAM proposes class-agnostic masks, CLIP embeds each mask, and the masked pixels are lifted to a point cloud, cleaned with DBSCAN and matched greedily to existing objects by the sum of a geometric overlap score and CLIP cosine similarity. A matched detection is fused by taking the union of points (then downsampling at 2.5 cm) and a running mean of the CLIP vectors. Once the whole sequence is processed, LLaVA captions each object from its 10 best views, GPT-4 condenses those captions into one tag, and edges come from a minimum spanning tree over 3D box overlap, with GPT-4 naming each relation (“a on b”). For planning, the graph is written out as JSON (id, box centre and extent, tag, caption) and pasted into a GPT-4 prompt.

Findings

  • Human raters on Replica judged node captions correct 71% of the time on average (0.61 for the detector variant CG-D), and edges correct 88% (0.91 for CG-D). Each scene had 0 to 5 duplicate objects. Most caption errors came from LLaVA.
  • Zero-shot semantic segmentation on Replica: mAcc 40.63 and F-mIoU 35.95, against ConceptFusion with SAM at 31.53 and 38.70. The paper says ConceptFusion’s per-point map has a much larger memory footprint but gives no figure.
  • Object retrieval shows where CLIP stops being enough. On descriptive queries CLIP similarity and the LLM are similar (Replica R@1 0.59 against 0.61). On negation (“something to drink other than soda”) CLIP gets R@1 0.26 on Replica and 0.00 in the lab scan, while GPT-4 reading captions gets 0.80 and 1.00. On affordance queries in the lab, 0.40 against 1.00.
  • Real robots: a Jackal UGV (VLP-16, RealSense D435i) answers abstract queries, re-plans when an object has moved by asking the LLM for a likely container, and asks the LLM which objects it can push (the prompt states a 2.27 kg limit) before adding the rest to its costmap. A Spot Arm does open-vocabulary pick and place, and an AI2Thor demo uses the graph for particle-filter localisation with simple object add and remove.

Limits

  • The authors list caption errors from LLaVA-7B (small objects often called toothbrushes or scissors), missed small or thin objects, duplicates, and the compute and money cost of many LVLM calls plus one or more proprietary LLM calls to build and query the graph.
  • Captioning and edge labelling run after mapping, so the full graph is offline. Clio measured the CLIP-only part at 2.0 to 8.1 s per frame.
  • The planner sends the whole graph to an 8K-token GPT-4 context. The authors point to SayPlan for scenes that do not fit.
  • Posed RGB-D input is assumed (poses came from RTAB-Map in the real tests), and the caption-summary prompt tells GPT-4 to focus on indoor objects. Captions carry no confidence, and one bad caption sinks the downstream plan.

Relevance

ConceptGraphs set the recipe that later open-vocabulary graphs measure themselves against: a CLIP vector and a caption per object. For a robot on a small drone the open questions are whether the caption and LLM steps can run on board at all, and how two robots that saw the same object from different sides would agree on its caption. See open-vocabulary mapping, 3D scene graphs, language-grounded planning and multi-robot multimodal perception.