Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships

3 min read

Open3DSG predicts a scene graph straight from a 3D point cloud with class-agnostic instance masks, by distilling 2D vision-language features into a 3D graph neural network so that no scene graph labels are needed. Node and edge features start from two PointNets and pass through five rounds of message passing in the style of 3DSSG. For training, it picks the top-k frames in which each object is visible enough, average-pools OpenSeg pixel embeddings over the object’s projected pixels, and for each object pair encodes the union crop at several scales with the InstructBLIP image encoder. The network learns, by a cosine loss, to reproduce these targets: 768-dimensional CLIP-space features on nodes and 1408-dimensional BLIP-space features on edges. At inference, an object’s class is the CLIP text prompt with the highest cosine similarity; a relation is produced by feeding the edge feature into InstructBLIP’s Q-Former in place of image features and asking its Vicuna-7B LLM to describe how the two predicted objects relate. When images are available, 2D and 3D features are averaged before querying.

Findings

  • On the closed-vocabulary 3DSSG benchmark (160 object classes, 27 predicates, with free-text LLM answers mapped to labels through BERT), Open3DSG gets object R@5 0.57, predicate R@3 0.63 and relationship R@50 0.64. That is close to the first supervised 3DSSG model (0.68, 0.89, 0.40) and well below the best supervised one, VL-SAT (0.78, 0.98, 0.90).
  • CLIP cannot do relations: a naive CLIP relationship query scores R@50 0.02, OpenSeg plus CLIP 0.05, and a caption-based baseline close to ConceptGraphs 0.30.
  • On rare tail object classes it reaches R@5 0.42 with zero labels, against 0.06 for 3DSSG and 0.31 for VL-SAT, both trained on about 10^5 labels. On common head classes it trails (0.60 against 0.92).
  • Fusing 2D and 3D features beats either alone, and OpenSeg beats CLIP as the object feature. Because it works on the point cloud, it can relate two objects that never appear in the same frame.
  • Training takes 50 epochs on one A100 with batch size 6. It distils on ScanNet because the portrait-mode images of 3RScan rarely show two whole objects in one frame.

Limits

  • The authors say open-set relationship prediction is still hard and that there is no good way to evaluate open-vocabulary scene graphs; scoring against closed labels hides what the method can say.
  • Predicate recall on common classes is low (0.38 on head predicates). The supplementary shows hallucinated relations (“connected via USB”) and contradictions such as two monitors each “to the left of” the other, and low variety in predicted relations.
  • It needs a finished point cloud with instance masks first, and runs offline. Edge features are built for every instance pair, and every edge goes through a 7B LLM, so the relation step grows quickly with object count.
  • Everything is indoor, distilled from good RGB-D scans.

Relevance

Open3DSG keeps promptable features on nodes and edges and only words a relation when asked; ConceptGraphs commits to captions while mapping. Its 3D-only branch raises an open question for robots whose imagery is poor (night, smoke): can a point-cloud network distilled from indoor RGB carry language features into lidar maps? See open-vocabulary mapping, 3D scene graphs and multi-robot multimodal perception.