Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions

3 min read

Wald et al. release 3DSSG, scene graph annotations on top of 3RScan, and train the first network that predicts a semantic scene graph directly from a 3D point cloud. 3RScan has 1482 reconstructions of 478 indoor environments that were rescanned as they changed, so the dataset carries both rich semantics and real change over time. Their network (SGPN) takes the point cloud plus a class-agnostic instance segmentation, extracts per-object features with one PointNet and per-pair features with another (over the union of the two bounding boxes), passes them through a graph convolutional network, and predicts object classes and multiple predicates per edge. They also propose a retrieval task: given one image or one rescan of a changed room, find its 3D reference scan by comparing scene graphs.

Findings

  • The dataset has about 48k object nodes, 544k edges, 534 classes each with a WordNet hypernym chain, 93 attributes (static properties, states such as open or closed, and affordances) and 40 relationship types grouped into support, proximity and comparative. Rendering the 3D graphs into the camera views gives 363k 2D graph-image pairs.
  • Predicting several predicates per edge helps: relationship recall R@50 / R@100 of 0.40 / 0.66 against 0.39 / 0.45 for an adapted relation-prediction baseline, and predicate R@3 of 0.89 against 0.62.
  • Object classes predicted straight from PointNet features beat those taken after the GCN (R@5 0.68 against 0.60).
  • Retrieval with ground-truth graphs is nearly solved (3D-3D top-1 0.96, 2D-3D top-1 1.00 with the Szymkiewicz-Simpson overlap, which suits sets of very different size). With predicted graphs it collapses: 3D-3D top-1 0.34 and 2D-3D top-1 0.13.

Limits

  • Assumes the instance segmentation is given and works on whole finished scans, offline.
  • The retrieval numbers show the representation is only as stable to change as the perception that fills it. The claim that graphs are “robust towards small scene changes and noise” holds for annotated graphs far better than for predicted ones.
  • Indoor rooms only, one capture setup, heavy class imbalance and ambiguous predicates, which the authors acknowledge by moving to per-class binary losses and focal loss.

Relevance

3DSSG is the benchmark under much of the learned 3D scene graph work, and looper-2023-3d-vsg trains on its rescans to predict change, which makes it one of the few data sources for long-term autonomy in 3D scene graphs. The retrieval gap is a problem for teams: matching graphs between robots to find shared places looks easy on clean graphs and much harder on graphs each robot predicted for itself; see collaborative SLAM.