Informing 3D Scene Graph Generation with Common-Sense Spatial Knowledge

2 min read

This four-page workshop paper is the first version of the logic tensor network idea later extended in strader-2023-ontology. It labels the place nodes of a Hydra scene graph with room types by node classification (a graph attention network plus an MLP) and adds a hand-written spatial ontology as a training signal: a 9 by 4 table saying which object labels (bed, toilet, sink, sofa) belong to which rooms (bedroom, bathroom, kitchen, living room). The LTN rewards predictions that agree with ground-truth labels where they exist and with the ontology everywhere else, so unlabelled places still produce a loss.

Findings

  • Setting: Matterport3D, 90 scenes, 4 trajectories of about 100 m each per scene, graphs built by Hydra from Habitat’s ground-truth 2D semantics.
  • With all labels, data-only and data-plus-ontology training both reach 0.487 accuracy. With 0.1% of labels, data-only falls to 0.053 and data-plus-ontology holds 0.246.
  • With six common room labels hidden from training, data-plus-ontology scores 0.255 against 0.168 at full data, and 0.244 against 0.035 at 0.1%.
  • Ontology-only training sits at 0.242 to 0.249 whatever the label budget, so without labels the hand-written ontology sets the level.

Limits

  • The authors name the main limit: the method needs a spatial ontology, and theirs covers only a few intuitive relations (the table puts sinks only in kitchens). They suggest language models could supply one, which the RA-L version does.
  • Indoor only, with ground-truth semantics feeding the mesh layer, so detector noise is not tested.
  • One high-level layer (rooms). The authors say the method could be applied repeatedly to build more layers but do not show it.

Relevance

A small test of how much a common-sense prior buys when labels are scarce, which is the normal case for any robot outside a house. See 3D scene graphs and multi-robot multimodal perception.