Indoor and Outdoor 3D Scene Graph Generation via Language-Enabled Spatial Ontologies

4 min read

This paper builds the upper layer of a Hydra scene graph outdoors, where nobody has written down what the regions are and there is almost no labelled data. It keeps Hydra’s mesh, object and place layers and labels each place node with a region (“road”, “field”, “beach”, “kitchen”) by node classification: a graph attention network plus an MLP. The common-sense part is a spatial ontology, a bipartite graph saying which low-level mesh labels (sand, sink) can be found in which regions. GPT-4 writes the ontology, either by scoring sentences like “sink is often found in kitchen” or by being asked for the k labels that best distinguish each region. At training time a Logic Tensor Network turns the ontology into fuzzy-logic axioms, so a place whose mesh neighbours are mostly sand is pushed towards “beach” even with no label for it. At inference the LTN is dropped and only the GNN and MLP run.

Findings

  • With 0.1% of the training labels, the LTN raises place-labelling accuracy from 12.3% to 25.1% on Matterport3D (90 scenes) and from 29.0% to 37.2% on average on two outdoor datasets: West Point (Jackal rover with a RealSense D455 and a Velodyne, LOCUS odometry) and Castle Island (hand-held rig on a Boston beach, Kimera-VIO). Each outdoor trajectory is roughly half a kilometre to a kilometre.
  • Inference is cheap: 0.027 s, 0.067 s and 0.207 s per scene graph on an Intel i7 CPU for MP3D, West Point and Castle Island, whose graphs average 1,283, 3,930 and 11,059 nodes.
  • Against 38 human-judged label and region pairs, text scoring gets the best accuracy (0.81) and recall (0.95), and text completion with k=1 or k=3 gets precision 1.0. Precision is what helps the LTN, because the ontology is sparse: ontology-only training reaches 23.1% with the text-completion ontology and 12.1% with the text-scoring one.
  • With ontology-only training, the GPT-4 ontology beats a hand-written one outdoors (West Point 42.7% against 27.7%, Castle Island 28.7% against 22.7%) and loses indoors (23.1% against 27.7% on MP3D).
  • When six common room labels (bathroom, kitchen, hallway, bedroom, living room, family room) are removed from training, ontology-plus-data training scores 23.3% at 0.1% data against 12.5% for cross-entropy, so the ontology lets the model predict classes it never saw labelled.

Limits

  • The ontology has inclusion edges only and builds one region layer. The authors list more high-level layers, other relation types and better features as future work.
  • k has to be picked by test accuracy or against a small human ontology, and results swing with it: With ontology-only training, West Point gives 11.9% ± 14.7 at k=2 and 42.7% at k=3.
  • On Castle Island, ontology and data-driven models score about the same because the mesh labels do not separate the regions well; “backshore” gets called “beach”. Contradicting evidence also misleads it: an MP3D lounge with a day-bed becomes a bedroom.
  • Low-level semantics come from OneFormer with the closed ADE20k label set, so the open-vocabulary part is the LLM building the ontology offline, with no language query at run time.
  • The outdoor sets are small enough that both trajectories had to be pooled and split 70/15/15. Absolute accuracies stay low: the best MP3D figure in the tables is 52.9% with all labels.

Relevance

The ontology is the shared vocabulary that lets an outdoor graph have regions at all, and it is a prior that can be wrong. An open question for robust robotics is how a robot notices that its ontology and its observations disagree, and for a team, whether robots with different ontologies can still merge region layers (multi-robot multimodal perception, 3D scene graphs). The workshop version is strader-2023-common-sense, it builds on Hydra, and OpenGraph takes the other route outdoors, with captions and lane graphs.