3-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agents

3 min read

Kim et al. build a 3D scene graph from a stream of RGB-D frames and then use it for question answering and PDDL task planning. Nodes are objects carrying a label, a 3D position stored as a Gaussian, a colour histogram and a thumbnail. Edges are directed and typed as actions, spatial relations, descriptions, prepositions or comparisons. The pipeline rejects blurry frames with an adaptive threshold on the variance of the Laplacian, groups frames into keyframe groups by overlap, detects objects with Faster R-CNN and relations with Factorizable Net, filters spurious detections with non-maximum suppression and relation priors mined from Visual Genome, and merges each per-frame graph into a global graph by “same node detection” on label, position and colour.

Findings

  • Same node detection is a weighted sum of label (0.375), colour (0.25) and position (0.375) similarity with a merge threshold of 0.7. The spurious detection module drops 68 predefined classes out of the 400 the detector knows (road, sky, building, moving objects).
  • Accuracy is judged by six human participants on one ScanNet living-room sequence of 5578 frames, counting spurious and missing nodes and edges with a two-thirds majority rule plus a 7-point Likert score. The result table is an image I could not extract, so I only have the text: the full model has the fewest spurious entities and the keyframe and blur modules cut runtime, but judges rated the slightly noisier 3D-efficient variant higher on overall accuracy. The authors read this as people preferring excess over omission.
  • The authors blame part of the recognition error on ScanNet blur: average blurriness around 200 against over 1000 for typical image datasets.
  • In the application demo a person scans a real kitchen, the graph is handed to a simulated Mybot in Webots, and a few rules turn it into a PDDL problem file for the FF planner, which then plans putting cups in the sink.

Limits

  • Static scenes only. The authors list moving objects, humans first, as the main gap.
  • Flat: objects and pairwise relations, with no rooms, places or traversable space. Rosinol et al. later make this criticism.
  • Very thin evaluation: one sequence, human judgement, no metric ground truth for positions.
  • Positions are relative to the first keyframe of a single camera, and the authors admit real-time operation is not guaranteed on one processing unit.
  • The merge rule is hand-tuned weights over colour histograms and word-vector label distances, so lighting changes or a different detector would shift it.

Relevance

It is one of the first papers to treat a 3D scene graph as the input to a symbolic planner, an early root of language-grounded planning over 3D scene graphs. Its same-node test (label, position, colour) is the association problem that multi-robot merging faces too, only here every observation comes from one trusted camera; see multi-robot multimodal perception.