3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera

3 min read

Armeni et al. define the four-layer 3D scene graph that most later work inherits: a building node, room nodes, object nodes and camera nodes, each with attributes (class, material, volume, pose, field of view) and with relations inside and across layers (occlusion, amodal mask, spatial order, relative volume, parent room). The graph is built offline from a 3D mesh plus registered RGB panoramas, the kind of data a Matterport-style scanner produces. Because labelling whole buildings by hand is slow, they bootstrap it with Mask R-CNN and two fixes: “framing”, which samples 225 rectilinear 800 by 800 crops per panorama so each object sits near the centre of some crop, and multi-view consistency, which projects panorama labels onto the mesh and votes across camera positions, weighting closer cameras more. Crowd workers then verify and patch the result.

Findings

  • On Gibson panoramas, detection AP rises from 0.079 (plain Mask R-CNN on 6 crops) to 0.160 with framing and 0.485 with framing plus multi-view consistency. On the mesh, AP goes from 0.222 to 0.306 to 0.409.
  • With the weaker BlitzNet detector (COCO AP 34.1 against 41.5 for Mask R-CNN) the relative gains hold: 2D AP 0.095 to 0.284, 3D AP 0.076 to 0.245.
  • Human labour per model: fully automatic gives AP 0.389 at zero hours; with manual verification AP 0.97 in 3 h 18 min; fully manual 2D labelling takes 12 h 44 min, and an expert 3D annotation of only 12 classes took 10 h 18 min.
  • Relations learned from the automatic labels beat a statistical guess: spatial order mAP 0.712 against 0.255, relative volume 0.820 against 0.555. Amodal mask prediction reaches average F1 0.672 against 0.545 for a class-specific average mask.
  • The pipeline annotates all 572 Gibson buildings; manual verification is only on the tiny split.

Limits

  • Offline and survey-based: it needs a complete mesh and registered panoramas, so it says nothing about building the graph from a moving robot’s sensor stream.
  • Indoor static scenes only, and no action relations, which the authors state.
  • No free-space or traversability layer, so the graph is good for organising knowledge but hard to plan on. Rosinol et al. point this out in rosinol-2020-3d-dsg.
  • Materials were labelled by hand because existing networks were not accurate enough; object classes come from COCO.

Relevance

Later 3D scene graphs start from this hierarchy; Kimera and Hydra keep its building and room layers and add places and agents. Its multi-view consistency step votes across camera positions of one careful survey and weights each view by distance alone. Across a robot team the same vote would mix views from agents with different poses, detectors and trustworthiness, which is an open question for multi-robot multimodal perception and robust-estimation.