3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans

3 min read

Rosinol et al. define the 3D Dynamic Scene Graph: a layered directed graph with five layers, from a metric-semantic mesh at the bottom, through objects and agents, places and structures, and rooms, up to a single building node. Agents (humans and robots) are nodes that carry a pose graph over time, a mesh and a class, so people and robots stay in the graph with their history. The graph is built automatically from stereo and IMU data by a “spatial perception engine” on top of Kimera: the mesh and agent nodes are built online, while objects, places, structures and rooms are extracted at the end of the run. Places come from the free-space topology of the ESDF, which is the layer that makes the graph usable for path planning.

Findings

  • All experiments are in a Unity simulator, a 65 m by 65 m office with 12, 24 and 60 simulated humans (the uHumans sequences).
  • IMU-aware feature tracking plus 2-point RANSAC keeps odometry usable in crowds: trajectory error on the three crowded sequences drops from 92, 145 and 160 cm (5-point RANSAC) to 59, 78 and 88 cm, while staying on par on the static EuRoC sequences.
  • Dynamic masking (treating human pixels as free space when ray casting) stops people leaving “contrails” in the mesh. With ground-truth poses, mesh RMSE goes from 0.089 / 0.133 / 0.192 m without masking to 0.060 / 0.061 / 0.061 m with it.
  • Human tracking with a pose graph, a zero-velocity prior and a joint-motion plausibility check (at most 3 m per joint in 1 s) cuts torso error from 1.07 to 1.20 m for single-image SMPL estimates down to 0.61 to 0.65 m.
  • Objects of unknown shape are localised to 1.31 to 1.70 m; fitting a CAD model with TEASER++ (only for couches) brings it to 0.20 to 0.38 m.
  • Classifying places into rooms on the first sequence gives 99.89% precision and 99.84% recall, with errors near doors.

Limits

  • Simulation only, and the simulator supplies ground-truth panoptic segmentation.
  • Everything above the mesh and agents is built offline after the run.
  • A single robot. The authors close by asking whether distributed engines could build a DSG from data collected by several robots.
  • The room heuristic cuts the ESDF 0.3 m below a detected ceiling, so it assumes flat ceilings and a known gravity direction.
  • Human data association is a set of hand-set gates; unknown-shape object positions are off by more than a metre.

Relevance

This paper adds the two things later robot systems rely on: a places layer for planning and agents as first-class nodes. Agent nodes with their own pose graphs are also how a team would represent its teammates, and the authors’ closing question about distributed engines is where multi-robot multimodal perception with 3D scene graphs starts. Its dynamic masking and human tracking are an early take on dynamic scenes; the journal version is rosinol-2021-kimera, and hughes-2022-hydra makes the whole graph incremental.