Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs

3 min read

Kimera is the journal version that joins the Kimera metric-semantic SLAM library with 3D Dynamic Scene Graphs. Kimera-Core runs in real time from stereo or RGB-D plus IMU: visual-inertial odometry with IMU preintegration and fixed-lag smoothing, a fast local mesher, a volumetric semantic mesh built on Voxblox, and a new Kimera-PGMO module that corrects both the trajectory and the mesh on loop closure by optimising a deformation graph, which they show reduces to ordinary pose graph optimisation in GTSAM. Loop closures pass through an incremental version of PCM (pairwise consistent measurement set maximisation) with an added odometry consistency check, so the back-end itself rejects wrong closures caused by perceptual aliasing. Kimera-DSG then builds objects, humans, places, structures, rooms and buildings on top of the optimised mesh, offline at the end of the run.

Findings

  • On EuRoC, Kimera-VIO’s trajectory RMSE is 0.05 to 0.21 m and best among the fixed-lag smoothers on most sequences. In the loop-closure group ORB-SLAM3 is clearly ahead (0.01 to 0.08 m). On V1_01, PGMO moves trajectory RMSE from 15 cm to 13 cm.
  • With PCM, Kimera-RPGO is fairly insensitive to the DBoW2 loop-closure threshold, so the loop detector can be left permissive and the back-end does the filtering.
  • Semantic mesh: bundled ray casting costs little (under 8 cm mesh error, over 94% label accuracy) and VIO drift is under 0.2% (4 cm over 32 m). Dense stereo on textureless walls is the biggest loss.
  • Timing on EuRoC: IMU preintegration about 40 µs, feature tracking 7.5 ms per frame and 51 ms per keyframe, per-frame mesh under 7 ms, back-end under 60 ms, loop detection up to 180 ms, RPGO up to 50 ms on 728 nodes and PGMO up to 140 ms on 1759 nodes. Kimera-Core also runs in real time on a Jetson TX2, with semantic updates averaging 65.8 ms.
  • Building the full DSG for the largest scene, a simulated subway of about 3000 m², takes about 30 minutes, dominated by the ESDF, the topological map and object segmentation.
  • Room parsing: 99% / 99% precision and recall on the simulated office, 87% / 92% on the subway, 93% / 94% and 91% / 90% on two real scenes. It fails on a simulated apartment with exposed ceiling beams (68% / 61%) and on a narrow-corridor real building (71% / 67%).
  • Hierarchical A* over buildings, rooms and places answers a one-building query in 0.340 s against 420.4 s for A* on the ESDF, and a six-building query in 1.244 s against 21389.2 s, with paths only slightly longer (56.3 m against 54.2 m for one building).

Limits

  • The graph above the mesh is still built offline; the authors put incremental real-time construction first in their future work.
  • Single robot and a single sensor suite; heterogeneous sensors and multi-robot data are listed as future work.
  • Simulation experiments use ground-truth 2D semantics, so the semantic numbers do not include segmentation error.
  • Room segmentation needs tuning and over-segments unusual architecture.
  • Human tracking results are worst in the apartment scene, which the authors attribute to specular reflections.

Relevance

The PCM outlier rejection here was designed for inter-robot loop closures and adapted to one robot, which ties robust-estimation to 3D scene graphs. PCM checks geometric consistency between loop closures, which helps against perceptual aliasing but assumes the odometry itself is honest, a gap for robust robotics. The planning numbers argue for hierarchy in language-grounded planning; hughes-2022-hydra makes the upper layers incremental and chang-2023-hydra-multi extends the pipeline to teams.