Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization

3 min read

Hydra builds a five-layer 3D scene graph (mesh, objects and agents, places, rooms, building) online from visual-inertial data, where its predecessors Kimera and 3D DSGs needed minutes of batch processing. Speed comes from locality: it keeps a TSDF and ESDF only inside an 8 m “active window” around the robot, grows a Generalized Voronoi Diagram as a by-product of the ESDF update, and sparsifies that into a graph of places. Rooms come from dilating the places graph (dropping places closer than δ to an obstacle, so doorways close), counting connected components over ten values of δ between 0.45 and 1.2 m, and filling in the rest with seeded Louvain community detection. The second half of the paper is about keeping the graph consistent: hierarchical loop-closure descriptors (place-distance histograms, then object-label histograms, then DBoW2 appearance) and an embedded deformation graph that corrects mesh, places, objects and agent poses together after a loop closure.

Findings

  • Built from ground-truth poses, the online graph is close to the batch Kimera graph: 80 to 100% of objects found and correct, and place position error under 25 cm (uHumans2 simulation and the real SidPac building, 5 trials each).
  • Room detection holds up on multi-floor scenes where Kimera’s fixed-height slice fails. On SidPac floors 3 and 4, Kimera reaches 0.88 precision but 0.06 recall (it finds 2 of 10 rooms).
  • The batch approach takes over 40 s to regenerate the graph for a moderate scene and grows linearly; Hydra’s mid-level cost stays flat. Per-layer times on a 24-core workstation are tens of milliseconds (objects 9.8 to 75.3 ms, places 3.4 to 8.1 ms, rooms 4.4 to 19.0 ms).
  • On an Nvidia Xavier NX (uHumans2 Office): objects 75 ± 35 ms, places 33 ± 6 ms, rooms 55 ± 41 ms, against a 5 Hz keyframe target.
  • Scene-graph loop closures give roughly twice as many loop closures within 10 cm and 1 degree of error as a permissive DBoW2 baseline (Office scene). Object-based registration with TEASER++ rescues matches that fail visual RANSAC.
  • The backend uses Graduated Non-Convexity in GTSAM, so wrong loop closures can be rejected as outliers, and an unmerged copy of the graph is kept so a bad merge can be undone.

Limits

  • Indoor, single building, single robot: the agent layer holds only one pose graph (the authors say so directly).
  • Rooms are found from free-space topology alone, so open floor plans with no doorways will not segment (admitted). Rooms are also unlabelled in this version; labelling comes in the journal version.
  • Closed-set semantics from a 2D segmentation network (HRNet on ADE20k); objects are Euclidean clusters with centroid and bounding box, so geometry is coarse.
  • The scene is assumed static. Moving objects are not modelled.
  • Ground-truth trajectories for SidPac are a hand-tuned pose graph, so “ground truth” is itself an estimate.

Relevance

Most later systems build on Hydra, including Hydra-Multi, Khronos and Hydra++. Its GNC backend is one of the few places in the 3D scene graph stack where outlier rejection is built in, which raises a question for outlier-robust estimation and robust robotics: what happens to the room and object layers when the loop closures that survive GNC are wrong in a coordinated way?