Hydra++: Real-Time Hierarchical 3D Scene Graph Construction With Object-Level Shape Estimation

3 min read

Hydra++ replaces the centroid-and-bounding-box objects of Hydra and Khronos with full object meshes from a learned shape estimator. Objects are tracked across frames with a max-IoU voxel tracker; when a track leaves the active window, the frame with the largest 3D bounding box is sent to CRISP (category-agnostic, DINOv2 backbone, predicts pose, metric scale and a 256-d shape code decoded through an SDF) or, as a slower alternative, SAM3D (scale recovered as the median ratio of rendered to measured depth). A reprojection-mask consistency check (RMCC) samples the predicted mesh, projects it into the image and rejects the prediction if too little of the projected mask overlaps the observed one. For outdoor use, a hybrid LiDAR-camera mode takes metric depth from LiDAR, and a ground-aware TSDF tweak keeps measurements up to 3 m beyond the truncation band for ground voxels, so oblique LiDAR rays stop leaving holes in the ground mesh.

Findings

  • On uHumans2 at a strict 0.2 m centroid threshold: F1 0.675 against 0.461 for Hydra, 0.288 for Khronos and 0.089 for SlideSLAM. Chamfer distance drops to 0.004 against 0.035 (Hydra) and 0.028 (Khronos); volumetric IoU is 0.598 against 0.298 and 0.350.
  • At a relaxed 0.5 m threshold Hydra has the best F1 (0.789 vs 0.776) because of higher recall, but its shape metrics are much worse. The authors’ point: accumulating partial observations finds objects but cannot fill unseen geometry, so centroids drift beyond 0.2 m.
  • RMCC trades a little recall for precision: precision rises from 0.670 to 0.787 at 0.2 m while recall moves from 0.602 to 0.591.
  • SAM3D generalises better to an out-of-domain “computer” class but takes roughly 11 to 14 s per object, so the real-time claim applies only to the CRISP configuration.
  • Outdoors (a Kimera-Multi sequence with a RealSense D455 and Velodyne), the hybrid mode recovers fire hydrants, cars and benches that Khronos under-segments from sparse depth. These outdoor results are qualitative.

Limits

  • CRISP for real data was trained on about 50 keyframes across three classes, with SAM masks and iPad LiDAR scans hand-repaired in Blender to be watertight. Each new deployment class needs this effort.
  • The authors say plainly that outdoor benchmarks with instance-level mesh ground truth do not exist, which limits both training and evaluation outdoors.
  • Shape is estimated once, when the track goes inactive, from a single best view. Objects that move or change after that are not revisited.
  • Closed-set categories (four indoor classes plus “computer”); intra-class variety is handled, new classes are not, unless one pays SAM3D’s latency.
  • Single robot. Nothing on how two robots’ shape estimates of the same object would be reconciled.

Relevance

Shape estimates give a richer signal for matching objects between robots than centroids, but RMCC is also a small runtime plausibility check on a learned module: accept the network’s output only if it agrees with the measurement that produced it. That idea carries over to runtime monitoring of perception in robust robotics, and the hybrid LiDAR-camera mode is a step toward mixed-sensor scene graphs.