vS-Graphs: Tightly Coupling Visual SLAM and 3D Scene Graphs Exploiting Hierarchical Scene Understanding

3 min read

vS-Graphs is the camera version of S-Graphs: it adds two threads to ORB-SLAM3 so that walls, ground surfaces, rooms and floors become variables in the same local bundle adjustment as the keyframe poses and map points. A “building component recognition” thread runs panoptic segmentation (Panoptic-FCN or YOSO) on each keyframe, keeps wall and ground pixels, back-projects them with depth, downsamples and range-filters the points, and fits planes with RANSAC. A “structural element recognition” thread runs every two seconds, merges duplicate planes by distance and normal angle, and detects rooms as convex regions bounded by at least two walls that face a free-space cluster from Voxblox. Room factors penalise deviation from parallel or perpendicular wall pairs where those exist, and otherwise only keep the room centroid at the mean of its wall centroids, so rooms need not be four-walled rectangles.

Findings

  • The abstract reports an average 15.22% accuracy gain over best published VSLAM methods across all tested datasets (ICL, OpenLORIS, ScanNet, TUM-RGBD and the authors’ SMapper set). In the text, room constraints reduce ATE by 13.82% and 15.22% against ORB-SLAM3 on the looped “deer-w” and multi-room “MR01” sequences, and loop closures plus the new constraints give up to 75.38% gain on “deer-w” with YOSO.
  • The choice of segmentation backbone has only a marginal effect on trajectory accuracy.
  • Mapping RMSE against SMapper’s LiDAR-derived ground truth has a lower median than ORB-SLAM3 while producing about 10.15% fewer map points.
  • On multi-room SMapper sequences, wall and room detection is comparable to LiDAR-based S-Graphs, with higher recall in the larger MR03 sequence. Hydra is run on visual point clouds for the comparison and scored on rooms only.
  • Runs at 22 ± 3 FPS on a laptop (i9-11950H, 4 GB T600 GPU), against 29 ± 3 FPS for plain ORB-SLAM3 on the same hardware.
  • The SMapper in-house dataset covers about 44 minutes and 620 m of trajectories.

Limits

  • The authors list low-texture scenes (uniformly painted corridors) as the main failure mode, since feature tracking errors propagate into wall and room detection. Rapid motion and noisy depth also hurt (sequence “SR01”).
  • Planar walls only; curved walls are out of scope (admitted). Association thresholds need careful tuning (admitted).
  • The floor layer is effectively disabled: the current implementation has a single-floor design constraint.
  • Loop closure is still geometric. Semantic loop closure from rooms and walls is future work.
  • RGB-D only, indoors, single robot, static scenes. Ground truth for SMapper is itself produced by S-Graphs on LiDAR.

Relevance

vS-Graphs shows a camera-only robot can recover the same room and wall structure as a LiDAR one, which helps a team where only some robots carry LiDAR: in principle a camera robot and a LiDAR robot could agree on rooms even if their point clouds look nothing alike. Whether those room estimates are consistent enough to merge across sensors is an open question for collaborative SLAM and multi-robot multimodal perception.