An Actionable Hierarchical Scene Representation Enhancing Autonomous Inspection Missions in Unknown Environments

3 min read

Layered Semantic Graphs (LSG) builds a hierarchical scene graph top-down from the concepts an inspection planner uses. The four layers are Target (a car or house worth inspecting), Level (each height band of inspection), Pose (the view poses the planner commanded) and Feature (segmented parts such as a door or a bonnet). Each lower graph is stored as an attribute of its parent node and there are no inter-layer edges, so the path planner only sees the small local graph it needs and opens the nested one through “layer frontier” nodes. The graph is tied to the FLIE inspect-explore planner, which keeps no global 3D map: duplicate targets are caught by checking whether a new detection falls inside the polygon of view poses around an already inspected target. Semantics come from closed-set YOLOv8 segmentation (a COCO model for targets, a CarParts model for car features).

Findings

  • Field trial on a Boston Dynamics Spot with a Jetson Orin, RealSense D455, Velodyne VLP-16 and VectorNav VN-100, inspecting two parked cars on the Luleå campus: both were detected, inspected and added to the graph. The robot returned to base at about 420 s and then answered operator queries from about 430 s to 710 s.
  • Dijkstra over the nested graphs planned in 0.049 to 0.059 ms on the Target layer (four of the five field queries needed that step) and 0.12 to 0.20 ms on the Pose layer, with Pose-layer graphs of 23 to 47 nodes.
  • Operator queries follow a fixed template, for example “Visit front-bumper-1 in Level-0 of car-1”, and Spot drove to the pose from which that bumper had been seen.
  • In a simulated city (Gazebo with RotorS) with seven targets, six were inspected. The seventh was never segmented by YOLO during exploration.
  • Against D*+ on an Octomap with 0.8 m voxels, the authors report orders of magnitude lower planning time in simulation; this is shown in a figure, without a table of numbers.
  • Target selection is a weighted sum of mask area, centrality among other targets and inverse distance to the robot. Detections within 5 m of each other are pruned to the one with higher confidence and larger mask.

Limits

  • Closed-set classes. A target the detector misses never enters the graph, which the authors give as the reason for the six of seven result.
  • No edge types for traversability. The authors say planning cost may grow once the graph serves several tasks, and leave this to future work.
  • Graph paths are not collision checked. The authors suggest using the graph route as a high-level plan for a volumetric planner to refine.
  • Each nested graph keeps a copy of its parent node, so node counts are inflated until the local graphs are merged by graph union.
  • One field site, two static cars, one robot. Queries are template strings over node labels, so there is no free-form language.

Relevance

LSG builds its hierarchy from planner concepts, where Hydra builds places, rooms and buildings from geometry. The nested layout raises a question for multi-robot multimodal perception: if one robot owns a target node and everything nested under it, how would a second robot that saw the same car from the other side merge its copy? The journal version, with more trials and an aerial simulation, is xFLIE; see also 3d-scene-graphs.