xFLIE: Leveraging Actionable Hierarchical Scene Representation for Autonomous Semantic-Aware Inspection Missions

3 min read

xFLIE is the journal-length version of LSG: the same four-layer Target, Level, Pose, Feature graph (renamed 3DLSG) built incrementally by the FLIE inspect-explore planner, now with a formal problem statement, more scenarios and a comparison against volumetric planning. It adds a scene-recognition step, LightGlue matching against images from the current level, to decide when one level of inspection is complete and the robot should move up. A pose-feature graph is pruned at the end of each level so every feature keeps only its highest-confidence view. Vehicles and car parts are segmented with closed-set YOLOv8 models; the collapsed-building simulation uses CLIPSeg, a text-prompted segmenter, for buildings, windows, people and rubble. The graph lives in NetworkX in Python, and only YOLO and LightGlue use the GPU.

Findings

  • Simulated city of 150 m by 100 m with 20 vehicles: 17 of 20 inspected in one run (85%), and 19 in other runs (95%). Average planning time on the graph was 0.029 to 0.037 ms across the 5, 10 and 20 vehicle scenarios; nearly doubling the node count raised planning time by 12%.
  • Against D*+ on Octomaps at 0.5 m and 0.7 m, graph planning took 1.6e-5 to 7.7e-5 s on the Target layer and up to 5.4e-4 s on the Pose layer. The volumetric planner took 0.11 to 1.9 s. Graph paths were on average 1.29 and 1.25 times longer.
  • Against a baseline using the same graph for decisions with D*+ for motion, and against GBPlanner2.0, xFLIE used 42.7 to 52.1% CPU (baseline 103.6 to 118.4%, GBPlanner2.0 87.6 to 101.4%), covered up to 11.6% more target volume within a fixed time budget, and localised targets with RMSE 0.17 to 0.24 m.
  • Field trials on Spot (Jetson Orin, RealSense D455, VLP-16, VN-100, LIO-SAM odometry): a 40 m by 24 m campus with up to five vehicles, and a dark, GPS-denied subterranean site of 10 m by 6 m by 4 m with one car. Every target was detected and inspected, planning took 0.05 to 0.07 ms, and merging local graphs into the unified graph cut the cumulative node count by 26% on average.
  • The campus was not controlled: pedestrians, service vehicles and changing light were present throughout.

Limits

The authors list most of these in a “lessons learned” section.

  • Segmentation quality drove most failures. Over-segmentation pulled background pixels into a target’s position; the mask erosion used to fix it was hand-tuned per class and fixed per scenario.
  • Targets seen in non-consecutive frames slipped past de-duplication. The duplicates inflated centrality and in one trial sent the robot past a nearer car to a distant one. The workaround was a gentler yaw rate.
  • A timing mismatch between YOLO and odometry misplaced a target during the initial 360 degree survey, and one false target node survived in the three-car run.
  • The aerial platform was only simulated (RotorS). The planner assumes a holonomic robot, and a single parameter tells it whether vertical inspection is possible.
  • No comparison with other 3D scene graph systems; the authors say those address different problems.

Relevance

For small drones the useful number is the split: planning over the graph costs fractions of a millisecond on a CPU, so the compute budget goes to segmentation. The authors’ proposed future work, one robot exploring and filling the upper layers while another inspects and fills the lower ones, is an open question for multi-robot multimodal perception: duplicate target nodes already confuse one robot, and two robots would need a rule for reconciling them. See 3d-scene-graphs and aerial-swarms.