Hierarchical Object Representation for Spatial Robot Perception: Points, Meshes, and Superquadrics

4 min read

Hickory gives each object node in a 3D scene graph four geometric layers, where most pipelines keep only a partial point cloud or a bounding box. Layer 1 is the accumulated object point cloud, Layer 2 a full textured mesh from SAM3D, Layer 3 a few superquadric primitives fitted to that mesh, and Layer 4 the same superquadrics inflated by a safety margin for collision checking. The pipeline takes posed RGB-D frames, segments objects class-agnostically with FastSAM, throws away walls, floors, sky and people using OneFormer, and tracks objects by 3D IoU on oriented boxes. SAM3D is expensive, so each object gets one “best view”, scored from DINOv2 feature variance, mask area and the share of valid depth pixels; FoundationPose then refines the pose using the SAM3D mesh as a template. For map alignment the authors keep ROMAN’s machinery (unary and pairwise association scores, CLIPPER for inliers) and swap only its point-cloud geometric score for one that compares the shape and scale parameters of the first K superquadrics.

Findings

  • Storage per scene: dense meshes take 232 to 484 MiB across HOPE, ReplicaCAD, Kimera-Multi and their own NUS-CLB sequence, while the superquadric layers take 51 to 73 KiB. Mesh building is the slowest stage (168 to 393 s per scene on an i9 with an RTX 4090), and the authors put that down to SAM3D.
  • Best-view selection gives mean F1 of 0.931 on HOPE and 0.872 on ReplicaCAD, against 0.922 and 0.865 for the strongest heuristic (largest mask area, largest 3D volume). ADD-S moves from 0.83 to 0.84 on HOPE. The gains are real but small.
  • Map alignment on Kimera-Multi (outdoor, two robots, three overlapping scene pairs): average translation error 0.24 m, against 0.82 m for ROMAN-XL and 1.29 m for ROMAN-One, using fewer associations (15.7 against 27.3 and 40.3). ROMAN-One’s 3.26 m error on the opposite-direction pair comes from piling up false matches.
  • ROMAN still wins some cells: ROMAN-XL has the lowest average rotation error outdoors (0.18° against 0.56°), and ROMAN-One the lowest translation error indoors on ReplicaCAD (0.030 m against 0.039 m).
  • Navigation with RRT on a Unitree B2 (100 trials for each of 5 seeds): inflated superquadrics reach 100% success and an efficiency ratio of 0.885, close to the full mesh (100%, 0.893), but build the occupancy map in 0.27 s instead of 209 s. Mesh bounding boxes build in 0.014 s but drop success to 89.8%, mostly because the box under a table blocks the path the robot could duck through.

Limits

  • It needs camera poses from elsewhere and runs as an offline back end. The authors admit the runtime and compute of SAM3D-style models is the main limitation.
  • Static world: people and other dynamic classes are filtered at the first step, and nothing tracks objects that move between sessions, although the dataset caption notes that dynamic objects differ between Kimera-Multi recordings.
  • Each mesh comes from one view, so the hidden side is the model’s guess. Alignment works only if that guess comes out the same in both sessions. The claim that the first few superquadrics stay stable across views and trajectories is an empirical observation with no guarantee behind it.
  • The opposite-direction pair (Kimera-Multi scene 3) is still the hardest case: 0.50 m and 1.31° error.

Relevance

Shape-based object association is one of the few cross-robot cues that needs neither a shared frame nor matching appearance, so it bears on collaborative SLAM and multi-robot perception. Untested here: whether superquadric parameters stay comparable when two robots carry different cameras, or when one robot’s segmentation is simply wrong. It deepens the object layer of 3d-scene-graphs built by systems like Hydra; Picasso shares an author and also post-corrects SAM3D output.