Seeing Fast and Slow: Bimodal 3D Scene Graphs for Open-set Tasks

3 min read

BiMoSG builds a 3D scene graph cheaply by default and pays for foundation models only where the task needs them. The “fast” mode uses closed-set panoptic segmentation (Mask2Former trained on ADE20K’s 150 classes) and turns each object into a polygonal prism: the mask contour is projected to 3D, cleaned, flattened to a convex polygon in the ground plane, and given a minimum and maximum height. Objects are grouped into “vicinities” around a central object, ranked by size and co-visibility, and vicinities into zones that roughly match rooms, with edges labelled ON, IN, ABOVE, UNDER or BESIDE. A GPT-4o planner lists trigger labels for the task, and when one appears it switches to the “slow” mode: SmolVLM2 captions the object, LangSAM segments it, and the open-set object is added to the graph. A hybrid mode instead stores a CLIP embedding per mask and per wider crop.

Findings

  • Replica, 8 scenes, geometry stage only, on an RTX 4090: 0.1094 s per frame against 0.4003 s for ConceptGraphs and 0.2277 s for BBQ. The authors summarise this as at least 3 times faster than ConceptGraphs.
  • Prism accuracy averaged over the same 8 scenes: precision 0.350, recall 0.705, geometric IoU 0.279, and 0.614 for an mIoU computed on cube roots of volume.
  • Habitat-Matterport3D object search in three scenes (9, 15 and 21 rooms): faster than Clio in most of the 30 listed tasks, for example 50 s against 2:11 for “unicorn head” in the large scene. Clio is faster for toilet paper in the small scene (44.68 s against 58 s), and Clio returned more than 10 false positives in most tasks.
  • Four simulated mode-switching tasks take 19 s to 57 s, with 80% to 90% of ground-truth objects in the graph at success. There is no baseline in that table.
  • A Clearpath Jackal in a museum mock-up tracks “bag” and “toy” in fast mode, switches to the VLM, and confirms a backpack. Qualitative only.

Limits

  • If the target is outside the closed set and not where the planner expects, the robot has to search every corner (authors).
  • Prisms suit furniture packed in cuboid-like shapes. Precision of 0.23 to 0.43 per scene is blamed on Mask2Former errors, which the evaluation then leaves out.
  • The runtime comparison times only the geometric stage. The Clio runs mapped the environment before each task, which the authors say inflates Clio’s times, so the search-time gap mixes method and protocol.
  • The planner calls GPT-4o, so task execution depends on a network link and a remote model.
  • Single robot, indoor, static scenes. Some merge rules are left out “for brevity”.
  • I read the v1 LaTeX source; arXiv now lists a v3 (October 2026), which may differ.

Relevance

Spending heavy perception only where the task points is a sensible pattern for robots on small power and compute budgets, such as small aerial vehicles, and it fits open-vocabulary mapping and language-grounded planning. It switches between closed-set building, as in Hydra, and open-set mapping, as in Clio, which gives one more option for multi-robot multimodal perception.