Clio: Real-time Task-Driven Open-Set 3D Scene Graphs

4 min read

Clio decides how finely to carve a scene into objects by looking at the task list. The robot is given natural-language tasks at the start (“read brown textbook”), over-segments the world into task-agnostic primitives, and then clusters them with an incremental Agglomerative Information Bottleneck so that the map keeps only what is informative about those tasks. Object primitives come from FastSAM segments tracked into 3D in the style of Khronos, each with an averaged CLIP vector; place primitives come from Hydra’s Voronoi-based places, each holding the average CLIP embedding of every whole image from which its centroid is visible. Each primitive’s task distribution p(y|x) is built from CLIP cosine similarity to each task, plus a null task scored at α = 0.23 that soaks up anything irrelevant. Neighbouring primitives merge in order of a Jensen-Shannon cost until the fractional loss of task information passes a threshold. Clustering runs per connected component, so only the parts touched by new data are redone and the cost per update does not grow with the map.

Findings

  • Cubicle scene (CLIP ViT-L/14, RTX 3090): Clio-online reaches strict F1 0.62 with 92 objects at 0.30 s per frame. ConceptGraphs (run with CLIP only) gets 0.25 with 181 objects at 2.0 s per frame, and the unclustered primitives number 1070.
  • Office scene: Clio-online keeps 49 objects out of 1883 primitives, F1 0.60, against ConceptGraphs at 0.13 and 8.1 s per frame. Across scenes Clio keeps up to an order of magnitude fewer objects and runs about 6 times faster than ConceptGraphs.
  • Filtering a task-agnostic map after the fact (the “-thres” baselines) raises precision but cannot fix how objects were formed, such as whether a stack of notebooks is one object or several, so recall drops.
  • Batch clustering over a whole scene takes about 30 s. With the larger OpenCLIP ViT-H-14 the per-frame time rises to about 1.0 to 1.2 s and the null threshold has to move to 0.26, so only the batch version was run.
  • Closed-set Replica segmentation: mAcc 37.95 and F-mIoU 36.98, near ConceptGraphs (40.63 and 35.95).
  • Region clustering beats Hydra’s geometric rooms in the five-floor Building (F1 0.81 against 0.78) and Office (0.76 against 0.70) where open-plan areas need semantics, and loses in the Apartment (0.69 against 0.90) where walls already separate rooms.
  • On Spot, with a laptop carrying an i9-13950HX, 64 GB RAM and an RTX 4090 laptop GPU, 21 grasp attempts succeeded 57% of the time, 71% if Spot API grasp failures are excluded; Clio picked the wrong object once. Paths are Dijkstra over place nodes and the target is the node with the highest CLIP similarity to the prompt. The counted trials ran the laptop off board over WiFi; the video shows an on-board run.

Limits

  • The authors list sensitivity to prompt wording and CLIP model, plain averaging of CLIP vectors when primitives merge, over-clustering when two objects score alike against one task but must stay separate (a fork and a knife when laying a table), and simple single-step tasks only.
  • The task list is fixed before the robot moves. What happens when a new task arrives mid-mission is not evaluated.
  • Thresholds depend on the embedding model (0.23 for one CLIP, 0.26 for another), so swapping models means retuning.
  • Indoor RGB-D with good odometry is assumed. Objects carry no relations beyond the object, region, place hierarchy.

Relevance

Clio argues that a map should be compressed for its tasks, which helps when a map has to cross a thin radio link. If two robots carry different task lists their compressed graphs will disagree on what counts as an object, so merging them needs more than geometric alignment. It builds on Hydra and is compared against ConceptGraphs; see open-vocabulary mapping, multi-robot multimodal perception and long-term autonomy.