HOV-SG builds a floor, room and object hierarchy with CLIP features at every level, offline, from RGB-D frames and odometry, and uses it to answer queries like “towel in the bathroom on the upper floor”. SAM masks from each frame are lifted to 3D segments and merged across frames by grouping tightly connected parts of a segment-overlap graph, so one new segment can join several old ones. Each mask’s feature is a weighted sum of three CLIP embeddings: the whole frame, the mask’s bounding-box crop, and the mask with the background blanked out. These are written onto a reference point cloud and averaged per point; each segment then takes the feature nearest the mean of the largest DBSCAN cluster of its point features, a majority vote that throws out stray CLIP vectors. Floors come from peaks in a height histogram, rooms from a watershed over a bird’s-eye wall skeleton, and each room stores 10 representative CLIP embeddings (k-means over images whose camera pose lies inside it). A Voronoi graph covering every floor, joined through camera poses on stairs, makes the graph navigable. GPT-3.5 splits a query into floor, room and object, and each is matched top-down by cosine similarity.
Findings
- Open-vocabulary segmentation with ViT-H-14: Replica mIoU 0.231 and F-mIoU 0.386 against ConceptGraphs at 0.18 and 0.23; ScanNet mIoU 0.222 against 0.16.
- Storage over 8 HM3DSem scenes: 1493 MB in total, against 1638 MB for ConceptGraphs and 6068 MB for VLMaps, about 75% below VLMaps. HOV-SG and ConceptGraphs store 1024-dimensional ViT-H-14 features; VLMaps stores 512-dimensional LSeg features at 0.05 m voxels.
- Floors counted correctly in all 8 scenes. Region precision 84.10 and recall 83.59, against Hydra’s 86.18 and 77.55.
- Room labels from the 10 view embeddings: 73.93% exact and 84.10% judged correct by a human, close to GPT-4 given ground-truth object lists (84.25%) and well above GPT-4 given predicted objects (59.47% and 62.55%).
- Object labels against 1624 HM3DSem categories: AUC of top-k accuracy 84.88 against ConceptGraphs at 84.07. VLMaps finds the right label within its top 500 guesses only 40.02% of the time.
- Object, room and floor queries: 11.69 points better retrieval than a ConceptGraphs variant given privileged floor information, and 2.2 points better on object and room queries.
- Spot in a two-storey office (Azure Kinect, 3D LiDAR, FAST-LIO2 poses with loop closure): retrieval succeeded for 29 of 41 objects (70.7%), 5 of 9 rooms and 2 of 2 floors; navigation reached 23 of 41 objects (56.1%). Room failures came from meeting, seminar and dining rooms that all look like tables and chairs.
- The supplementary material localises a robot in the graph with a particle filter from RGB and odometry alone, at floor and room level within 10 frames.
Limits
- The authors admit many hyperparameters, a slow build that is not real time (no timing is given in the main text), and a static-world assumption.
- Floors from height peaks and rooms from wall skeletons only make sense in buildings with flat storeys. Open-plan kitchen and living areas already confuse the room step.
- Accurate odometry is assumed; the real run used LiDAR SLAM with loop closure. Room categories are subjective, so the 55.6% room score partly measures the labels.
Relevance
HOV-SG shows that a hierarchy helps language queries (“on floor 2”), and that a handful of CLIP vectors per room beats asking an LLM to guess the room from an object list. The floor and room layers are specific to buildings; outdoors the hierarchy has to come from somewhere else, as in language-enabled ontologies and OpenGraph. It builds on ConceptGraphs and Hydra; see open-vocabulary mapping, language-grounded planning and multi-robot multimodal perception.