OpenGraph: Open-Vocabulary Hierarchical 3D Graph Representation in Large-Scale Outdoor Environments

3 min read

OpenGraph builds an open-vocabulary hierarchical graph of a driving-scale outdoor scene from RGB images, LiDAR scans and poses. Each frame goes through three foundation models in turn: RAM names the categories in the image, Grounding DINO boxes them, and TAP (Tokenize Anything via Prompting) segments and captions each box. Each node’s semantics is its caption, embedded with SBERT, with no CLIP vector stored. The mask is lifted onto LiDAR points (after 4DMOS removes moving points and DBSCAN cleans the cluster). Detections merge into existing objects by a weighted sum of 3D box IoU, TF-IDF caption similarity and SBERT cosine similarity, and LLaMA rewrites the two captions into one when objects merge. The hierarchy is point cloud, lane graph (built from the robot’s own past trajectory by finding turns, junctions and breakpoints), instances with LLaMA-inferred edges over a minimum spanning tree, road segments (straight, L-, T- and full intersections), and one environment node.

Findings

  • Zero-shot 3D semantic segmentation on SemanticKITTI (19 classes, sequences 03, 05, 08) gives average IoU 0.605, 0.660 and 0.646 and F1 0.730, 0.775 and 0.763. That edges out the supervised baselines in these tables (RangeNet++, DeepLab v3, DecoupleSegNets; DecoupleSegNets reaches 0.603, 0.590 and 0.638 average IoU) and is well above swapping in CLIP or BLIP features (0.275 to 0.402).
  • Object retrieval with 30 queries per type: on “ontology” queries (“a tall tree”) R@1 is 0.90 against 0.60 for CLIP features; on proximity queries 0.80 against 0.37; on function queries (“something that can be used for driving”) 0.63, and 0.67 when GPT-4 reads the stored captions directly (OpenGraph-LLM).
  • The lane and segment layers support coarse self-localisation from a description of the road type and nearby objects, path planning on the lane graph between described start and goal points, and a human-reported map edit (a removed bush); these are shown qualitatively.

Limits

  • No runtime, memory or model-size figures are given, even though every frame runs three large vision models, SBERT and LLaMA merges. On-board use is untested.
  • The lane graph comes from the robot’s own driven trajectory projected to 2D, so the hierarchy assumes roads and covers only places the robot drove. Segments are defined by road topology, which has no obvious analogue off-road or in the air.
  • Evaluation is a single dataset, retrieval ground truth is hand-picked on small segments with 10 descriptions per query type per sequence, and per-class IoU is uneven (vegetation 0.008 and traffic sign 0.0 on sequence 03).
  • The authors list richer edge semantics between instances and part-level understanding as future work.

Relevance

OpenGraph replaces CLIP vectors with captions as the node feature outdoors, at a compute cost the paper does not measure. The trade is relevant to multi-robot multimodal perception: a caption is a short string that could plausibly cross a weak link more cheaply than a CLIP vector. Compare strader-2023-ontology for the other outdoor hierarchy and ConceptGraphs for the indoor caption pipeline it extends; see also open-vocabulary mapping.