TRACKGRAPH: Online Open-Vocabulary 3D Scene Graphs via Image-Space Tracking

2 min read

TrackGraph builds an open-vocabulary 3D scene graph online by keeping mask identities in the image stream before anything is fused into 3D. FastSAM masks and CLIP features are computed only at keyframes (every 24 frames in the benchmark setup). Between keyframes, dense DINOv3 ViT-S+ features carry the masks and their track identities forward by patch-similarity label propagation. Tracked masks are fused into a TSDF inside Hydra, where each voxel keeps up to four track hypotheses; persistent class-agnostic 3D segments come from shared mesh vertices, and a k-d tree over the centroids of segments that have left the 8 m active window handles revisits. Each segment keeps one CLIP feature per source track and is scored by the best match in that gallery.

Findings

  • On OpenLex3D: highest synonym frequency on Replica (0.50), second on ScanNet++ (0.35 against HOV-SG’s 0.40), and 0.32 on HM3D against 0.37 for OVI-MAP with SigLIP, while running 2.4x fewer segmentation and vision-language inference steps.
  • On Replica with an A100 at 3 cm voxels: 8.9 min and 8.3 GB of GPU memory, against 15.1 min and 27.2 GB for ViT-H OVI-MAP.
  • Replacing DINOv3 propagation with per-frame FastSAM plus BoT-SORT drops the rate from 15.0 Hz to 6.0 Hz with nearly identical AP.
  • Onboard a quadruped (Jetson AGX Thor, RealSense D455, LiDAR-inertial odometry) at 7.5 Hz, every queried object was found in three buildings. Recorded indoor drone RGB-D data is also mapped.

Limits

  • Retrieval AP on HM3D trails every encoder-matched baseline. The authors suspect segments stop covering whole objects in large scenes because instance continuity breaks.
  • Assumes good odometry and RGB-D. Thresholds were set once by hand. The efficiency table mixes hardware and frame rates, which the authors flag themselves.
  • Indoor scenes only.

Relevance

Resolving identity in image space first cuts segmentation and vision-language calls by more than half, which brings it to onboard rates. Revisit matching by centroid and appearance is the step most likely to break over weeks, when objects move or lighting differs. Built on Hydra; see 3d-scene-graphs, open-vocabulary-mapping and long-term autonomy.