CoGaussian: Open-Vocabulary Gaussian-based Collaborative 3D Scene Understanding

2 min read

Collaborative semantic occupancy methods for connected vehicles need dense 3D voxel labels and a fixed class list. CoGaussian has each agent predict a sparse set of 3D Gaussians carrying DINOv2 features, trained by rendering them back into the cameras, so no 3D labels are needed and classes can be queried in language. The Gaussians double as the message between agents. Three ways to combine them are compared: concatenating neighbours’ Gaussians zero-shot, joint per-scene training where all agents’ Gaussians are supervised together, and a radius-gated attention module that updates the ego agent’s Gaussians from its neighbours’.

Findings

On Semantic-OPV2V, against the single-agent self-supervised baseline GaussTR (IoU 20.58, mIoU 6.26):

  • Zero-shot concatenation hurts: IoU 16.13, mIoU 4.81.
  • Joint training helps: 22.54 and 6.74.
  • Radius-gated attention is best: 22.57 and 6.95.
  • Fully supervised, closed-vocabulary methods in the same table reach IoU 38.5 to 72.9 and mIoU 24.8 to 37.4. The authors say these are not comparable because they use dense 3D labels, but the gap shows how far behind the self-supervised version is.
  • Road, sidewalk and vehicles gain most from collaboration, the classes agents overlap on most. Thin, rare structures stay hard.

Limits

  • Not real time yet; efficient communication and inference are future work (authors).
  • One simulated driving dataset (Semantic-OPV2V). The paper reports nothing on pose error, delay or lost messages.
  • Merging Gaussians naively makes things worse, so agents have to be trained together. That suggests a new teammate running a different model would not simply plug in.

Relevance

The Gaussians are the shared map and the message at once, which is one answer to the question in semantic-coordination of what robots with different views should exchange. The zero-shot result is a warning for open-vocabulary-mapping in teams: features from two viewpoints do not simply add up. The mIoU gap to supervised methods (about 7 against up to 37) shows how far open-vocabulary occupancy still has to go. See multi-domain-perception and gaussian-splatting.