3D VSG: Long-term Semantic Scene Change Prediction through 3D Variable Scene Graphs

3 min read

Looper et al. add a “variability” attribute to each object node of a 3D scene graph: the probability that the object changes before the robot next looks. They split long-term change into three kinds: position (the object moved beyond a threshold), state (open to closed, on to off) and instance (the object appeared or disappeared). A two-layer graph convolutional network, DeltaVSG, predicts all three from the current graph alone, using PCA-compressed semantic embeddings for nodes and edges that join nearby objects with their relation labels and relative position. They train on pairs of 3RScan scans of the same room taken hours to days apart, and then use the predictions to plan where to look first when checking a room for changes.

Findings

  • Over all three kinds of change DeltaVSG reaches 77.1% accuracy and 72.3% recall.
  • By kind: state 62.9% accuracy and 48.9 F1, position 88.1% and 66.6 F1, instance 68.5% and 30.1 F1. The 3DSSG network fed the same graph embeddings is slightly better on position (88.7% / 68.7) and instance F1 (31.6), so the gain is modest and uneven.
  • Change is rare: only 21% of nodes have a state change, 17% a position change and 13% an instance change. Treating scan pairs as unordered and symmetric gives 3650 training samples.
  • Local context is what matters: connecting objects within 3.70 m works best, and connecting every pair (up to 15.5 m) hurts. Raw 619-dimensional binary embeddings fail outright (F1 of 0.0 on state and instance); PCA to 120 dimensions works.
  • In active change detection, a planner that visits the most variable objects first finds a shorter path in 68.7% of cases and finishes 66.0% faster on average than a travelling-salesman coverage tour.

Limits

  • Trained and tested on ground-truth scene graphs, so perception noise is not part of the problem.
  • Assumes a Markovian world and that change is equally likely in either time direction, which makes the data augmentation work.
  • Many false positives and low precision, which the authors explain by many plausible futures with only one realised in the data.
  • Predicts whether an object changes, not where it goes or what appears; indoor rooms only.

Relevance

A learned prior over which nodes are expected to change could also help outside change detection. If a team of robots disagrees about an object, a variability prior could help tell an expected change (a moved mug) from a suspicious report (a moved fridge), which is an open question that touches both long-term autonomy and robust robotics. It builds on wald-2020-3dssg; compare schmid-2024-khronos in the dynamic scenes thread.