3D Scene Graph Prediction: Generating Hierarchical Models from Partially Observed Environments

2 min read

A robot that has explored part of a building would like to guess the rest, for exploration or object search. This paper predicts the unexplored part as a hierarchical 3D scene graph, top-down. For the room layer (room categories, floor boundaries and which rooms connect), a new mixed discrete-continuous graph diffusion model generates the whole floor plan. Corruption and masking let it accept partial floor plans as constraints without being trained on partial data. For the object layer, an existing mixed diffusion model (MiDiffusion) places objects in each room given its floor plan. The pipeline runs on real scenes built by Hydra from robot data. Presented at IROS 2026; from the MIT SPARK Lab.

Findings

  • In unconditioned generation, continuous labels with discrete edges give the best trade-off of the four combinations (layout FID 20.95 against 4.06 for the held-out ground truth).
  • Completing partial scenes from 3D-FRONT (in distribution): FID 26.33, against 30.79 for an occupancy-based model (SSC) and above 120 for the LLM-based AnyHome and Holodeck.
  • Completing partial scenes from Matterport3D (out of distribution): FID 99.29, against 190.91 for SSC, 167.67 for AnyHome and 203.98 for Holodeck. This is the “generalises substantially better” claim.
  • The LLM-based methods do not respect the given partial scene, and SSC can drop semantic constraints on unfamiliar input. The cost of the method is a little overlap between predicted rooms.

Limits

  • Trained on 3D-FRONT, a synthetic dataset of furnished homes, so predictions follow its layout distribution. Offices, hospitals or outdoor sites would need other data.
  • The real-world demonstration is qualitative. There is no test yet of whether the predictions speed up exploration or object search.

Relevance

Most scene graph work describes what a robot has seen. This predicts what it has not, which turns the graph into a prior for planning; see language-grounded-planning. For a team, a predicted room layer could tell robots where to split up before they have seen the rooms. A prediction is a guess, so a map that mixes observed and predicted nodes needs to keep them apart, the same bookkeeping problem as knowing what is stale.