3D Scene Graphs: Open Challenges and Future Directions

9 min read

This survey of 3D scene graphs is written by people from most of the groups that built the field (Stuttgart, Sapienza, Google, MIT, Freiburg, UTN, Montreal, TU Munich). It gives one formal definition that covers the variants, splits construction into five design axes, sorts applications and their evaluation protocols, and ends every section with a paragraph of open challenges. A companion site, 3dscenegraphs.com, indexes the papers by keyword and already listed more than 150 when the survey was submitted. It is a good first read, and the open-challenge paragraphs can be read on their own.

Taxonomy

The definition is a tuple G = (V, V_g, V_f, E, E_f, [L]):

  • V, E: nodes and directed edges. Several edges between the same pair are allowed, so the graph is a multigraph (Taskography’s choice).
  • V_g, grounding: maps each node to the subset of metric map points it occupies. Point clouds are the common denominator; meshes and Gaussian splats reduce to them.
  • V_f, E_f, features: geometric attributes (centroid, box, shape) and semantic ones (class, material, colour, CLIP-style embeddings, captions). Edge attributes are spatial (“next to”), structural (“part of”), semantic (“smaller than”) or functional (a switch “controls” a light). A graph that focuses on semantic relations gets called a 3D Semantic Scene Graph (3DSSG).
  • L, layers (optional): assigns nodes to levels. Hierarchical edges link adjacent layers; intra-layer edges link nodes within a layer. A flat graph omits L.
  • Time: any component can be indexed by t. A graph with any time-indexed part is a 4D scene graph.

Hierarchies come in three families:

  • Indoor: building, (floor), room, place, object. Armeni, Hydra and Clio follow this hierarchy. S-Graphs swap the object layer for geometric primitives such as wall and corridor planes.
  • Urban: district, roads and intersections, lane graph, objects (CURB-SG, OpenGraph).
  • Functional or terrain-based: sidewalk, grass and asphalt regions, or concepts such as orchards and verges (Strader et al., xFLIE). Strader et al. is the only work the survey finds that handles indoor and outdoor in one framework.

Dynamics come in three representational styles, each with a cost:

  1. A full graph per timestamp. Easy to learn from, but static content gets copied every step.
  2. A separate node per observation of an entity (Khronos). History is kept, but something must decide which nodes are the same thing.
  3. Nodes with time-varying attributes (Kimera, CURB-SG). The most common and the most compact, but one data-association error corrupts the attributes. Best for short-term motion, such as human nodes that store SMPL joints.

Functional and actionable graphs add part nodes (knobs, handles) with kinematic state, action nodes (RoboEXP), traversable free-space nodes, and “blind nodes” that hold beliefs about unobserved parts of the scene.

Construction axes

The survey’s conceptual map has five axes: processing (offline batch or online incremental), nodes (objects, places, attributes), edges (hierarchical or relational), priors (knowledge bases, multimodal LLMs, task) and consistency (spatial or temporal).

  • Batch vs incremental. Vision papers say “prediction” or “learning” and work on a fused, instance-segmented point cloud (3DSSG, Open3DSG). Robotics papers say “construction” and work from live sensor streams (Hydra, Hydra-Multi, vS-Graphs). Batch methods skip long-term consistency entirely.
  • Nodes. The dominant recipe lifts 2D instance masks into 3D with depth and pose, then fuses them across views. The closed-set version used Mask R-CNN; the open-vocabulary version chains a tagger or VLM, GroundingDINO or YOLO-World, SAM and CLIP. Rooms come from structural boundaries (S-Graphs), from slicing the ESDF below the ceiling and running 2D connected components (Kimera), or from clustering free space (Hydra). Urban region nodes come from trajectory inflection and intersection points.
  • Edges. Candidate pairs are picked by geometric heuristics such as overlap or co-visibility, then labelled by a GCN (batch) or by an LLM or VLM (open-vocabulary).
  • SLAM-integrated is the survey’s name for any pipeline that has to keep the graph spatially consistent without ground-truth pose, or temporally consistent in a changing scene. Spatial consistency is inherited from object SLAM: greedy or linear assignment on distance, overlap or embedding similarity; odometry priors; loop closures from graph descriptors (Hydra). The S-Graph family puts the whole graph into the factor graph, so walls, rooms and floors constrain the pose. Floor-level constraints help reject false loop closures across storeys.

The multi-robot treatment is one sentence. Spatial consistency becomes a shared-frame alignment problem in which each robot’s partial graph lives in its own drifting frame and needs inter-robot loop closures and cross-agent data association (Hydra-Multi). CURB-SG adds that anchoring on persistent structure (lanes, curbs) beats associating through dynamic or ambiguous entities. Compression and synchronisation for distributed teams get one line under “emerging applications”.

Evaluation and applications

  • Intrinsic evaluation. Node quality borrows detection and segmentation metrics (precision, recall, F1, mIoU, AP, Chamfer distance). Edges use triplet metrics such as recall@k, because exact graph similarity is NP-complete. Only a handful of datasets carry ground-truth scene graphs (Armeni, 3DSSG and a few more), so online RGB-D systems end up on custom data or crowd-sourced edge checks. SLAM-integrated systems also report ATE.
  • Scene understanding. CLIP matching works for “the red chair next to the window” and fails on multi-hop queries across layers. Serialising the graph as JSON for an LLM (SayPlan’s approach) handles multi-hop queries until the JSON outgrows the context window. Subgraph retrieval and GraphRAG are the current fixes.
  • Planning and navigation. Kim et al. grounded PDDL goals in the graph by hand-written rules. Taskography started scalable planning. LLM planners (SayPlan) trade away completeness and soundness; translating language into PDDL or LTL and calling a classical planner keeps them. Only one paper does joint task and motion planning on a scene graph. Most evaluation runs in Habitat, AI2-THOR or iGibson, partly because static and full-observability assumptions make real-robot trials expensive.
  • Manipulation. The graph mostly just localises the target object, so most of what it stores goes unused at execution time. The survey’s example: a bottle stored as “full” should change grasp force even though it looks empty.
  • Dataset grouping. Their table groups evaluation by base dataset: 3RScan, ARKitScenes, Gibson, KITTI, MP3D, Replica, ScanNet, simulators, custom. KITTI and the CARLA simulator (CURB-SG) are the only outdoor sources; terrain and agricultural graphs sit under custom data. The custom row is the longest, and it holds almost every SLAM-integrated and multi-robot system (Hydra, Clio, Hydra-Multi, S-Graphs, vS-Graphs), so those systems are rarely compared on shared data.

Open challenges

These are the survey’s open-challenge paragraphs, in their order:

  • Abstract layers. Layer definitions are hand-coded. Open areas without walls have no natural regions, and only ASHiTA starts to infer layers from the task.
  • Dynamics at scale. Graphs grow with the data and have only been shown over short durations. Months or years of operation need compression, summarisation or marginalisation. Most systems keep a single world estimate and no belief over past, present and future states. Prediction of how entities and relations will evolve is understudied.
  • Function, physics and causality. Full functional graphs are too large for small platforms. They suggest parametric nodes that expand detail on demand, mass and material attributes, and causally consistent updates when an action changes what else can be reached.
  • Node construction. Cascaded perception means early errors propagate into duplicate, fragmented or missing nodes. Objects are assumed to be discrete and well bounded. Not all semantics can be extracted up front, so they argue for hybrid graphs that keep a compact implicit store and make attributes explicit on demand.
  • Edges. Co-visibility heuristics miss long-range and functional relations, upstream errors flow into edges, and the number of candidate pairs explodes. Relations should be generated on demand.
  • Consistency. Consistency is treated as purely geometric. Nobody checks whether a node’s embedding, attributes and relations still agree across visits, even though that could catch changes geometry misses. Change detection is sensitive to tuning. There is no principled rule for when to commit to a hard data association, or for when to forget an entity (for example, replacing a frequently moved object with a distribution over its room).
  • Reasoning. They want benchmarks for long-horizon spatio-temporal and multi-hop reasoning, faithfulness to the graph, LLMs distilled into small scene-specific agents, and query-adaptive graphs.
  • Planning. Scenes are assumed static and fully observed, and tasks are mostly mobile pick-and-place. They call for tighter coupling, with the graph acting as the planner’s memory.
  • Manipulation. VLAs see only the current frame and the instruction. A graph could give them memory, check preconditions and effects, and bind “the bottle on the shelf” to a node.
  • Overall (conclusion). Evaluation is the most systemic gap: graph quality and task success are measured on different datasets, so the effect of perception errors on behaviour is rarely quantified. Learning pipelines lack spatial consistency. Indoor and outdoor use incompatible decompositions. Most methods assume quasi-static scenes.

Findings

  • The field went from offline, static, closed-set graphs (2019 to 2020) to online SLAM-integrated, open-vocabulary, LLM-facing graphs (2023 to 2026). The timeline’s colour coding follows that arc: datasets, hierarchy, semantic relationships, SLAM, planning, open-vocabulary, functionality.
  • The authors split the field into two camps: vision (batch, learned relations, 3DSSG-style metrics) and robotics (incremental, pose drift, ATE). Methods from one camp are rarely tested on the other’s benchmarks.
  • Hughes et al. show that indoor scene graphs have low treewidth under some assumptions, which is the theoretical case for hierarchy that the survey leans on.

Limits

  • Outdoor coverage is thin: the only non-indoor rows are urban driving and terrain or agricultural hierarchies, and LiDAR appears only as an input stream.
  • Multi-robot construction gets a sentence and a citation to Hydra-Multi and CURB-SG. Communication limits, trust between robots, heterogeneous sensors, and faulty or adversarial inputs are not discussed anywhere in the text. When the survey talks about robust deployment, it means coping with perception noise, drift and scene change.
  • The author list overlaps heavily with the authors of the systems reviewed (Hydra, Khronos, 3DSSG, Open3DSG, FunGraph). The critique is candid about evaluation gaps, but the framing follows their lines of work.
  • It is a survey, so it reports no new numbers. Its claims of what is “underexplored” are bounded by the 150-odd papers it indexes.

Relevance

Background for 3d-scene-graphs and for the scene-graph half of multi-robot multimodal perception. The parts it leaves blank are the interesting ones for field teams: what fusion looks like when robots see through different sensors, over bad links, and cannot all be trusted (see collaborative SLAM and robust robotics). Its forgetting and belief-over-time challenges map directly onto long-term autonomy and dynamic-scenes.