SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning

4 min read

SayPlan plans long-horizon mobile manipulation tasks with GPT-4 over a pre-built 3D scene graph serialised as JSON (floors, rooms, assets, objects, with pose nodes linking rooms). The LLM starts from a collapsed graph that shows only the top levels and runs a “semantic search”: it calls expand on a node to reveal its children and contract to hide them again when nothing useful is there, with a list of already expanded nodes passed back as memory so the history never has to be replayed. Once the subgraph holds what the task needs, the LLM writes a high-level plan, Dijkstra fills in the pose-to-pose navigation, and a scene graph simulator forward-simulates the plan and returns text feedback such as “cannot pick up banana” for up to 5 rounds of replanning.

Findings

  • Graph sizes: the office has 37 rooms and 151 assets and objects (407 entities counting 218 edges), 6731 tokens in full and 878 collapsed. The Home scene (Klickitat from the Stanford 3D Scene Graph dataset) has 28 rooms over 3 floors and 112 assets and objects, 6598 tokens in full and 1817 collapsed. Those counts give reductions of 86.9% and 72.5%; the running text quotes 82.1% and 60.4%, so one of the two is stale.
  • The static prompt (role, API description, examples) is about 3900 tokens, which on GPT-4’s 8192-token window is a large fixed cost.
  • Semantic search with GPT-4 finds the right subgraph in 86.7% of simple and 73.3% of complex queries in both scenes. GPT-3.5 manages 6.6% and 0.0% in the office and 0.0% in the home, mostly by hallucinating nodes or expanding the same node again and again.
  • Causal planning over 30 tasks: all methods reach 93.3% correctness on simple tasks, but executability is 100% for SayPlan against 80.0% for an open-loop LLM planner and 13.3% for LLM plus path planner. On long-horizon tasks SayPlan gets 73.3% correct and 86.6% executable, against 66.7% and 13.3% for the open-loop baseline. The 6.67% of tasks it still fails are hallucinated nodes that 5 replanning rounds did not fix.
  • Contracting irrelevant nodes keeps the token count roughly flat during search. In an ablation that grows the office from 30 to 300 room nodes, every failure came from overflowing the 8192-token limit, which the authors read as the LLM still attending to the relevant nodes as the graph grows; larger graphs mainly cost more search steps.
  • Search falls back to breadth-first behaviour when the query has no semantic cue (“find me object K31X”), so it depends on meaningful node names.
  • Plans ran on a Franka Panda on an Omron LD-60 base, localised by LiDAR, with RGB-D grasp detection and a behaviour tree for execution.

Limits

  • The authors list the main ones: a pre-built graph, objects assumed static after mapping, text-only node descriptions, and an LLM that fails at negation (“a bathroom with no toilet”), counting and distance reasoning.
  • The simulator’s feedback messages are written by hand for each failure type. The authors expect this to stop scaling as predicates and affordances multiply.
  • No wall-clock planning time or number of LLM calls per task is reported, a gap for anything with a slow or intermittent link to a hosted model.
  • The evaluation is 90 instructions in two scenes, and correctness is judged against what a human would do.
  • The plan is only as good as the graph: a stale or wrong node passes the simulator check and fails on the robot.

Relevance

SayPlan lets an LLM page through a 3D scene graph instead of reading it whole, and the expand/contract step is a cheap form of the task-driven pruning that Taskography does symbolically and Clio does during mapping. Its static-world assumption is where long-term autonomy starts, and the dependence on a hosted model with an 8k context raises a question for robots on poor links underground or in the air: which part of this loop can run on board? See language-grounded-planning, foundation-models-for-robotics, and the open-vocabulary graphs it would need in ConceptGraphs and HOV-SG.