TASKOGRAPHY: Evaluating robot task planning over large 3D scene graphs

4 min read

Taskography is a benchmark of 20 symbolic task planning domains (3734 tasks in total) written in PDDL over the 3D scene graphs of Gibson buildings, which average 2 to 3 floors and 7 rooms per floor. It deliberately takes perception out of the loop: the graphs are the Stanford 3D Scene Graph annotations, extended with affordances from a hand-built object-action knowledge base, room connectivity derived from doors, and other “plannable attributes”. On top of the benchmark the authors propose two ways to cut the graph down before planning. SCRUB starts from the goal objects and the robot and keeps adding any object that shares a binary predicate with the set until nothing new appears. SEEK takes a learned object scorer (PLOI’s graph network) and, for every object it keeps, also keeps all its ancestors in the hierarchy so the retained objects stay reachable.

Findings

  • Optimal planners only cope with toy cases. FD with lmcut, SatPlan and Delfi fail on every Rearrangement(k) and Courier(n, k) task with k > 2 inside a 10 minute budget; MCTS and regression planning solve nothing in 30 seconds.
  • Difficulty tracks the domain and barely moves with building size. On Rearrangement(10), BFWS takes 0.57 s on the Tiny split and 0.41 s on Medium, with failure rates of 0.18 and 0.23. The easiest domains have about 1000 state variables and a branching factor of 5; the hardest about 4000 and 60.
  • Without pruning, the learned planner PLOI is the best performer (0.17 s, no failures on Rearrangement(10) Medium), but it needs many replanning rounds because thresholded object scores often drop an object’s container or room.
  • SCRUB removes more than two thirds of operators and state variables on grounded problems and about a third on lifted ones. It brings FF, FD and BFWS down to a few milliseconds on Rearrangement(10) and caps Courier(10, 10) at 5 s. Optimal FD on Lifted Rearrangement(5) did not finish in 24 hours on the full graph; with SCRUB it solved 72.86% of tasks within 30 s, at 2.26 s each on average.
  • SEEK cuts replanning by about an order of magnitude: PLOI goes from 141 to 14 replans on Rearrangement(2) Medium and from 41 to 7 on Courier(10, 3) Medium, with the same success rate. On Lifted Courier(5, 5) it drops from 169 to 53 replans and success rises from 0.73 to 0.77.
  • Rule of thumb from the authors: SCRUB on grounded tasks (it is provably minimal there), SEEK on lifted tasks (SCRUB keeps every instance of a goal class, SEEK picks opportunistically).

Limits

  • The authors say plainly that the first release covers offline planning only, fully observable, deterministic and closed world, with no motion planning. Real robots, they note, face partial observability, sensor noise and resource limits.
  • The graphs are clean annotations. Room connectivity was missing from the source data, so it is estimated as a minimum spanning tree over room centroids, with a single staircase between floors.
  • SCRUB is guaranteed sufficient only if the predicates are correct. A missed inReceptacle edge or a phantom object changes the “sufficient” set without warning. SEEK carries no satisficing guarantee at all.
  • Nodes carry class labels and hand-coded affordances. There is no open-vocabulary content, so goals must be phrased in the benchmark’s vocabulary.

Relevance

In Taskography, pruning the graph makes more difference than the choice of planner on building-scale 3d-scene-graphs. Its guarantees assume a perfect graph, so the open question is what a sufficient subgraph means when the graph comes from noisy perception or from several robots that disagree, a question for both robust robotics and multi-robot perception. Compare the LLM-driven pruning in SayPlan, the formal planner in Dai et al. and the task-driven compression in Clio; see also language-grounded-planning and embodied-ai-benchmarks.