ASHiTA: Automatic Scene-grounded HIerarchical Task Analysis

2 min read

ASHiTA takes a high-level instruction such as “prepare for dinner”, which may not name a single object, and breaks it into subtasks that are each tied to objects in a 3D scene graph. It alternates two steps: an LLM (GPT-4o-mini) proposes and scores a task hierarchy top-down, and a task-driven scene graph is built bottom-up so that its layers (primitives, items, subtasks, tasks) follow that hierarchy. The graph construction generalises the Information Bottleneck used by Clio into a hierarchical version (H-IB). Primitives come from EfficientViT segments with MobileCLIP embeddings, with RTAB-Map poses.

Findings

  • On the SG3D benchmark (890 high-level tasks over 35 HM3DSem scenes) with ground-truth segmentation, ASHiTA reaches subtask grounding accuracy comparable to the best fine-tuned 3D grounding baselines.
  • On 8 incrementally built HM3DSem sequences (205 tasks), it beats Hydra plus GPT and HOV-SG plus GPT at both task breakdown and grounding.
  • Shown on a Boston Dynamics Spot in a snack bar, a seminar room and an office hardware area.

Limits

  • The authors admit the graph has no relations, so “inside of” and “on top of” are lost. That is where it falls short of GPT given ground-truth labels.
  • Granularity is capped by the word bank: different kinds of cup collapse into “cup”, and many identical chairs got reduced to three.
  • No guarantee that the generated subtasks are enough to complete the task. They suggest classical planning to check them.
  • I have only read the abstract, intro and limitations.

Relevance

ASHiTA pushes Clio’s idea one level up: the task decides the graph, and the graph also reshapes the task. It combines open-vocabulary mapping with LLM planning of the SayPlan kind; see language-grounded planning and 3d-scene-graphs. An open question for robot teams is what happens when two robots build task-driven graphs for different tasks over the same space.