Language-Grounded Hierarchical Planning and Execution with Multi-Robot 3D Scene Graphs

2 min read

This system joins several MIT pieces into one outdoor multi-robot pipeline. In a mapping stage, a Husky and a Spot each build a Hydra scene graph and a ROMAN open-set object map; ROMAN supplies intra- and inter-robot loop closures, and a modified Hydra-Multi fuses the graphs into one. In an execution stage, each robot starts with no pose, builds a fresh object map and relocalises against the fused one with ROMAN. An operator’s instruction is turned into one PDDL goal per robot by an LLM prompted with a text description of the scene graph and the robots’ capabilities, and a PDDLStream planner solves it over the places layer.

Findings

  • Field site: Camp Buckner, West Point, a large unstructured outdoor area. Laptops on both robots run mapping; a base station runs the LLM and the planner.
  • Relocalisation every 10 m along a shared stretch of about 500 m, with Spot’s side camera against the Husky’s forward camera: 64% success for the Husky and 65% for Spot (within 5 m and 10 degrees).
  • Grounding, planning and execution over 8 single-robot and 12 multi-robot trials at three levels of language difficulty; multi-robot execution ranges from 66% to 100% across levels.
  • ROMAN’s relocalisation database is 9.8 MB outdoors against 15.0 GB for the image-retrieval baseline SALAD, with similar pose error.

Limits

  • In the reported grounding failure, every object was picked correctly, but “check out two boxes” became “move to” instead of “inspect”.
  • The planner solves single-agent PDDL problems; splitting work between robots is left to the LLM translation step.
  • Small trial counts. Seedling: I read the abstract, the system description and the results tables only.

Relevance

Here a team of robots shares one language-queryable scene graph outdoors, and the system depends on ROMAN for the step where two robots agree on where they are. The open questions are what happens when the fused graph is wrong, and when the base station link drops; see multi-robot multimodal perception, language-grounded planning and robust robotics. Its outdoor graphs build on language-enabled spatial ontologies.