PaLM-E: An Embodied Multimodal Language Model

1 min read

PaLM-E feeds continuous sensor inputs (images and state estimates) into a pretrained large language model as “multi-modal sentences” that interleave encoded observations with words, and trains the encoders end to end with the LLM. One model handles sequential manipulation planning, visual question answering and captioning.

Findings

  • Joint training on internet-scale language, vision and vision-language data transfers positively to the embodied tasks.
  • The largest model has 562B parameters and reported the best published result on OK-VQA at the time.
  • Language ability is kept as the model grows.

Limits

  • Abstract-level reading only; no task success rates in the abstract.

Relevance

Treating a state estimate as just another token is convenient and also means a wrong or spoofed estimate is read with the same trust as a word. Open question for robust robotics: can such a model tell when one of its input tokens disagrees with the rest? Cited in GoGet; see language-grounded-planning.