RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

1 min read

RT-2 turns a vision-language model into a robot policy by writing robot actions as text tokens and co-fine-tuning on robot trajectories together with internet-scale vision-language tasks such as visual question answering. The authors call the result a vision-language-action (VLA) model. One model both maps observations to actions and keeps what it learned from web data.

Findings

  • About 6,000 evaluation trials.
  • Reported abilities: better generalisation to new objects, following commands absent from the robot data (placing an object on a particular number or icon), and simple reasoning (pick the smallest or largest object, or the one closest to another).
  • With chain-of-thought prompting it does multi-stage semantic reasoning, such as choosing a rock as an improvised hammer or an energy drink for someone tired.

Limits

  • Abstract-level reading only. The abstract gives no success rates.

Relevance

VLAs bring web knowledge into control, and with it the web’s confident mistakes. When a VLA misreads a scene there is no separate perception output to check, unless a second model is asked, the approach taken in GoGet. See foundation-models-for-robotics and language-grounded-planning.