Foundation models that let a robot see, reason and act from vision and language, and the network that has to carry all of it on a budget of bandwidth, latency and compute.
Main ideas
- Vision transformers, diffusion transformers, and language and vision-language models.
- Vision-language-action models, which turn what a robot sees and is told into motor commands. Much of the difficulty is in how an action is represented: as discrete tokens, as a diffusion process, or as a flow.
- Running these models in closed loop, inside a robot simulator.
- Wireless basics and radio propagation, and the choice of what runs on the robot and what goes to the cloud. That split decides whether an answer arrives in time to act on.
- Distributed SLAM and federated learning: two ways for agents to pool perception and learning without sending everything to one place.
Project
In a team of four. Most of the hands-on work put a robot simulator next to a model of the radio link, so a policy’s compute and the robot’s connectivity could be studied together. A model that is accurate offline can still miss its window once perception, a round trip to a server and the action all have to fit inside a latency budget, and the radio link changes as the robot moves.
Relevance to my research
Language as a way to query a scene, and foundation models for perception, are part of multi-robot perception, and distributed SLAM leads to collaborative SLAM. A robot that keeps acting sensibly when its link drops or lags is a robust robotics problem, and deciding what to keep on board for months at a time is one for long-term autonomy.