Multimodal Generative AI

2 min read

Generative models for text, images and audio, from the transformer up to systems that take several modalities at once. About half maths and half implementation.

Main ideas

  • Tokenisation, attention, multiple heads, and what pretraining optimises. Having written a transformer once makes the papers that build on it much easier to read.
  • Contrastive training, which puts images or audio and text in one shared space so you can retrieve one from the other. Open-vocabulary robot maps are built on this (see open-vocabulary mapping).
  • Diffusion: generating by learning to undo noise.
  • Grounding and retrieval, which tie a model’s answer to evidence it can point at.

Project

In a team of four, GoGet: an assistant you talk to while it looks through a camera, running entirely on one local GPU. A question goes through five stages:

  1. speech recognition (Parakeet),
  2. a vision-language model (Gemma) that turns the question into a short noun phrase, such as “red cup”,
  3. text-prompted segmentation (SAM), which finds and counts the matching objects,
  4. the vision-language model again, which answers using the segmenter’s count,
  5. speech synthesis (Kokoro).

Counting was the hard part. Asked “how many cups?”, a vision-language model often gives a wrong number with full confidence. Feeding it the segmenter’s count doesn’t fix that either: it repeats whatever the segmenter says, right or wrong. So GoGet gives a confident count only when the two agree, and says “about N” when they don’t. That cut confidently wrong answers from 63% to 25%. We also replaced keyword matching with the model itself for deciding whether a request is about the camera, a chat, or a desktop task, and a grounded answer takes under five seconds. On a robot, “about N” can turn into an action: move closer and look again.

I built the agent and most of the code. The paper is GoGet, an oral at AANN 2026.

Relevance to my research

Answering only when two models agree is selective prediction, and the same idea runs through disagreement between models and robust robotics: the worst answer is a wrong one given with confidence. Segmentation by text prompt, and language as a query, belong with multi-robot perception, since robots that see different things need a shared way to name them.