Do Embodied VLMs Actually Use Their Scene-Graph Memory?

2 min read

A diagnostic study of whether a fine-tuned vision-language model relies on the scene graph it is given as memory. Accuracy alone cannot tell whether the graph holds the facts, whether they reach the model, or whether the model uses them. The authors add a graph-swap test for the last question: give each episode another episode’s graph and count how many answers change. The setting is embodied question answering over routes through two to seven rooms in simulated houses, with one backbone (MomaGraph-R1). The graph is delivered either as text or through a graph encoder.

Findings

  • As text, the memory is used heavily: it adds 30 points on route questions.
  • Through the standard graph encoder, it is not used. The encoder keeps the information, but pooled input and late fusion throw it away.
  • Giving each node its own input token recovers part of it: 8.3 points on route questions, and the swap test confirms the gain comes from the graph.
  • When the same facts are also in the text, the model ignores the graph entirely, a shortcut that accuracy hides.
  • Fine-tuned priors already answer most set questions, and there the limit is the recall of the scene graphs themselves. Episodes come from 53 houses.

Limits

  • One simulator, one task family and one backbone. One seed for some models and one derangement for the swap test (authors).
  • Pooled input and late cross-attention were tested together, not separately (authors).
  • Room segmentation uses the simulator’s record of door crossings; a deployed robot would have to estimate it (authors).

Relevance

Most planners in language-grounded-planning serialise the scene graph into text, as SayPlan does, and this paper suggests that is the format the model reads. The swap test is cheap and should be run on any graph-to-VLM pipeline, including SceneGraphVLM-style generation. For teams it raises a worry: if a model ignores the graph whenever the same facts appear in text, a graph merged from several robots may matter less than the prompt.