SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models

2 min read

SceneGraphVLM has a small vision-language model write an image or video scene graph directly as text, in place of the usual detector-then-relation pipeline. Graphs are serialised in TOON, a compact text format that shortens the output. Training is supervised fine-tuning for schema following, then GRPO with graph rewards that penalise hallucinated nodes and edges. For video, the previous frame’s graph can be fed back in the prompt. It is evaluated on PSG, PVSG and Action Genome, which are benchmarks of 2D panoptic and video scene graphs; there is no 3D map.

Findings

  • The abstract reports about one second of latency for a complete graph, and better precision-oriented metrics with “reasonable” recall. It gives no metric values. I have read only the abstract and README.
  • The README lists Qwen models served through vLLM, with checkpoints and a Docker image on Zenodo.

Limits

  • 2D only; no geometry, no robot.
  • The design favours precision over recall, so missing relations are expected.
  • Feeding back the previous frame’s graph could carry an error forward from frame to frame; the abstract does not say whether this happens.

Relevance

Hallucination-aware rewards are one way to make a VLM’s graph output more trustworthy, which would help if robots later share and fuse those edges. See dynamic-scenes and foundation-models-for-robotics.