Describe Anything Anywhere At Any Moment

2 min read

DAAAM builds a spatio-temporal memory of a large scene in real time, where each object carries a rich free-text description instead of a class label or a CLIP vector. Per the README, SAM segments, BoT-SORT tracks, and a localised captioning model (the Describe Anything Model) describes each tracked region. An optimisation-based frontend turns those captions into descriptions, and batching speeds online inference by an order of magnitude. The results feed Hydra to build a hierarchical “4D” scene graph, which a tool-calling agent queries to answer questions. I have read only the abstract and the README.

Findings

  • Against the strongest baselines (from the abstract): question accuracy up 53.6%, position error down 21.9% and temporal error down 21.6% on NaVQA spatio-temporal question answering, and SG3D sequential task-grounding accuracy up 27.8%.
  • The authors curate OC-NaVQA, an extended benchmark for large-scale, long-duration evaluation.
  • Code and data are released, with a separate ROS 2 interface (DAAAM-ROS).

Limits

  • Abstract-level reading only, so the failure cases, compute budget and handling of moved objects are unknown to me.
  • It relies on several heavy foundation models in the loop (SAM, a captioner, an LLM agent), which sets the hardware floor.
  • Nothing in the abstract on multiple robots or degraded sensing.

Relevance

DAAAM continues the MIT SPARK line from Khronos towards memory you can ask questions of; see long-term autonomy and open-vocabulary mapping. Free-text node descriptions also raise a question for teams: two robots’ captions of the same object will rarely match word for word.