LT-Mem: Volatility-Aware Spatio-Temporal Memory for Lifelong Scene Understanding

2 min read

Long-running maps either overwrite the past to stay current or store semantic snapshots with no consistent object identity across visits. The authors call the resulting loss of object history “temporal amnesia”: the robot cannot answer “where has the green chair been across all sessions?”. LT-Mem has three layers. A multi-session SLAM backbone aligns per-object observations across visits; a session whose global alignment looks wrong is held and logged rather than allowed to corrupt memory. A reasoning layer keeps identities by deterministic evidence scoring and, per object, chooses to overwrite, hold, or keep several hypotheses depending on how volatile that object has been. A Tri-Memory stores the current state (Live), change events (Delta) and statistics such as how often things move (Meta). The authors also release LT-VQA, multi-session recordings with persistent identity labels and temporal question-answer pairs. Presented at IROS 2026, where it was listed as a candidate in the Best Paper / Best Student Paper award session.

Findings

On LT-VQA (two indoor scenes, Lab-S with 10 tracked objects and the larger Lab-L, combined):

  • Event F1 0.885 with Qwen2.5 and 0.910 with Gemini 2.5, against 0.790 for the best baseline (VLM-Batch) and 0.630 for geometry alone.
  • Event questions 0.800 and 0.820 against 0.680; frequency questions 0.567 and 0.600 against 0.333.
  • 352K and 438K tokens over the evaluation set, against 7,114K for the VLM baseline and 45,859K for STAR.
  • Ablations attribute the gains to the memory structure, not the size of the language model.
  • An outdoor parking-lot scene shows the Meta memory answering questions such as when it is hardest to find a space.

Limits

  • Small scenes: one room with 10 tracked objects and one larger indoor space, plus a parking lot used only for statistics.
  • One robot. Identity matching across sessions depends on the SLAM alignment being right; held sessions are skipped, not repaired.

Relevance

LT-Mem keeps a record of object events instead of a single current map, close to Khronos and OASIS-Map on change across sessions, but organised around questions a person would ask. Its per-object volatility is a form of the forgetting policy discussed in dynamic-scenes and in Learning to Forget. For long-term-autonomy, the token counts matter as much as accuracy: a structured memory is far cheaper to query than replaying frames.