Unsupervised Discovery of Failure Taxonomies from Deployment Logs

2 min read

Robots in deployment log many failures, and nobody has time to read them all. This paper turns raw failure logs into a taxonomy of recurring failure modes without labels. A vision-language model first writes a structured explanation of each failed trajectory. Those explanations are then clustered in a semantic reasoning space, so the clusters are failure modes rather than one-off episode descriptions. The taxonomy is then put to work in two ways: a VLM runtime monitor is prompted with the known failure modes, and data collection is aimed at the conditions behind them. Gemini 2.5 Pro writes the explanations and o4-mini does the clustering, assignment and monitoring. First posted in June 2025, presented at IROS 2026.

Findings

  • Manipulation, where expert taxonomies exist: assigning failure episodes to discovered modes reaches a weighted F1 of 85.53%, against 32.41% for an embedding-similarity baseline.
  • Indoor navigation: collecting 40K extra samples in high-risk regions found by the taxonomy (featureless walls, thin objects) cuts the failure rate of sampled trajectories from 46% to 18%. The same number of uniformly collected samples only reaches 34%.
  • Also tested on real car-crash videos, where a taxonomy-aware monitor is compared with VideoMAE and other supervised classifiers.

Limits

  • In driving and navigation there are no expert labels, so the quality of the taxonomy there is judged qualitatively (authors).
  • The pipeline rests on hosted models. A wrong explanation at the first step carries into the clusters.

Relevance

This is the step before runtime-monitoring: deciding what a monitor should look for. For long-term-autonomy, a taxonomy that grows from logs is how a robot that runs for months could learn its own failure modes. Monitoring with taxonomy context is one more VLM judge, so the cautions in FailBench and coverage shift apply. See failure-recovery.