FailBench collects 2,197 manipulation attempts from 14 public sources (12 real, 2 simulated) and asks 13 VLM-based detectors whether each attempt succeeded. The best reaches only 0.77 mean balanced accuracy, models fine-tuned for failure detection do worse than general VLMs, and contact-heavy assembly tasks fall below 0.60. The detectors lean towards saying “success” when the evidence is ambiguous, and extra reasoning effort does not remove that bias.
Relevance
Many recovery pipelines assume a VLM can tell when a task failed. This measures how often that holds, and a bias towards “success” is the worst direction for failure recovery. Read it before building robust robotics on VLM judges.
Links
- arXiv
- Subfield: Safety and verification, Robot learning, Perception