Progress Probes monitor a VLA from its own activations instead of calling a second, larger VLM. Linear probes are trained on the policy’s internal activations from successful demonstrations only, to predict how far through the task the robot is. At runtime the monitor flags an execution whose predicted progress falls behind what is expected. No failure data is needed, and a linear probe on features the policy computes anyway costs almost nothing. Experiments use the open-source π0.5, the successor to π0.
Findings
- AUROC for failure detection of 0.788 on 6 VLABench simulation tasks and 0.78 on 6 real-world manipulation tasks, matching or beating SAFE and Robometer.
- Competitive with SAFE, which is trained on failure data, and stays competitive on held-out tasks.
- Detection time on simulated failures, as true-positive rate at 5% false positives: 0.00, 0.19, 0.54 and 0.64 at 25, 50, 75 and 100% of the episode. SAFE-LSTM catches more early (0.11 and 0.37 at 25 and 50%), so the probe is cheap but not the earliest.
Limits
- Progress is learned from demonstrations of each task, so a new task needs demonstrations before the monitor knows what “on track” looks like.
- One policy (π0.5). Whether the same linear progress signal exists in other VLAs is not shown here.
- A policy that makes steady progress towards the wrong goal would look healthy to a progress monitor.
Relevance
Another point for reading a policy’s internals rather than asking an outside judge, next to SAFE, Hide-and-Seek and Backtrack. It does without failure labels, like FAIL-Detect. It belongs in comparison-failure-detectors. For runtime-monitoring, the appeal is cost: a monitor that adds almost nothing could run on every robot of a team, where a VLM judge per robot could not.
Links
- arXiv
- Subfield: Safety and verification, Robot learning