A fixed VLM judges whether a pick-and-place succeeded while the policy it watches improves. The policy is trained only on simulator-verified successes, so it gets better. Its remaining failures also move to where the judge cannot see them. In a tray-placement task the typical failure is a “hover”: the cube is held above the tray, not placed. As the policy improves, the hovers that remain sit below the tray rim, which hides the gap under the grasped cube, and the judge accepts more of them. The authors call this coverage shift: learning changes which failures reach a fixed judge.
Findings
- 20 seeds. Success rises from 66.8% to 97.0%, and accepted hover failures fall from 5.0% to 0.9% of executions.
- But the judge accepts 50.0% of the hovers that remain, against 31.1% under the untrained policy.
- A boundary probe: the judge accepts 148 of 160 below-rim hovers and 0 of 160 above-rim hovers.
- A wider diagnosis covers nine judges (open models such as Qwen3-VL-8B, RoboReward-8B, a DINOv2 probe, and five closed models) on 1,392 stimuli. Errors depend on failure type and camera view.
Limits
- One main task and one failure type in the central experiment. The authors note that a failure name such as “hover” does not guarantee the failures look alike (authors).
- Most VLM judges see one final frame (RoboReward gets a 16-frame clip). Monitors that watch the whole episode may be less affected; the paper does not test that.
Relevance
A sharper version of the warning in FailBench and No Free Checker: a judge’s error rate is not a fixed property, because the policy decides which failures it gets shown. The suggested report (task success, failure incidence, absolute accepted failures and conditional acceptance, side by side) is worth copying in any evaluation of monitors and LLM judges. For teams, the same effect would apply to a robot that judges a teammate’s work.
Links
- arXiv
- Subfield: Safety and verification, Robot learning