How the field measures whether a robot policy works. Almost always: a success rate on one task, in one simulated room or tabletop, with a check written by the people who built the simulator.
Benchmarks
- The embodied AI survey reviews nine simulators. All of them model apartments. Point navigation with GPS and a compass already gets within 3 to 5% of a shortest-path oracle, so the easy version is nearly solved.
- RoboTwin 2.0 moves to bimanual tabletop work and generates its data with strong randomisation.
- LIBERO runs a sequence of 130 tasks to test whether a policy keeps what it learned.
- EmboCoach-Bench measures the engineering loop around training: can an AI agent write and tune the code that trains a robot?
Gaps
Across all of these, no benchmark has more than one robot acting. None injects a sensor fault or a teammate that lies. None runs outdoors, on the water or in the air. “Long horizon” means a task with many steps, or a sequence of new tasks, and never one robot running for weeks while its world changes. The survey pointed out the missing multi-agent work in 2021, and the newer benchmarks still don’t have it.
The drops are large even without faults. On RoboTwin 2.0, randomising lighting, texture, clutter and table height takes π0 from 46.4% to 16.3% success, and Diffusion Policy from 28.0% to 0.6%.
Data generation
RoboTwin 2.0 varies five things: distractor objects (deliberately excluding look-alikes), 11,000 generated textures, light colour, type, intensity and position, table height, and LLM-written rewordings of the instruction, plus 1 cm of camera jitter for the real-robot runs. Mass, friction and latency are not varied, though its related work lists them as standard. Every axis is drawn from what the designers think is plausible. Nobody randomises failures (a dropped frame, a frozen encoder) or adversarial inputs. Domain randomisation is built for the average case, and security is about the worst case.
The generators share one more assumption with AlphaEvolve and EmboCoach: the simulator’s success check is the truth. AlphaEvolve names its reliance on an automatic evaluator as its main limitation. An agent that can rewrite training code against a fixed check will eventually find a way to pass the check.
Policies
Most of the robot learning on my disk is built on a small set of open policies: ACT, Diffusion Policy, OpenVLA, π0 and SmolVLA, trained and run through LeRobot. RT-2 and PaLM-E are where vision-language-action models started.
Questions
- What would a benchmark look like with three robots, one of them broken, and a task that lasts a week?
- Can a data generator randomise faults and attacks as easily as it randomises textures?
- How do you score a policy that is right to refuse?
Related: foundation models for robotics, failure recovery, robust robotics, long-term autonomy.