EmboCoach-Bench: Benchmarking AI Agents on Developing Embodied Robots

3 min read

EmboCoach-Bench tests whether LLM agents can do the engineering around robot learning: write dense rewards, fix training code, and tune diffusion policies, ACT and VLAs. Each of 32 tasks across ManiSkill, RoboTwin, Robomimic and MetaWorld gives the agent a requirements document (goal, time budget, the files it may touch, hints), a tool protocol (terminal, file editor, task tracker) and a full codebase in Docker. Training jobs go to a Kubernetes GPU cluster. The agent loop is ML-Master’s tree search with OpenHands for repository work, run as Draft, Debug, Improve, and a debug_test tool allows 10-episode dry runs before full training. The score is the success rate of the final trained policy under the simulator’s own checks, within wall-clock limits.

Findings

  • 21 “improving” tasks, which start from a working human baseline: average success 0.47 for the human baseline, 0.59 for single-shot LLM edits and 0.80 with the agentic loop.
  • 11 “from scratch” tasks: 0.85 human, 0.88 single-shot, 0.99 agentic.
  • Averaged over all 32 tasks, single-shot GPT-5.2 and Gemini 3.0 Pro reach 40.7% and 39.5% against a 61.8% human baseline. With the loop, Gemini 3.0 Pro reaches 81.3% and Claude Opus 4.5 79.5%; DeepSeek-V3.2-Thinking goes from 53.2% to 79.5% and Kimi-K2-Thinking from 42.4% to 77.1%.
  • Broken baselines get fixed: peg-insertion-side goes from 0.00 to 0.94 and coffee-pull-st from 0.15 to 0.95.
  • A worked trace on put_object_cabinet: the agent adds gradient checkpointing and mixed precision, doubles the prediction horizon from 8 to 16, then finds a validation data leak and a GradScaler initialisation bug.

Limits

  • The per-task tables name a “Best LLM” for each task, so the agentic column looks like a best-of-several-models figure.
  • Seven of the 11 from-scratch MetaWorld tasks have a human baseline of 1.00 already, so they cannot separate agents.
  • Single-shot edits can collapse a working task (reach-mt10 goes from 1.00 to 0.10).
  • Everything is in simulation and on tabletop arms. Agent-tuned policies are never tried on hardware.
  • The agent edits the training pipeline while the success check is fixed. The paper does not report any audit for overfitting to the evaluation episodes or for gaming the check.
  • I did not find run counts or variance per model.

Relevance

This measures whether an agent can make a policy score higher in simulation; whether the policy is safe, recovers from faults or keeps working is a separate question that a robust robotics benchmark would have to add. Open question: what should an engineering agent be allowed to change when its only feedback is a simulated success rate? Compare AlphaEvolve, which leans on its evaluator in the same way, and llm-evaluation.