RoboColosseum: Evaluating Physical Intelligence at Scale In-The-Wild

5 min read

RoboColosseum tests robot policies on real robots at many sites instead of in one lab or one simulator. It covers four robots (the low-cost SO-101 arm, a Franka in the DROID setup, a bimanual YAM and a Unitree G1 humanoid), five vision-language-action models (GR00T N1.7, π0.5, MolmoAct2, G0.5, LingBot-VLA v2) and three frontier language models driving the robot through an API (GPT-6 Astra, Claude Opus 5.5, Grok 4.7). Up to 30 paid operators run the trials in offices, libraries, parks, homes, food courts and factories in Singapore, Malaysia and China. There are two tracks. In the Open Track an operator invents a task with whatever is at hand, two anonymous models try it one after the other on the same setup, and the operator records which did better, how far each got, and a difficulty rating from 1 to 10. Rankings come from a Rasch model that estimates each policy’s ability and each task’s difficulty together, with the human rating as a prior on difficulty, reported on an Elo-like scale. In the Fine-tune Track every model is fine-tuned on the same 100 demonstrations per task (five tasks per robot, 20 in all) and run 25 times per task, scored on success and on the fraction of task-specific progress criteria met.

Findings

  • Data: 5,699 Open Track comparisons at 124 sites over 41 days (17 September to 28 October 2026).
  • The language-model agents hold their own against the VLAs. In 901 agent-against-VLA comparisons the agent made more progress in 394, the VLA in 265, with 242 ties: 5.3 points of progress in the agents’ favour. That average hides a split: Claude Opus 5.5 and GPT-6 Astra are 12.1 and 11.0 points ahead, Grok 4.7 7.5 points behind.
  • Nobody finishes open-world tasks. No system passes 13% success in any of the six task categories. Success falls from 30.7% on tasks rated 1 to 4 to 0.4% on tasks rated 9 to 10, and mean progress on failed runs from 56.6% to 3.5%.
  • After fine-tuning, GR00T N1.7 leads on SO-101 (82.4% success) and π0.5 on Franka (84.0%) and the bimanual YAM (93.6%). On the G1 humanoid no policy completes any of the five tasks; average progress is 13.3% to 23.2%.
  • Rankings don’t carry over. Agreement between the four robots’ fine-tune rankings is weak (Kendall’s W = 0.24). Open Track rank against fine-tune rank: Spearman ρ = −0.20 on SO-101, −0.40 on Franka, 0.20 on YAM. π0.5 is fifth of five VLAs on Franka in the Open Track and first after fine-tuning.
  • Some policies learn tasks one at a time. On Franka, LingBot-VLA v2 gets 100% on opening a drawer and 0% on putting a marker in a holder; MolmoAct2 gets 0% and 88%.
  • Progress criteria show where runs fail. On closing a laptop, G0.5 makes first contact in all 25 trials but gets halfway in only one; MolmoAct2 nearly closes the lid in 24 and finishes 7.
  • Splitting the sites into two random halves gives about 85% agreement on model pairs for Franka and YAM, but 59% (ρ = 0.30) for SO-101.
  • Choosing which pair to test next by uncertainty (Chatbot Arena’s method) reaches 90% agreement with the full ranking after 400 to 500 trials, against about 1,000 with uniform sampling. 95% agreement needs about 4,755 trials.

Limits

  • Each model follows its own fine-tuning recipe, so the Fine-tune Track compares whole pipelines on the same data, not architectures (authors).
  • Five tasks per robot, and coverage differs by model and robot because not every checkpoint runs on every robot (authors).
  • The Open Track ranking is checked against itself (subsets against the full set). Nothing says how it would change with other sites or task mixes.
  • The operator who invents the task also judges it. Blinding hides which model is which, but the task distribution is whatever operators thought feasible.
  • Runs that risk damage are stopped and counted as failures; two policy–task pairs were stopped early for safety. There is no measure of how close a run came to damaging something, or of what a policy does after a disturbance.
  • One robot at a time, and no faults, wrong instructions or adversarial inputs.

Relevance

The strongest real-world evidence so far for what embodied AI benchmarks say in simulation: a ranking measured on one robot or one task set doesn’t transfer, and a single success rate hides very different failures. Its progress criteria are a step towards measuring how long a robot stays useful after something goes wrong, which failure recovery needs and nobody measures. Two things carry over to robust robotics: the frontier agents in the Open Track are LLMs judging their own actions from camera images, so the worries in evaluating language models apply, and the site-split check is a cheap test of whether a leaderboard depends on a few sites, the benchmark-level version of asking whether a result depends on a few teammates. Policies it evaluates that already have notes: π0 (π0.5 is its successor) and OpenVLA (cited as background). Hugging Face, one of the founding members, also maintains LeRobot.