A Survey of Embodied AI: From Simulators to Research Tasks

3 min read

The survey compares nine embodied AI simulators (DeepMind Lab, AI2-THOR, CHALET, VirtualHome, VRKitchen, Habitat-Sim, iGibson, SAPIEN and ThreeDWorld) on seven features: environment, physics, object type, object property, controller, action and multi-agent support. It groups these into realism, scalability and interactivity, then reviews three tasks it stacks as a pyramid: visual exploration, visual navigation and embodied question answering. The simulators were taken from the CVPR Embodied AI workshop challenges, and Sim2Real and physical robots are left out on purpose.

Findings

  • Only Habitat-Sim and iGibson build scenes from real-world scans. The other seven use game assets, and ThreeDWorld is the only one with advanced physics (cloth, fluid, soft bodies).
  • Multi-agent support is thin: AI2-THOR, iGibson and ThreeDWorld have it, and the authors note that multi-agent RL was still mostly done in OpenAI Gym.
  • Point navigation with GPS and compass is close to solved in simulation: DD-PPO trained for 2.5 billion steps gets within 3 to 5% of a shortest-path oracle in unseen scenes. The Habitat Challenge 2020 then took the GPS and compass away.
  • Navigation is scored mainly by success rate and SPL. Embodied QA adds navigation error, stopping rates, room-entry rates, IoU, mean rank and answer accuracy.
  • Dataset sizes quoted: EQA has 5,000 questions over 750 environments, 45 objects and 7 room types. IQUAD V1 has 75,000 multiple-choice questions. R2R has 21,567 instructions averaging 29 words.
  • Challenges the authors name: few simulators combine scanned scenes with advanced physics, large 3D scan collection is expensive, symbolic “point and select” actions trade off against fine-grained manipulation, nobody agrees which memory architecture suits long trajectories, and multi-agent setups get little attention.

Limits

  • Every simulator surveyed models rooms or apartments. There are no outdoor, marine or aerial settings.
  • Sim2Real and hardware are excluded by design, so sensor faults, drift and broken actuators never come up.
  • The comparison is a feature checklist. No common task is run across the nine simulators, and rendering speeds come from each simulator’s own hardware.
  • It is a 2016 to 2020 snapshot that predates VLAs and foundation-model agents.

Relevance

A map of what the field measured around 2020: reaching a point or an object inside a house, often with perfect pose. It leaves open what an embodied AI benchmark would look like for several robots across land, sea and air, which feeds multi-robot multimodal perception, and how agent memory should behave over weeks, which feeds long-term autonomy.