AutoScientists runs a decentralised team of AI agents on long computational experiments. Agents organise themselves around promising hypotheses and critique each other’s proposals before any compute is spent on them.
Findings
- BioML-Bench (24 tasks): mean leaderboard percentile of 74.4%, 8.33% above the strongest earlier AI agent.
- GPT training optimisation: reaches a target validation bits-per-byte 1.9 times faster than Karpathy’s Autoresearch, with 7 accepted improvements from a starting champion against 0 for a single agent.
- ProteinGym: a method found for ACE2-Spike binding improves Spearman correlation by 12.5% over the best earlier model, and by 6.5% on average when applied unchanged to all 217 assays.
Limits
- Abstract and repository README only.
Relevance
A team of agents that critique each other is a software analogue of a robot team that cross-checks reports. Open question for byzantine-fault-tolerance and Multi-robot systems: what happens when one agent is wrong but persuasive, and does peer critique catch it or spread it? Compare the single-agent setups in EmboCoach-Bench and Brain2Qwerty v2.
Links
- arXiv · code
- Subfield: Robot learning