OS-Atlas is a foundation action model for agents that operate graphical interfaces, which have to ground an instruction to an on-screen element before acting on it. The authors release what they call the largest open-source cross-platform GUI grounding corpus to date.
Findings
- Over 13 million GUI elements from Windows, Linux, macOS, Android and the web.
- Evaluated on six benchmarks across mobile, desktop and web.
Limits
- Abstract-level reading only; no scores in the abstract summary I read.
Relevance
GUI grounding is object grounding on a screen, and the screen mode of an assistive agent such as GoGet needs it. A grounding error on a screen costs a wrong click; on a robot it moves hardware, so checking grounding before acting is more important there. See open-vocabulary-mapping.
Links
- arXiv
- Subfield: Perception