AndroidWorld
AndroidWorld is a public benchmark from researchers at Google DeepMind and Google that tests AI agents on 116 tasks across 20 real Android apps, checking each result against the device's system state. AGI-0, a model trained by AGI, Inc., scored 100% on AndroidWorld (116/116 tasks), listed on the official leaderboard.
Last updated
Related terms
Mobile use
Mobile use is an AI agent's ability to operate the apps on a phone or tablet and finish tasks for a person.
Phone use
Phone use is an AI agent using the apps on a phone to finish tasks, often on the person's own phone and accounts.
OSWorld
OSWorld is a public benchmark that tests computer-use agents on 369 tasks on real computers, across desktop apps, files and websites.
Built by | Rawles et al., Google DeepMind and Google (ICLR 2025) |
|---|---|
Tasks | 116, across 20 real Android apps |
Scored by | A success check on the device's system state |
People | 80.0% of tasks completed |
AGI-0 | 100% (116/116 tasks), official leaderboard |
What does AndroidWorld test?
It tests whether an agent can finish everyday phone tasks in real apps, such as logging an expense or editing a note. Tasks are generated with new parameters on each run, so an agent cannot pass by memorizing one fixed version.
How is AndroidWorld scored?
Each task comes with its own setup and success check, and the check reads the device's system state when the agent finishes. The score is the share of tasks completed. In the original paper, people completed 80.0% of the tasks.
What did AGI, Inc. score?
AGI-0, a model trained by AGI, Inc., scored 100% on AndroidWorld (116/116 tasks), as listed on the official AndroidWorld leaderboard. AndroidWorld is a common way to compare mobile-use agents.
FAQ
What is AndroidWorld?
A public benchmark for AI agents that operate Android. It has 116 tasks across 20 real apps, and a check on the device's system state decides whether each task succeeded. Researchers at Google DeepMind and Google introduced it, and it was presented at ICLR 2025.
What is a good score on AndroidWorld?
In the original paper, people completed 80.0% of the tasks. AGI-0, a model trained by AGI, Inc., scored 100% on AndroidWorld (116/116 tasks), listed on the official leaderboard.
How is AndroidWorld different from OSWorld?
AndroidWorld tests agents on Android phone apps, with 116 tasks across 20 apps. OSWorld tests computer-use agents on desktop computers, with 369 tasks across desktop apps, files and websites.
