Agent eval benchmarks by size
Task-counted benchmarks ranked by size. Environment-based suites (AgentBench is 8 environments, not 8 tasks) and unpublished counts sit in the full table below. Live scores in the Leaderboard.
| name | domain | task count | leaderboard url | src |
|---|---|---|---|---|
| GAIA | general assistant | 466 | ↗ link | ↗ T1 |
| SWE-bench Verified | software engineering | 500 | ↗ link | ↗ T1 |
| OSWorld | computer use | 369 | ↗ link | ↗ T1 |
| WebArena | web navigation | 812 | ↗ link | ↗ T1 |
| τ-bench | tool / customer service | – | ↗ link | ↗ T1 |
| Terminal-Bench | terminal / CLI agents | – | ↗ link | ↗ T1 |
| AgentBench | multi-environment | 8 environments | ↗ link | ↗ T1 |
| Artificial Analysis Image Arena | image generation | ELO | ↗ link | ↗ T2 |
| Artificial Analysis Video Arena | video generation | ELO | ↗ link | ↗ T2 |
| Artificial Analysis Speech Arena | audio / text-to-speech | ELO | ↗ link | ↗ T2 |
| Artificial Analysis Output Speed | inference speed (tokens/sec) | t/s | ↗ link | ↗ T2 |
| BIG-Bench Hard (BBH) | Reasoning | 23 | ↗ link | ↗ T1 |
The eval suites that measure what agents can do, ranked by task count
Twelve benchmarks and 144 scores. Seven are long-horizon agentic harnesses, GAIA, SWE-bench Verified, OSWorld, WebArena, tau-bench, Terminal-Bench and AgentBench, which put a model in an environment and score whether the task actually completed.
Show more
That is a real shift. Only one entry here, BIG-Bench Hard, is classic single-turn reasoning; the rest measure doing rather than answering, including image, video, speech and output speed. Scores move with the harness as much as the model, so a number is only comparable against the same suite.
What is an AI benchmark?
A fixed set of tasks with a scoring rule, run against every model so results can be compared. Benchmarks turn vendor claims into numbers someone else can reproduce. Each one measures a narrow thing: fixing a real GitHub issue, driving a desktop, completing a web task, or holding a tool-using conversation.
What is the difference between a benchmark and a leaderboard?
A benchmark is the task set and the scoring rule. A leaderboard is the table of results from running that benchmark against different models. One is the exam, the other is the results board.
Why do AI benchmarks become saturated?
Because models catch up. Once the leading scores cluster near the ceiling the benchmark stops separating them, and what is left of the gap is mostly noise or errors in the task set. Harder suites then get built, which is why benchmark lists turn over every year or two.
What is the best benchmark for AI coding agents?
SWE-bench Verified is the usual reference: 500 real GitHub issues, checked by human engineers, scored by whether the repository tests pass. Aider polyglot tests code editing across several languages. Match the benchmark to the work you actually want done. (SWE-bench)
Do benchmark scores predict real-world performance?
Only loosely. A benchmark measures a fixed task set under fixed conditions, while production work has messier inputs, longer horizons and real consequences for a wrong step. Long-horizon tool use is barely measured at all. Use scores to shortlist, then test on your own tasks.