--- title: "Agent eval benchmarks · ranked by task count · Agent Fieldbook" url: https://agentfieldbook.org/software/benchmarks/ description: "The evaluation benchmarks that measure what AI agents can do, ranked by task count. Task-counted suites in the chart, environment-based and unpublished ones in the full table. A source-graded field book to agent evals, with live scores in the Leaderboard." section: "Stack · Benchmarks" source: Agent Fieldbook — generated from the published page --- # Agent eval benchmarks by size Task-counted benchmarks ranked by size. Environment-based suites (AgentBench is 8 environments, not 8 tasks) and unpublished counts sit in the full table below. Live scores in the Leaderboard. ## 12 rows Live data · click a row for its full spec | name | domain | task count | leaderboard url | src | | --- | --- | --- | --- | --- | | GAIA | general assistant | 466 | [↗ link](https://huggingface.co/spaces/gaia-benchmark/leaderboard) | [↗ T1](https://huggingface.co/gaia-benchmark) | | SWE-bench Verified | software engineering | 500 | [↗ link](https://www.swebench.com) | [↗ T1](https://www.swebench.com) | | OSWorld | computer use | 369 | [↗ link](https://os-world.github.io) | [↗ T1](https://os-world.github.io) | | WebArena | web navigation | 812 | [↗ link](https://webarena.dev) | [↗ T1](https://webarena.dev) | | τ-bench | tool / customer service | – | [↗ link](https://github.com/sierra-research/tau-bench) | [↗ T1](https://github.com/sierra-research/tau-bench) | | Terminal-Bench | terminal / CLI agents | – | [↗ link](https://www.tbench.ai) | [↗ T1](https://www.tbench.ai) | | AgentBench | multi-environment | 8 environments | [↗ link](https://github.com/THUDM/AgentBench) | [↗ T1](https://github.com/THUDM/AgentBench) | | Artificial Analysis Image Arena | image generation | ELO | [↗ link](https://artificialanalysis.ai/text-to-image/arena/leaderboard-text) | [↗ T2](https://artificialanalysis.ai/text-to-image/arena/leaderboard-text) | | Artificial Analysis Video Arena | video generation | ELO | [↗ link](https://artificialanalysis.ai/video/leaderboard/text-to-video) | [↗ T2](https://artificialanalysis.ai/video/leaderboard/text-to-video) | | Artificial Analysis Speech Arena | audio / text-to-speech | ELO | [↗ link](https://artificialanalysis.ai/text-to-speech/leaderboard/selected-voice) | [↗ T2](https://artificialanalysis.ai/text-to-speech/leaderboard/selected-voice) | | Artificial Analysis Output Speed | inference speed (tokens/sec) | t/s | [↗ link](https://artificialanalysis.ai/leaderboards/models) | [↗ T2](https://artificialanalysis.ai/leaderboards/models) | | BIG-Bench Hard (BBH) | Reasoning | 23 | [↗ link](https://paperswithcode.com/sota/multi-task-language-understanding-on-bbh) | [↗ T1](https://arxiv.org/abs/2210.09261) | ## Field notes The eval suites that measure what agents can do, ranked by task count Twelve benchmarks and 144 scores. Seven are long-horizon agentic harnesses, GAIA, SWE-bench Verified, OSWorld, WebArena, tau-bench, Terminal-Bench and AgentBench, which put a model in an environment and score whether the task actually completed. That is a real shift. Only one entry here, BIG-Bench Hard, is classic single-turn reasoning; the rest measure doing rather than answering, including image, video, speech and output speed. Scores move with the harness as much as the model, so a number is only comparable against the same suite. - [Brandon Chaplin](https://www.linkedin.com/in/brandon-chaplin-digital-marketing-strategist) ## Common questions ### What is an AI benchmark? A fixed set of tasks with a scoring rule, run against every model so results can be compared. Benchmarks turn vendor claims into numbers someone else can reproduce. Each one measures a narrow thing: fixing a real GitHub issue, driving a desktop, completing a web task, or holding a tool-using conversation. ### What is the difference between a benchmark and a leaderboard? A benchmark is the task set and the scoring rule. A leaderboard is the table of results from running that benchmark against different models. One is the exam, the other is the results board. ### Why do AI benchmarks become saturated? Because models catch up. Once the leading scores cluster near the ceiling the benchmark stops separating them, and what is left of the gap is mostly noise or errors in the task set. Harder suites then get built, which is why benchmark lists turn over every year or two. ### What is the best benchmark for AI coding agents? SWE-bench Verified is the usual reference: 500 real GitHub issues, checked by human engineers, scored by whether the repository tests pass. Aider polyglot tests code editing across several languages. Match the benchmark to the work you actually want done. [(SWE-bench)](https://www.swebench.com/) ### Do benchmark scores predict real-world performance? Only loosely. A benchmark measures a fixed task set under fixed conditions, while production work has messier inputs, longer horizons and real consequences for a wrong step. Long-horizon tool use is barely measured at all. Use scores to shortlist, then test on your own tasks. ## Next in the learning path - [Leaderboard Live model scores across these benchmarks](https://agentfieldbook.org/software/leaderboard/) - [Models The models being scored](https://agentfieldbook.org/software/models/) - [Frameworks & Tools What agents are built from](https://agentfieldbook.org/software/frameworks/) - [Safety Evals Benchmarks for safety, not just capability](https://agentfieldbook.org/security/evals/) - [Examples Reference agents you can measure](https://agentfieldbook.org/software/cookbooks/) ## Data - [benchmarks.json](https://agentfieldbook.org/data/benchmarks.json) Licensed [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).