Data Sources & Methodology
Agent Fieldbook is built to be reference-grade: every datapoint traces to an established primary source so it can be cited. We don't own the sources; the proprietary lift is how we classify, tier, blend and verify them. Here is exactly how the data is sourced, graded and kept current.
T1: Primary / authoritative
Official vendor APIs, docs & pricing pages · SEC filings · standards bodies (ISO, NIST, EU OJ) · peer-reviewed & university research (METR, NBER, arXiv) · authoritative datasets (Epoch AI, LiteLLM) · official registries (MCP).
T2, Reputable secondary
Well-maintained open-source GitHub · established research/industry orgs publishing data (e.g. McKinsey, PwC, labelled secondary) · reputable aggregators.
T3, Weak / disallowed
Marketing pages without primary data · undated or low-quality blogs · AI-content farms · single unverified claims. Used only as a pointer, never the basis of a "confirmed" datapoint.
Convergence, not divergence. Where a fact should agree across sources (a price, a ranking), we corroborate; divergence is a signal to re-source, not to average. The build gate flags any "confirmed" row resting only on a weak/T3 source.
Data = a sourced fact. Concept = our classification or framework with no single raw source (the agency spectrum, archetypes, ontologies, the memory taxonomy, parts of the Workforce models). Concept tables are flagged in the manifest and never presented as measured data. Where we genuinely don't know, the cell reads –: never a guess.
Live
Fetchers hit public APIs (models, chips, MCP registry, GitHub stars): re-run weekly, near-zero effort.
Curated
Authored sourced facts (pricing, adoption, security): reviewed on a window; the build prints what's due.
Static
Slow-moving taxonomies/frameworks: reviewed rarely.
A single validator (scripts/build.mjs) gates every change: it checks referential integrity, enforces the controlled vocabulary, derives each source's tier, and flags weak-confirmed rows + a review worklist. One amend → re-run → every gap surfaces.
| Source | License | Feeds |
|---|---|---|
| Epoch AI | CC-BY: attribution required | chips / hardware |
| LiteLLM | MIT | model pricing |
| Aider | Apache-2.0 | coding leaderboard |
| OpenRouter | open API | model catalogue |
| Official MCP Registry | open | MCP servers |
| GitHub API | per-repo | framework/tool stars |
| METR | research | task-horizon |
| NBER | research | productivity benchmarks |
| Cloudflare Workers AI | open docs | edge inference |
Every datapoint carries its own source link in-place (the ↗ on each table row and in each agent profile). Corrections welcome: Agent Fieldbook is an open intelligence project.
Chaplin, B. (2026). Agent Fieldbook: the open field book to AI agents. Scalarity. https://agentfieldbook.org/ @misc{agentatlas,
author = {Chaplin, Brandon},
title = {Agent Fieldbook: the open field book to AI agents},
year = {2026},
publisher = {Scalarity},
howpublished = {\url{https://agentfieldbook.org/}}
} https://agentfieldbook.org/ How are sources graded?
On three tiers. Tier one is primary and authoritative: vendor documentation and pricing, regulatory filings, standards bodies, peer-reviewed research, official registries. Open source counts as tier one when it is published by an established organisation, a lab, university, vendor or standards project, because that is the organisation publishing its own work. Tier two is reputable secondary work: analysts and industry researchers who study the primary record rather than being it. Tier three is weak. Marketing roundups and 'best tools' listicles land here and are never the basis of a confirmed row.
What do the confidence grades mean?
Every row carries one. Confirmed means checked against a tier-one or tier-two source. Likely is an informed estimate, emerging is a best read from early signals, and single-source is one credible source with nothing corroborating it yet. Projected means a forward estimate, and it is always labelled as one.
What happens when two sources disagree?
We go back and re-source rather than split the difference. Where two sources should agree, such as on a price, disagreement usually means one is stale or wrong. Averaging would hide that.
How often is the data updated?
It depends on the table. Live tables such as models, chips and the MCP registry are refetched weekly from public APIs. Curated tables like pricing and adoption are reviewed on a fixed window, and the build prints what is due. Taxonomies move slowly and are reviewed rarely.
What is the difference between data and concept here?
Data is a sourced fact. Concept is our own classification or framework, such as the agency spectrum or the memory taxonomy, which has no single source to cite. Concept tables are flagged as such, so they are never read as measurements.
How is data quality enforced?
By the build script, not by reviewer discipline. It checks that every reference resolves, enforces the controlled vocabulary, works out each source's tier from an allow-list of domains, and flags any confirmed row resting only on a weak or unrecognised source so it can be re-sourced or downgraded. It also prints what is due for review.