Inference throughput
Independent output speed (Artificial Analysis), tagged with the model measured, tokens/sec swings 5–20x by model size, so a single unqualified figure misleads. Bars show the headline number; the table carries the sourcing.
| name | country | type | throughput | src |
|---|---|---|---|---|
| Groq | US | inference_api | ~321 t/s · Llama 3.3 70B (≈800 t/s on 8B-class) | ↗ T1↗Groq |
| Cerebras | US | inference_api | ~2,314 t/s · Llama 3.3 70B (~981 t/s on Kimi K2.6) | ↗ T1↗Cerebra |
| Together AI | US | inference_api | ~198 t/s · Kimi K2.6 (35–434 t/s across models) | ◆4↗ T1↗Togethe |
| Fireworks AI | US | inference_api | ~373 t/s · Kimi K2.6 (up to 740 t/s on gpt-oss-120b) | ◆5↗ T1↗Firewor |
| SambaNova | US | inference_api | ~691 t/s · gpt-oss-120b (high) | ◆2↗ T1↗SambaNo |
| Nebius | NL | inference_api | ~343 t/s · gpt-oss-120b (high) | ◆1↗ T1↗Nebius |
| Baseten | US | inference_api | ~223 t/s · gpt-oss-120b (high) | ↗ T1↗Baseten |
| DeepInfra | US | inference_api | ~154 t/s · gpt-oss-120b (high, Turbo) | ↗ T1↗DeepInf |
| Novita AI | US | inference_api | ~133 t/s · gpt-oss-120b (high) | ↗ T1↗Novita |
Beyond a single host: the managed cloud catalogues, edge hosts and gateways an agent draws models from. Cloud catalogues serve their own line-up; gateways proxy hundreds behind one endpoint. Every count is from the platform's own docs.
| Platform | Kind | Models | Notable | src |
|---|---|---|---|---|
| Amazon Bedrock Amazon Web Services | Managed cloud catalogue | 100+ models | Anthropic Claude · Meta Llama · Mistral · Cohere · AI21 · Amazon Nova/Titan · DeepSeek · OpenAI · Stability | ↗ T1 |
| Azure AI Foundry (Microsoft Foundry) Microsoft | Managed cloud catalogue | 1,900+ sold directly (11K+ incl. community) | Azure OpenAI · Claude · Meta · Mistral · DeepSeek · xAI Grok · Cohere · NVIDIA · Fireworks · Hugging Face | ↗ T1 |
| Google Vertex AI Model Garden Google | Managed cloud catalogue | 200+ models | Gemini · Claude · Llama · Mistral · Gemma · plus curated open models | ↗ T1 |
| Cloudflare Workers AI Cloudflare | Edge-hosted | ~50 open models | Llama · Mistral · Qwen · Gemma · embeddings + vision, served at the edge | ↗ T1 |
| Netlify AI Gateway Netlify | Gateway (proxy) | Multi-provider | Provider proxy with usage controls for agents on Netlify Functions + Edge | ↗ T2 |
| OpenRouter OpenRouter | Gateway (proxy) | 300+ models | One API key across every major provider · routing, load-balancing, fallbacks | ↗ T1 |
| Vercel AI Gateway Vercel | Gateway (proxy) | Hundreds of models | Unified API across providers · text/image/video/embeddings · BYOK, budgets, fallbacks | ↗ T1 |
| Salesforce Einstein / Agentforce Salesforce | Model-choice layer | Curated + bring-your-own-LLM | Model choice behind the Einstein Trust Layer (masking, zero-retention) · Models API · BYOLLM | ↗ T2 |
The reasoning cores and capability models, voice, vision, embeddings, that agents plug into at the edge. A supporting layer: agents are the subject; these are what powers them.
Independent tokens-per-second, plus where agents get their models
Nine inference providers sit between the model and the application, selling tokens rather than machines. For most teams this is the layer they actually buy.
Show more
Competition here is on latency and throughput more than price, because the price floor is set by hardware everyone rents from the same suppliers. Time-to-first-token is the number that matters for agents, since a loop that calls the model ten times pays that latency ten times.
What is inference in AI?
Inference is running a trained model to get an answer, as opposed to training it in the first place. Every time an agent calls a model, that is one inference. It is the cost a production system actually pays, over and over.
What is a token, and what is tokens per second?
A token is a chunk of text, roughly three-quarters of a word on average. Tokens per second is how fast a model produces its answer. Independent benchmarks publish measured speeds per model and provider (Artificial Analysis).
What is time to first token?
The delay between sending a request and the first word coming back, as distinct from how fast the rest streams. It matters more for agents than raw speed, because a loop that calls the model ten times waits ten times.
Why is the same model faster on one provider than another?
Different hardware, different batching, and different quantisation. Providers trade a little accuracy or a lot of cost against speed, so an identical model name can behave very differently depending on who is serving it.
What is edge inference?
Running models on servers close to the user instead of in one distant region. It cuts round-trip latency and can keep data inside a country. It usually means smaller models, since the biggest ones will not fit at the edge.