Spend expensive tokens only when they matter
The techniques that decide which model handles which query: routing, cascades, ensembles, caching, decoding. This is the cost/latency control layer of a multi-model system: how you get frontier quality on the hard 10% without paying frontier rates on the easy 90%. Two layers: the techniques (the levers) and the suppliers (the gateways + routers who productise them).
| Technique | How it works | Cost | Latency | When to use | Example | src |
|---|---|---|---|---|---|---|
| Mixture-of-Agents (MoA) Ensemble |
Layers of LLMs where each layer's agents take all prior-layer responses as auxiliary context, then a final aggregator synthesises. Quality compounds across layers. | Higher, runs many models per query | Higher, sequential layers | Quality-critical tasks where you can trade cost/latency for a better answer, and want to avoid a single frontier dependency. | Together AI MoA · Self-MoA | ↗ T1 |
| LLM Routing (RouteLLM) Routing |
A learned router scores each query and sends easy ones to a cheap model, hard ones to a strong model. Trained on human-preference signals to hold quality while cutting spend. | Much lower, up to 85% cost cut at GPT-4-level quality on MT-Bench | Lower on average, most queries hit the small model | High-volume mixed-difficulty traffic where most queries are easy and a minority need a frontier model. | LMSYS RouteLLM · Martian · OpenRouter auto | ↗ T1 |
| Model Cascade (FrugalGPT) Cascade |
Try the cheapest model first; a confidence/verifier signal decides whether to escalate to a stronger (pricier) model. Queries stop at the first model good enough. | Dramatically lower, up to 98% cost reduction reported | Lower for easy queries, higher for escalated ones | Budget-constrained workloads with a long tail of easy queries and a clear signal for when the cheap answer is unreliable. | FrugalGPT · FORC | ↗ T1 |
| Speculative Decoding Decoding |
A small draft model proposes several tokens ahead; the large target model verifies them all in one parallel pass. Correct guesses are kept, so many tokens land per expensive forward pass. | Neutral on cost, same target model | 2–3× faster inference with no accuracy loss | Latency-sensitive serving of a fixed large model, chat, coding autocomplete, where you want speed without changing outputs. | vLLM · TensorRT-LLM · Medusa | ↗ T1 |
| Semantic Caching Caching |
Embed each incoming query; if a semantically similar past query is in the cache, return its stored answer instead of calling the model. Cache-hit queries cost nothing. | Lower, cache hits skip the model entirely | Much lower on cache hits (ms vs seconds) | Workloads with repetitive or near-duplicate queries (FAQ bots, support, docs search). | GPTCache (Zilliz) · Portkey semantic cache | ↗ T2 |
| Prompt Caching Caching |
Cache the model's internal state for a long, stable prefix (system prompt, tool defs, docs) so repeat calls only pay for the new suffix. Vendor-native at the API layer. | Lower, cached prefix tokens billed at ~10% of input rate | Lower, cached prefix skips recompute | Agents with large fixed context (long system prompt, big tool schema, RAG docs) reused across many turns. | Anthropic prompt caching · OpenAI / Gemini cached content | ↗ T1 |
| Difficulty-Aware Orchestration Routing |
Estimate each query's difficulty first, then adapt workflow depth, which operators fire, and which model handles it. Hard queries get more agents + a stronger model; easy ones get a shallow cheap path. | Lower on average, spends compute only where difficulty warrants | Variable, matched to query difficulty | Agent workflows facing a wide difficulty range, where a fixed pipeline over-serves easy tasks and under-serves hard ones. | DAAO · MoMA generalised routing | ↗ T1 |
| Vendor | Category | How it works | Hosts | OSS | src |
|---|---|---|---|---|---|
| OpenRouter | Managed gateway + router | One API key, 300+ models, zero infrastructure: handles routing, load-balancing and fallbacks. The simplest managed way to route across providers. | 300+ models | no (managed) | ↗ T1 |
| LiteLLM | Open-source proxy/gateway | OpenAI-compatible proxy for 100+ providers with fallbacks and budget controls. Self-host on any VPS: the OSS default for teams that want to own the routing layer. | 100+ providers | yes (open-source) | ↗ T1 |
| Cloudflare AI Gateway | Edge gateway | A gateway in front of any provider, caching, rate-limiting, retries, fallbacks and analytics at the edge. Semantic + response caching cut repeat-query cost to zero. | any provider | no (platform) | ↗ T1 |
| Vercel AI Gateway | Managed gateway | Unified endpoint for hundreds of models with automatic failover, budget controls and observability: built into the Vercel AI SDK stack. | hundreds of models | no (platform) | ↗ T1 |
| Together AI | Inference + ensemble | Fast inference for open models plus a reference Mixture-of-Agents implementation: the vendor behind the MoA ensemble technique. | 200+ open models | partial | ↗ T1 |
| Martian | Model router | A dedicated model router that predicts the best model per query on cost/quality, sending each request to the cheapest model that will still answer it well. | multi-provider | no (managed) | ↗ T2 |
| Not Diamond | Model router | An intelligent LLM router that learns which model wins on which prompt type and routes accordingly: routing-as-a-service without running your own classifier. | multi-provider | partial | ↗ T2 |
| Unify AI | Router + gateway | Routes across providers on a live quality/cost/latency benchmark, with a single endpoint and runtime routing. Suited to straightforward cost-aware routing. | multi-provider | partial | ↗ T2 |
Frontier quality on the hard 10% without frontier rates on the easy 90%
The seven techniques here are about cost and latency, not choreography: mixture-of-agents, LLM routing, model cascades, speculative decoding, semantic caching, prompt caching and difficulty-aware orchestration. Each decides which model answers, or whether a model needs to answer at all.
Show more
The common move is to stop paying frontier prices for easy work. Routing and cascades send the cheap question to the cheap model and escalate only what needs it; caching removes the call entirely; speculative decoding makes the same call finish sooner. The saving is real and the cost is a second system to reason about when an answer looks wrong.
What is model orchestration?
The layer that decides which model handles which request. Rather than sending everything to the most expensive model, it routes, retries, caches and falls back. Once an application has real traffic this is the main lever on both cost and latency.
What is model routing?
A classifier reads each request and sends it to the cheapest model that can handle it, keeping the frontier model for the hard ones. Because most real traffic is easy, routing usually keeps most of the quality for a fraction of the spend.
What is a model cascade?
Try a small model first, check the answer, and escalate to a larger model only if the check fails. Routing decides up front, a cascade decides after a first attempt. It costs an extra call when it escalates and saves a lot when it does not.
What is prompt caching?
Reusing the model's processing of a repeated chunk of prompt instead of paying for it again. Providers charge much less for cached input tokens, so long system prompts and fixed context get cheaper and faster on every call after the first (OpenAI).
What is speculative decoding?
A small draft model guesses the next few tokens and the large model verifies them in a single pass. Correct guesses come almost free, so responses arrive faster with identical output (arXiv).
What is an LLM gateway?
One endpoint sitting in front of several model providers. It handles routing, retries, failover when a provider goes down, spend limits and logging, so your application code never needs to know which model answered.