Design · Orchestration
Speculative Decoding
A small draft model proposes several tokens ahead; the large target model verifies them all in one parallel pass. Correct guesses are kept, so many tokens land per expensive forward pass.
Verified against arxiv.org
How it works
A small draft model proposes several tokens ahead; the large target model verifies them all in one parallel pass. Correct guesses are kept, so many tokens land per expensive forward pass.
At a glance
TechniqueDecoding
Cost leverNeutral on cost, same target model
Latency lever2–3× faster inference with no accuracy loss
Quality leverIdentical to the target model (verification is exact)
ExamplevLLM · TensorRT-LLM · Medusa
When to use
Latency-sensitive serving of a fixed large model, chat, coding autocomplete, where you want speed without changing outputs.
Basis
Verified against
https://arxiv.org/abs/2211.17192 The corpus does not rewrite vendor documentation. Where copy is quoted verbatim it is marked as such and attributed to its source.