Speculative decoding speeds up LLM generation by pairing a big, accurate model with a small, fast draft model. The draft guesses several tokens ahead; the big model verifies them all in a single forward pass and accepts the longest correct prefix. You get the big model's quality at a fraction of the latency.
Why it works
Generating one token at a time is memory-bandwidth bound — the GPU spends most of its time loading weights, not computing. Verifying 5 proposed tokens costs almost the same as generating 1, so every accepted guess is nearly free.
draft model: proposes [the, cat, sat, on, the]
big model: verifies → accepts [the, cat, sat], rejects "on"
→ 3 tokens for the price of ~1 stepIn practice
- Speedups of 2–3× are common with no quality loss (the big model has final say).
- The draft model must share the same tokenizer / vocabulary.
- Built into vLLM and llama.cpp (
--draft-modelstyle flags).
Free latency if you have spare VRAM for the small model — pure win for interactive use.
Related: vLLM vs Ollama vs llama.cpp · Ollama Parameters Guide