What is a draft model?
A draft model is the small, fast model used alongside a larger target model in speculative decoding. Instead of generating output itself, the draft model's job is to guess several tokens ahead cheaply, so the target model can check all of those guesses in one pass rather than generating each token through its own, slower forward pass.
What the draft model actually does
At each step, the draft model generates a short run of candidate tokens on its own, using whatever context has been accepted so far. Because it's small, it can do this several times over at little cost.
The target model then scores all the candidates in a single forward pass, checking token by token whether its own probability distribution would have picked the same token. Matches are accepted. At the first mismatch the draft token is discarded, the target's own token is used instead, and drafting resumes from there. The draft model never decides what the final output is; it only proposes.
Why a small model can do this job
A draft model doesn't need to match the target model's quality. It needs to be right often enough on the easy parts of a sequence to save work, and cheap enough that being wrong sometimes costs little. Predictable spans (common phrasing, boilerplate code, repeated structure) are where a much smaller model tends to agree with a much larger one. The pairing works because language has many low-entropy stretches a lightweight model can predict correctly.
How draft models are chosen
The most common approach pairs a target model with a smaller model from the same family, such as a distilled or lower-parameter variant trained on similar data, since shared training tends to produce more agreement. Some serving stacks skip the second model: they draft from the target model's own early layers (early exit) or from an n-gram table built from the current context, so no second set of weights has to be loaded. Which approach is best depends on how much unified memory is free to hold a second model next to the target, and on how much draft quality pays off for your workload.
The tradeoff that decides whether it's worth it
Every draft token costs compute, whether or not the target model accepts it. A draft model that is too weak gets rejected often, and the wasted drafting can erase the speedup or make requests slower than plain decoding. A draft model too close in size to the target saves less compute per accepted token, even with a high acceptance rate.
Picking a draft model means balancing those two failure modes, so test it against your own prompts rather than assuming a published pairing transfers. Running a target and a draft model together also means budgeting unified memory for both; check what fits in memory before you plan a pairing on a single node.