← Learn / Notebooks

Models & toolsTechnicalStudy guide

Speculative Decoding: Draft and Verification

Inspect a greedy draft-token prefix, locate the first mismatch and follow the target correction.

Before you start: Autoregressive token generation and the meaning of greedy decoding.

AI-assisted study notebook · Published 14 Nov 2026. Source-support checks and synthetic examples are recorded; automated review is not independent expert fact verification.

In [1]: Architecture

1. Architecture and scope

Autoregressive generation produces a continuation one token after another. Greedy decoding chooses the most likely next token at each step. Assisted generation introduces a draft model that proposes several candidate tokens, then a target model checks the proposal. The saved Intel article describes that greedy proposal-and-check arrangement. [1] [2]

The original speculative decoding paper also describes preserving the target distribution under its full method. That result does not follow from accepting tokens merely because their text matches in a toy display. Stochastic acceptance and correction rules are outside this notebook; the example isolates the simpler greedy prefix boundary. [3]

Scope. A greedy token-agreement illustration. It does not implement the general stochastic speculative sampling algorithm or measure acceleration.

Workflow and scope notes
  1. Draft candidates. A draft proposes a sequence following the committed prefix. The toy supplies tokens by hand; it does not run a draft model.
  2. Check with target. Compare greedy target choices while the preceding draft prefix still agrees. A target check at a later position conditions on earlier draft tokens.
  3. Keep matching prefix. Accept the contiguous agreeing prefix and use the target choice at the first mismatch. Later draft tokens are discarded even if a displayed token happens to match.
  4. Continue generation. Resume from the corrected prefix for the next round. This toy omits the extra target token that some algorithms emit when every candidate is accepted.
In [2]: Experiment

2. Draft prefix acceptance

The fixed greedy target continuation is A B C D E F. Choose a proposed draft length and the position of its first mismatch (X). Tokens after that position are shown but are not eligible for acceptance in this round.

Draft prefix acceptance

Original synthetic experiment · Change an input to update the figure.

If the chosen mismatch is beyond K, the entire proposal agrees. On a mismatch, commit only the matching prefix and one target correction. The display compares a known greedy target sequence; no speed, model call count or stochastic distribution is measured.

Draft, target and committed tokens2 draft tokens accepted · Target correction C · Commit A B C. Tokens after the first mismatch are discarded.DraftABXDFirst mismatchTarget referenceABCDCommittedABC·Purple: accepted draft · Coral: mismatch · Green: target correction
2 draft tokens accepted · Target correction C · Commit A B C
Calculation record
{
  "inputs": {
    "length": "4",
    "mismatch": "3"
  },
  "draft": [
    "A",
    "B",
    "X",
    "D"
  ],
  "accepted": 2,
  "correction": "C",
  "committed": [
    "A",
    "B",
    "C"
  ]
}
CaseResultInterpretation
K=4, mismatch at 3Accept A B; correct with CDraft token X and the remaining proposal are discarded.
K=4, mismatch at 1Accept no draft tokens; correct with AAgreement must be contiguous from the start.
K=4, no mismatchAccept A B C DThis toy stops at K and omits the possible bonus target token.
In [3]: Interpretation

3. Interpretation and limitations

After the first mismatch, subsequent draft tokens were proposed under a prefix that will not be committed. Their apparent agreement with a prewritten target string is not a reason to accept them. The shaded discard area marks that conditional dependency in this greedy illustration.

The draft needs to be inexpensive enough, and its useful accepted work large enough, to offset extra operations in an actual implementation. This notebook does not estimate that balance. In particular, “four tokens proposed” is not a fourfold speed result. [1]

Editorial experiment plan: record target and draft versions, tokenizer compatibility, decoding settings, acceptance statistics, latency and hardware. Compare output behavior under the actual algorithm. The paper’s reported results belong to its stated experiments, not to this synthetic animation. [1] [3]

Practice exercise

Use K=6 and a mismatch at 2. Explain why positions 3–6 are discarded even though their letters match the fixed reference. Then distinguish the greedy example from preserving a sampled output distribution.

In [4]: Python

4. Worked Python example

The core calculation below uses only the Python standard library. The full download adds the verification assertions and input fixtures. It runs locally on synthetic data; it does not load a model or contact a service.

lab.py · Core example · Python 3.13
TARGET = list("ABCDEF")


def greedy_round(length, mismatch):
    # Greedy prefix illustration, not stochastic speculative sampling.
    draft = TARGET[:length]
    if 1 <= mismatch <= length:
        draft[mismatch - 1] = "X"
    accepted = 0
    while accepted < length and draft[accepted] == TARGET[accepted]:
        accepted += 1
    correction = TARGET[accepted] if accepted < length else None
    committed = draft[:accepted] + ([correction] if correction else [])
    return {"draft": draft, "accepted": accepted,
            "correction": correction, "committed": committed}

Download full Python example and checks ↓

Run: python -I lab.py. Recorded run: Python 3.13; 21 fixture states plus scenario assertions passed. The JSON output is a reproducibility record, not a real-system benchmark.

In [5]: Knowledge check

5. Questions and explanations

Predict before opening the explanation. Answers refer to the stated scope and synthetic inputs.

Q1 · PrefixK=4 and first mismatch at 3: how many draft tokens are accepted?
K=4 and first mismatch at 3: how many draft tokens are accepted?
Answer and explanation

Answer: 2. Only positions before the first mismatch are accepted from the draft.

Q2 · ConditioningWhy are later draft tokens discarded after a mismatch?
Why are later draft tokens discarded after a mismatch?
Answer and explanation

Answer: Their prefix is no longer the committed prefix. They were proposed conditioned on the draft prefix that included the mismatch.

Q3 · ScopeDoes this toy implement distribution-preserving stochastic sampling?
Does this toy implement distribution-preserving stochastic sampling?
Answer and explanation

Answer: No. It is explicitly a greedy token-agreement illustration. [3]

Q4 · PerformanceWhat speedup has this notebook measured?
What speedup has this notebook measured?
Answer and explanation

Answer: None. It runs no target or draft model and measures no inference latency.

In [6]: Sources

References and source links

  1. Intel / Hugging Face authors. (2024). Accelerate StarCoder with 🤗 Optimum Intel on Xeon: Q8/Q4 and Speculative Decoding.https://huggingface.co/blog/intel-starcoder-quantization
  2. Hugging Face maintainers. (Main documentation; accessed 2026). Generation strategies · Hugging Face.https://huggingface.co/docs/transformers/main/en/generation_strategies
  3. Leviathan, Y., Kalman, M., & Matias, Y. (2023). Fast Inference from Transformers via Speculative Decoding.https://proceedings.mlr.press/v202/leviathan23a.html
Source checks and provenance

Primary publisher pages and relevant passages checked on 13 Nov 2026. Source support is not independent expert fact verification. Main documentation can change; retrieved-body hashes, dates, version labels and saved revision IDs are in the audit download.

[1] Accelerate StarCoder with 🤗 Optimum Intel on Xeon: Q8/Q4 and Speculative Decoding

Evidence: Step 4: Assisted Generation. Saved public original and current article read. Hardware-specific speed figures are excluded.

Public ResearchNode original c475c54b6d53b2d013c77bba1e480f33
Revision a95abf005502e6800d2c6c8ab60d9cf8. Captured 2026-11-06.

[2] Generation strategies · Hugging Face

Evidence: Greedy search. Greedy decoding definition; not a claim that all generation uses this policy.

[3] Fast Inference from Transformers via Speculative Decoding

Evidence: PMLR abstract; ICML 2023, vol. 202, pp. 19274–19286. The paper’s abstract describes the full method. It is not used as detailed algorithm evidence for this simplified toy.

Claim-to-evidence ledger ↓ · Review report ↓