emerging-techniques-speculative-decoding

What emerging-techniques-speculative-decoding does

Three ways to make an LLM answer faster are covered here: a small draft model that proposes tokens for a larger one to verify, Medusa with extra prediction heads, and lookahead decoding built on Jacobi iteration. The agent gets Python snippets: loading target and draft models with transformers, calling medusa_generate with posterior_threshold and posterior_alpha, and setting up LookaheadDecoding with window_size, ngram_size and guess_size.

The algorithm is spelled out too. The draft model guesses K tokens, the target model scores all of them in a single forward pass, and matching ones are kept. Medusa builds a tree of candidates, and the skill separates Medusa-1, where only the heads are trained, from Medusa-2, where the base model is fine-tuned as well.

Examples rely on transformers and torch. Medusa and lookahead decoding each need their own repository cloned and installed with pip install -e ., so without those dependencies the snippets will not run.