Multi Token Prediction is being sold as a free speed hack for local LLMs. Flip one flag in your inference engine and generation speeds up by anywhere from a quarter to a factor of three, at no cost in output quality. That pitch is accurate as far as it goes. However, there is an interesting diagnostic aspect to MTP and running local models, since the size of the MTP gain on a given model is a direct measurement of how badly that model was starved before you turned it on. And what it was starved of is not compute.
The wrong bottleneck
Autoregressive generation has a kind of wasteful inner loop. To produce one token, the engine pulls the model's entire parameter set from VRAM through the GPU's compute units. Then it does it again for the next token, and the next. The arithmetic per token is trivial for a modern GPU, but the data movement is not. A flagship card like a 5090 moves memory at somewhere near 1.8 terabytes per second; the cards most people run local models on manage a few hundred gigabytes. At those rates the compute units spend most of each generation step waiting for weights to arrive. That is the actual constraint on local inference. Not FLOPS, it's bandwidth.
A drafter proposes, a target verifies
MTP belongs to a family of techniques called speculative decoding, first published in 2022, and the surprising part is that nobody teaches the big model to predict several tokens at once. A second, much smaller model does the predicting instead. In the standard terminology the small model is the drafter, the large one the target. The drafter races ahead and proposes a handful of candidate tokens one at a time, which it can afford to do because it is small and fast. The target then checks the whole batch in a single forward pass. Verification is the cheap direction: while checking the drafts, the target also produces one token of its own, so a pass that confirms three draft tokens yields four tokens total. Where the target disagrees with a draft, its own token wins and generation resumes from the disagreement.
Heise's technical explainer walks through the arithmetic with an illustrative example: at a five-in-six acceptance rate, two target passes yield seven verified tokens, turning a nominal 10 tokens per second into an effective 35. But these numbers are more illustrative than a benchmark, but the shape is right. The technique decouples prediction from verification and lets each run at its own speed. How often the drafter guesses right depends on the text. High-probability continuations are easy for a small model, and nothing has higher-probability continuations than code, which follows rules. Free prose in a morphologically rich language drafts worst.
Implementations differ in where the drafter comes from. Google trains an integrated draft head that shares the target's key-value cache and internal activations, a design the vendor credits with better than a doubling of speed; that figure is Google's own. Community lineages include EAGLE, and a preprint from UC San Diego uses a small diffusion model as the drafter, since diffusion architectures emit several tokens per step natively.
Verification spends what generation wastes
The trick is in the accounting. Verifying multiple tokens in one pass costs extra compute, but compute is the resource sitting idle. The expensive part, hauling the full parameter set across the memory bus, happens once per pass regardless of how many tokens come out the other end. When the draft head guesses well, one trip over the bus yields several tokens instead of one. Note that nothing is approximated. The target remains the arbiter of every accepted token, so output is identical to what standard decoding would produce without loosing precision. One boundary follows directly from the design. The drafter only assists generation; parsing the prompt remains the target's job alone, so MTP does nothing for prompt processing. Approaches to speculative parsing exist but are early. That is why measured processing throughput stays flat or dips slightly when MTP is on.
Asymmetry with MoE
Dense models, which activate every parameter for every token, see the largest speedups. A dense 12-billion-parameter model that crawled along at just over 21 tokens per second on a 16-gigabyte consumer card more than tripled to 74 with MTP tuned in a test . A dense 9-billion-parameter model doubled. Mixture-of-experts models gained less. A 35-billion-parameter MoE model with three billion active parameters improved by around 70 per cent on the same card, and by roughly 30 per cent on a faster one. Sparse activation already reduces traffic over the memory bus, so there is less starvation for MTP to relieve. MoE also adds a drafting complication of its own: the drafter's routing can land on a different expert than the target would choose, which depresses acceptance rates. The gain shrinks precisely where the bottleneck loosens.
What one flag buys
The numbers above come from a heise benchmark, run once per configuration: across two consumer GPUs, one AMD and one Nvidia, every tested model gained at least 25 per cent. The best cases exceeded a factor of 3.5. On the faster card, the dense 12-billion-parameter model went from 47 tokens per second to 171. A second benchmark, also from heise, averaged ten runs per configuration and adds two qualifications the single-run test could not surface. First, the gain is content-dependent: coding prompts sped up by 73 to 150 per cent across engines, while German-language prose managed as little as 10 per cent on the same models. Second, the gain is stack-dependent: on older Apple silicon the measured speedup was near zero, which the can be attributed to software not yet optimised for that hardware. So "MTP always helps" might be not true for all loads, but MTP definitely helps most where the drafter can predict and the software is mature.
The tuning surface stays small either way. In Llama.cpp it is two command-line parameters: one to enable speculative MTP decoding, one to cap how many tokens the draft head proposes per pass, with per-model optima between two and seven found by trial. vLLM exposes the same two decisions through its speculative config, and one Llama.cpp fork built on the diffusion drafter sets the draft depth dynamically, removing the last manual knob.
Local Agent Loops
Generation throughput is the binding constraint on local agent systems. An agent loop is not a chat session; it burns tokens on reasoning, tool output, retries and self-correction, and every one of those tokens sits on the critical path. At 21 tokens per second a local loop is a demonstration. At 74 it starts to be useful and since MTP gains are largest on predictable text, it's a perfect fit for code. After all, code is the most predictable text there is. The workload where MTP performs best, structured output under fixed rules, is exactly the workload local agent loops produce. Overall, local open-weight models close the gap to bigger hosted models and the economics start to favour local for high-volume loops. With MTP giving people an edge on hardware people already own, this looks more and more like a path worth taking.
Unlock the Future of Business with AI
Dive into our immersive workshops and equip your team with the tools and knowledge to lead in the AI era.