TL;DRQwen3.8 ships a built-in multi-token prediction head: a 65th block that drafts the next few tokens so the main model can verify several at once. On live traffic it lands 2.49 of every 4 drafted tokens, turning one forward pass into 3.49 tokens of output. But the drafts decay predictably — acceptance falls from 82% at the first position to 45% at the fourth — so more depth is not more speed. The interesting part is that the best depth changed: it was 2 on one card under llama.cpp and is 4 on two cards under vLLM, because the thing that punished depth in July was memory pressure that no longer exists.
How the head works
During decode the MTP head proposes the next n tokens; the main model
verifies all of them in a single forward pass. Accepted tokens are free — they cost no pass of
their own. A rejected token also discards every draft behind it, because those were conditioned
on a guess that turned out wrong. So the head is a lottery where the tickets get cheaper and the
odds get worse the further down the chain you buy.
What the drafts are actually worth
This is not a benchmark. These are the engine's own counters over real user traffic since the last restart — 43,210 draft steps, 172,838 tokens proposed, 107,615 accepted — and they show the decay directly:
| Position | Proposed | Accepted | Rate | Reading |
|---|---|---|---|---|
| 1st | 43,210 | 35,590 | 82.4% | Conditioned only on verified text |
| 2nd | 43,210 | 28,813 | 66.7% | One assumption deep |
| 3rd | 43,210 | 23,596 | 54.6% | Two deep |
| 4th | 43,210 | 19,616 | 45.4% | Three deep — below a coin flip |
| All four | 172,838 | 107,615 | 62.3% | 2.49 tokens per step, free |
Read from the engine's per-position acceptance counters on the production box, 16 August 2026. Because the verified token is produced anyway, a forward pass yields 3.49 tokens on average rather than one — which is where the free speed comes from, and why the site decodes at ~101 tok/s on an idle engine instead of ~29.
The depth sweep, on the machine that runs today
Acceptance decaying does not by itself tell you where to stop. Each extra position costs draft compute and verification width whether it lands or not, so the answer is empirical. On the two-card vLLM stack:
| Draft depth | Decode tok/s | Verdict |
|---|---|---|
| k = 2 | 102.6 | Leaves acceptance on the table |
| k = 4 — shipped | 108.2 | Best measured |
| k = 6 | 95.9 | Draft and verify cost outruns the gain |
Single stream, idle engine, ~2.8K prompt. The spread is narrow — about 12% from best to worst — which is itself the finding: on hardware with headroom, depth is a tuning knob. On hardware without it, as below, the same knob was a cliff.
The same sweep, one card, July
| Draft depth | Short | Prose | Code | 16K + reasoning | Mean tok/s |
|---|---|---|---|---|---|
| n = 2 — chosen then | 55.4 | 47.2 | 54.7 | 47.4 | 51.2 |
| n = 3 | 30.1 | 28.9 | 35.1 | 29.9 | 31.0 |
| n = 4 | 25.8 | 20.7 | 26.3 | 18.3 | 22.8 |
| n = 5 | 18.6 | 15.0 | 22.3 | 13.3 | 17.3 |
Why deeper lost twice — and why one of those reasons went away
1 — The draft context was real memory
The head keeps a draft context per slot, and it scales with depth. At depth 3+ on a 2-slot 84K pool, that extra context blew the layout budget and llama.cpp evicted layer 0 to the CPU — the exact failure documented in Note 001, with the same log signature: fused Gated-DeltaNet disabled, VRAM going down, decode halved. The speed you hoped to win was spent before the first token.
This is the reason that no longer applies. Two cards, tensor-parallel, with the pool sized explicitly rather than auto-fitted, means depth 4 does not push anything off the card. The cliff between n=2 and n=3 in the July table is not a fact about speculation; it is a fact about a 24 GB budget that was already full.
2 — Deeper guesses are worse guesses
This one is structural and did not go away. Acceptance fell from 76% to 41% on short prompts as depth rose in July, and the live counters in section 02 show the same shape today: 82% down to 45% across four positions on the current stack. Each extra drafted token is conditioned on unverified guesses, so the chain breaks more often, and every broken chain wastes the verification pass it was riding on. Deeper speculation buys more losing lottery tickets — that part is the model, not the machine.
What this leaves
Two rules came out of this, and only one of them travels. The depth limit is real — acceptance decays monotonically, so there is always a k past which drafting costs more than it returns. Where the limit sits is a property of your hardware, and a number inherited from someone else's box, or from your own box two months ago, is a guess wearing a measurement's clothes. The July configuration was right in July and would cost about 5% today.
The head's own ceiling is its acceptance rate, and that is where the next gain has to come from. A dedicated draft model is in training to push past it; an off-the-shelf drafter benchmarked in July lost to the built-in head on this workload and was rejected. Until the data changes: the depth the measurements chose, not the maximum the flag allows.