TL;DRQwen3.8-27B is a hybrid: only every fourth block runs full attention. The other 48 are Gated-DeltaNet linear-attention blocks whose state is a fixed size — it does not grow with the conversation at all. So a token of context costs 32 KiB here against the 128 KiB a dense 27B would pay at the same precision. Context is 4× cheaper by construction, before a single flag is tuned. That is what lets two 24 GB cards hold a 262,144-token request beside the weights, and it is the quiet foundation under "free."
The stack, verified
The model's config declares full_attention_interval = 4, and the tensor
names confirm it rather than merely repeating it: only blocks 3, 7, 11 … 63 carry
attn_k and attn_v weights — 16 of 64 — plus block 64, the multi-token
prediction head (Note 002). The remaining 48 blocks carry
ssm_* recurrent-state tensors instead.
What a token of context costs
Per token, only the 16 attention blocks pay: 16 layers × 4 KV-heads × 256 key-length, keys and values both, is 32,768 elements per token. Everything else follows from the precision those elements are stored in:
| Cache | Per token | A full 262,144-token request | Note |
|---|---|---|---|
| Hybrid, f16 | 64 KiB | 16.0 GiB | Uncompressed |
| Hybrid, fp8 — shipped | 32 KiB | 8.0 GiB | What runs today |
| Dense 27B, fp8 | 128 KiB | 32.0 GiB | If all 64 blocks attended |
| Dense 27B, f16 | 256 KiB | 64.0 GiB | Three cards, for one conversation |
The live pool is a little larger per token than the arithmetic above, and for a legitimate reason: the 48 recurrent blocks also need somewhere to keep their state, and vLLM allocates that from the same arena. 10.5 GiB of pool (5.25 GiB on each card) holds 289,661 tokens, an effective 38 KiB per token. The 6 KiB gap is the fixed-size state and the engine's paging overhead — which is exactly the cost that stops growing once a conversation is under way.
Why fp8 rather than f16
The choice is not free in either direction, and the trade is not the one you would guess. fp8 KV on these cards' compute capability works only through FlashInfer, and FlashInfer cannot capture full CUDA graphs alongside speculative decoding — so the engine quietly downgrades to piecewise graphs. Measured, f16 with the Triton backend is faster:
| KV precision | Decode | Pool | Trade |
|---|---|---|---|
| fp8 — shipped | ~101 tok/s | 289,661 | Piecewise CUDA graphs, worse TTFT |
| f16 + Triton | ~112 tok/s | ~half | Full graphs, much better TTFT |
The f16 figure and the pool halving were measured before the image model took its share, on a larger arena; the ratio is what carries over, not the absolute. Roughly 11% more speed for roughly half the context. The context was chosen — an 11% faster answer is not worth halving how much of your conversation the model can still see, and long context is the thing this site exists to give away.
Why not go coarser still
The obvious next move is to halve the cache again. It does not survive contact with the measurement: quantization error in the KV cache compounds with depth, which is exactly where a long window is supposed to be earning its keep.
q8_0 and q4_0 KV formats rather than today's fp8. The
precisions differ; the direction is the finding, and it is why nothing coarser than fp8 has been
tried since.| Context | Perplexity, q8_0 | Perplexity, q4_0 | Gap |
|---|---|---|---|
| 4K | 1.2423 | 1.2429 | Within noise |
| 32K | 1.0416 | 1.0434 | ~3× wider — and widening |
A gap that triples between 4K and 32K is a gap that will be worse at 256K. The cheap cache is cheapest precisely where you would least want to pay for it.
Where the constraint moved
Because context is 4× cheaper by construction, the KV cache stopped being the binding constraint on this machine — and you can see that in the budget. Of the 19.12 GiB the language model holds on each card, 10.34 GiB is weights and only 5.25 GiB is cache. The fixed costs dominate. That is an unusual position to be in, and it has a practical consequence: the thing that finally competed for that memory was not a longer conversation but an entirely different model — the image generator, which took its share out of the cache in August. Note 001 has that ledger.
Prefix caching then compounds the win. Because a follow-up turn re-reads none of the history it already sent, 84.8% of 11.0 million lookups on this engine have been hits — so the marginal cost of turn twenty in a long conversation is close to the cost of turn one, not twenty times it.
Why this makes free possible
The expensive part of serving long conversations is memory that grows with every token. A hybrid stack caps that growth at a quarter of the dense price; fp8 halves what is left; prefix caching stops you paying for the same history twice. What remains is small enough that one owned machine — no fleet, no per-token vendor bill — serves 200-page conversations at interactive speed, with an image model sitting on the same silicon.
The architecture is the subsidy. There is no venture maths hiding under the $0; there is a fixed-size recurrent state.