What a long conversation actually costs

TL;DRThe model accepts 262,144 tokens in a single request — a quarter of a million, and what the API will take from you today. Behind it sits one shared pool of 289,661 tokens, which is what two 24 GB cards can hold live once the weights, the activations and an image model have taken their share. Because that pool is shared rather than per-user, the web app hands it out as a ladder: one 64K slice, three 32K slices, and eighteen short ones. What makes any of this affordable is not more VRAM but an fp8 KV cache and an architecture where only a quarter of the blocks pay per-token memory at all.

The two numbers people confuse

Almost every argument about context length collapses two different quantities into one word. They are worth separating before anything else, because the interesting engineering is entirely in the gap between them.

Per request

262,144The longest single prompt the engine will accept. A hard model limit, and the number the API is bound by.

Live, everyone at once

289,661The whole KV pool. Every conversation currently held open is drawing from this one arena.

A rented endpoint only ever quotes you the first. It can, because someone else owns the second and prices it into your bill. Running the machine yourself means the second number is yours to spend, and spending it is the entire subject of this note: the pool is barely larger than one maximum-length request, so a single user asking for everything would be the whole service.

What the rig actually is

Two RTX 3090s, 24 GB each, joined by NVLink and run tensor-parallel so weights and cache split across both. vLLM serves Qwen3.8-27B-AWQ-INT4 — INT4 weights, an fp8 KV cache. The table is the running configuration, not a target:

Read off the running engine, 16 August 2026. Prefix caching is measurably doing its job: 84.8% of 11.0 million lookups since the last restart were hits.

Why a window this size fits at all

Because most of the model never pays for it. Qwen3.8 is a hybrid: only 16 of its 64 blocks run full attention and carry a cache that grows with the conversation. The other 48 hold a recurrent state of fixed size — a 200-page chat costs them exactly what a one-line chat does. Note 003 works that arithmetic out in full.

The fp8 cache then halves what those 16 blocks do pay, and tensor-parallelism splits the result across two cards. None of the three is clever alone; together they are the difference between a context window you can advertise and one you can afford to give away.

Then an image model moved in

On 15 August 2026 Aelius gained image generation and editing, served by its own engine on the second card. It is a real feature and it was not free. The engine's cache had to shrink to make room, and the pool fell by almost half in a single afternoon:

Two things made that trade acceptable. The first is that the image model is barely resident at all: its diffusion weights and its text encoder live in system RAM and are streamed onto the card one graph segment at a time, with the encoder never touching the GPU. Peak VRAM for a plain generation is 2.70 GiB rather than 7.10, at a cost of about 11 seconds per image.

The second is that an edit, not a generation, is the worst case — the reference image's VAE encode peaks at 4.11 GiB, and notably does not respond to the server's own VRAM budget at all. Budgets of 2, 2.5 and 3 GiB all peak identically, so the pool had to be sized against the measurement rather than against the flag.

How the pool is handed out

289,661 tokens shared by everyone at once is not very many, and the shape of real demand here is lopsided: roughly 38 daily actives whose conversations are overwhelmingly short, with an occasional long one. A flat carve-up at that pool size bought exactly two concurrent users, which is the wrong answer for both groups. So the pool is a ladder — a few big slices and a lot of small ones — and sessions are promoted upward as bigger slices free:

Past the ladder, a session is not refused; it is given a short-question budget and an input cap, so it can still ask brief things while the big slices are held. The cap on those is derived from what the arena has left rather than picked — an early draft hardcoded twenty short sessions, which oversubscribed the pool by about 170% and handed out budgets the cache could not honour. That failure lands as an HTTP 500 mid-answer instead of as an honest wait, which is the wrong direction to be wrong in.

Where the ceiling came from

None of this started here. Through July the whole service ran on one 3090 under llama.cpp, and the serving context was stuck at 64K — anything higher silently halved decode speed. That investigation is worth keeping, because the wall turned out not to be memory.

The wall at 64K

Raising context past 64K produced a distinctive failure: VRAM usage went down while decode speed halved. That signature matters — it is not memory exhaustion, it is llama.cpp's auto-fit deciding the layout does not fit and evicting layer 0 to the CPU, which on this hybrid architecture also disables the fused Gated-DeltaNet kernel:

• W resolve_fused_ops: layer 0 is assigned to device CPU
• W resolve_fused_ops: fused Gated Delta Net (chunked) set to disabled

Losing that one fused kernel roughly halves generation on a model where 48 of 64 blocks are Gated-DeltaNet blocks.

The measurement trap

A load-only test lies. llama.cpp allocates KV at load, but speculative-decoding draft-verify and compute buffers grow during generation. Past ~80K they exceeded 24 GB and the display driver silently paged VRAM to host RAM — no warning, VRAM apparently fine, decode halved anyway. Every number below is therefore a decode measurement (timings.predicted_per_second), never a "model loaded" check.

The find: the margin was the wall

llama.cpp's fit pass defaults to reserving 1,024 MiB per device (--fit-target). When its conservative estimate cannot keep that margin, it evicts layer 0 rather than dip into it. At 80K the fit pass evicted — while the same configuration, once actually allowed to load, settled at 23.5 GB with 1.1 GB still free. The margin, not the memory, was the ceiling. Lowering the target changes only fit's decision threshold, not the real allocation:

Mean of prose + code decode, 2 slots, MTP on, vision on GPU, q8_0 KV cache, one card. 84K was a hard ceiling on that configuration: no combination of micro-batch size, draft-KV quantization, or a smaller margin bought the next 2K.

Validated under real load

Identical deep-context speed (so 84K was not silently host-paging), plus concurrency the old pool physically rejected — 2 × 34K does not fit in 64K.

What didn't work

• Smaller micro-batches (-ub 256 -b 1024) — does not prevent eviction at 86K+, only makes the evicted state less bad. No reason to carry it at 84K.
• Quantizing the draft KV (-ctkd/-ctvd q8_0) — saves the memory it promises and still evicts, while costing draft quality. At 2 slots these flags actually cause the eviction they are meant to avoid.
• Pinning layers (-ngl 99) — aborts at startup on that build; letting auto-fit place layers, with a sane margin, was strictly better.

What the second card bought

84K on one card was a hard stop, and every fix that would have gone further cost something real — a worse draft, a smaller batch, a coarser cache. Adding a second 3090 and moving to vLLM removed the question instead of answering it: the pool is shared across both cards, the cache is fp8 rather than q8_0, and one request may now run to 262,144 tokens, roughly 3.1× what the single-card configuration reached at its absolute ceiling.

The July finding still holds, and it is why the box was worth building on at all: the limit was never the silicon. It was a default nobody had read. The current limits are different in kind — they are a budget, written down above, that anyone can check.