Notes, published with their data.

The machine the notes are about

Every request — a chat message, an API call, an image — enters the same box. One web app fronts two engines: vLLM serving the language model across the two 3090s, and a separate image engine with a card of its own. Nothing here is rented, and nothing leaves the building.

Until August the image model was a lodger on the second 3090, and it charged rent: the language model's cache had to shrink from 530,081 tokens to 289,661 to make room for it. A third card ended that. The image model moved onto an RTX 3060 of its own, the KV pool went back up to 548,909, and image generation got faster rather than slower — the 3060 has the headroom to hold the diffusion weights resident instead of streaming them.

What it is running, exactly

These are read off the running service, not from a plan. The two that matter most are the first two: the model will accept a quarter of a million tokens in one request, and the pool those requests are served from is shared between everyone holding a conversation at once.

Context per request

262,144Tokens. The model's ceiling, and what the API will take.

Live KV pool

548,909Tokens resident across every conversation at once. The engine measured 586,974 at boot; the app is given the smaller figure on purpose.

Weights

INT4 · g3227B parameters, compressed-tensors, tensor-parallel across the two 3090s. The speculative head is INT8 at g128.

KV precision

fp8Half the cache per token, at no measured quality cost.

Decode, short prompt

89.8tok/s, idle engine. n=3, spread 0.2 tok/s. 2.76 tokens land per forward pass.

Decode at 80K

129.7tok/s, prompt already cached. n=3, spread 0.4. Faster than the short prompt, and that is not a typo — see below.

Concurrency at full context

2.24×Requests of 262,144 tokens that fit in the pool at once. Ordinary conversations are far smaller.

Image, 1 MP

19.1 sFour steps, distilled. n=3, identical to 0.1 s. Was 31.1 s when the image model shared a 3090.

Measured 26 Aug 2026 on the production engine while it was idle, with the engine's own running-request counter checked before and after to confirm no other user's traffic had landed inside the run. The pool and per-card figures are read from the engine's startup profile in the system journal, not estimated.

The number that is easy to publish and wrong

Decode at 80K tokens came out faster than decode on an eight-token prompt. That is real, repeatable, and almost meaningless without the number beside it: speculative acceptance was 61.4% on the long prompt and roughly 45% on the short one. This machine is bound by how many drafted tokens survive, not by how much cache it has to read.

The first attempt at this measurement built its long prompt by repeating one sentence 7,800 times. The model found that trivially predictable, acceptance went to 5.77 of 6 drafted tokens, and the result was 166 tok/s — a figure that says nothing about the hardware and everything about the prompt. It is not published above because it is not a fact about this box. The 80K figure uses varied prose instead, and even that is more predictable than a real conversation, so treat it as an upper bound. A tok/s number without its acceptance rate is not a measurement here.

The budget everything is bounded by

There are 23.56 GiB usable on each 3090 and no more. Every design decision on this site — how long a conversation may get, how many people may hold one at once, whether an image model can exist at all — is a claim on the bars below. For two weeks in August the image model was a claim on them too, and the language model's cache paid: the pool fell from 530,081 tokens to 289,661. Buying a third card was cheaper than living with that, and the pool is now 548,909 — higher than before the image model ever arrived, because the 3090s no longer have to hold anything back for it.

On its own card the image model can finally be lazy about none of it. The diffusion weights and the VAE are now resident on the 3060; only the text encoder's weights still live in system RAM, because they are used once per image and holding them would buy nothing. When the model was a lodger on a 3090 the diffusion weights had to stream in one graph segment at a time, and that cost about 11 seconds per image. Removing the streaming is most of why a 1 MP image went from 31.1 s to 19.1 s — on a card with less than half the memory bandwidth of the one it left.

The fallback is still there and still wired: if the 3060 is not on the bus at boot, the image server prints a warning and lays itself back out across the 3090s. That path is worth keeping, but it is not free — the KV pool is sized for clear cards, so vLLM has to be restarted too or it will run out of memory under load.

The notes

How these were measured

Confident-sounding conclusions in this project have been wrong before, and always for the same reason: two runs from different machine states compared as though they were controlled. The rules below are what came out of that.