Capacity is a label. Bandwidth is the machine.
A 24 GB card is not a 24 GB working set. It is a 936 GB/s pool sitting behind a pipe that is roughly sixty times slower on a PCIe 3.0 board, and the model only stays interactive while the next token’s data never leaves that pool.
Yesterday I named the working set. Today is the map of where it lives. If you cannot annotate your own machine after this, you will keep picking engines by VRAM labels, and the labels will keep lying.
The requirement that defines this class is not “the model loads.” Loading is a file copy. The requirement is that you can point at every large tensor in the run and say which pool it occupies during prefill, which pool it occupies during decode, and which pipe it crosses when those two disagree. Weights, KV cache, activation buffers, routed experts, draft tensors for speculative decoding, and any prefix or recurrent checkpoint all have a home. They do not share one. A box you cannot annotate is a box you cannot tune.
On a discrete-GPU machine the hot pool is GDDR. Compute lives there. If the working set for the current phase fits there, decode can be a conversation. On the RTX 3090 I measured on August 22, that pool is 24 GB at 936 GB/s. The OS, the compositor, and CUDA already took a cut, so “fits in 24 GB” always meant “fits in whatever is left.” Qwen3.8-27B at Unsloth UD-Q4_K_M is 15.33 GB of weights. At 128k with Q8 KV, one slot, and MTP, nvidia-smi showed 22,541 MiB. That layout stayed in the hot pool. Prefill on a 99,226-token prompt ran at 761 tok/s. Decode ran at 99 tok/s. The map was quiet. Nothing was commuting.
The warm pool is system RAM. Offloaded layers live here. Host-side bookkeeping lives here. On a MoE engine, the expert cache that did not win a VRAM slot lives here. The tax is not capacity. The tax is the pipe into the compute unit. DDR4 on that 8700K box sits in the same neighborhood as PCIe 3: tens of gigabytes per second, not hundreds. gpt-oss-120b is about 46 GB at every GGUF quant I tried. The process came up because RAM could hold the spill. Every token then crossed PCIe. 5 tok/s was the honest number. Same 3090. Different placement. Capacity looked fine. Bandwidth did the work.
The pipe is not storage. PCIe 3.0 x16 is about 16 GB/s. PCIe 4.0 x16 is about 32. PCIe 5.0 x16 is about 64. Those are peak line rates, not a promise that your motherboard wired the slot that way, and not a promise that two consumer cards of different generations can talk to each other without bouncing through host memory. llama.cpp’s default --split-mode layer puts each layer’s weights and that layer’s KV on the same device, which keeps the per-token hop to a layer boundary. --split-mode tensor splits both weights and KV across cards. It is experimental, it refuses quantized KV, and it is unimplemented for a long list of MoE and hybrid architectures. Ampere next to Blackwell typically has no peer-to-peer path. The hop goes through RAM. Two cards can partition capacity. They do not become one pool.
The cold pool is NVMe. The GGUF file lives here first. llama.cpp’s default load is a memory map, so “loaded” can mean the pages are reachable, not that they are resident. A first token that faults the map pays disk. A MoE engine that treats the SSD as an expert store pays disk on every cache miss. Colibrì’s own GLM-5.3-Flash numbers still make the point: roughly 12 GB resident, and a cold decode that can sit around 44 seconds per token because one token moves about 4.8 GB of experts. Gen4 NVMe sequential is in the 7 GB/s neighborhood. That is a legitimate architecture. It is also why “runs in 25 GB of RAM” and “usable chat” are different sentences.
Prefill and decode do not draw the same map. Prefill is prompt processing. It wants the weights, or nearly every expert on a long prompt, plus activation buffers, plus a KV cache that grows with every token you accept. Decode is generation. It streams the hot weights once per token, or the routed experts, and it reads and writes KV. A chat demo is mostly decode. A tool-using agent edits the context after every call. If the prefix cannot be reused, you pay prefill again. On that 3090, 131 seconds of prefill on a 99k prompt is the bill the screenshot never shows. Checkpoints, prefix cache, and recurrent state exist to keep you from paying it twice. They occupy the hot pool too. Budget them or they evict the model you thought you had room for.
Annotate the 3090 box as a worked example and the hierarchy stops being abstract. Hot pool: 24 GB GDDR6X at 936 GB/s, maybe 22 GB usable after the desktop took its cut. Warm pool: 30 GB of DDR4, of which the OS already owns a slice, on a six-core 8700K that is not a spill engine. Pipe: PCIe 3.0 x16, about 16 GB/s, the slot the card actually occupies. Cold pool: local NVMe, used once at load when the working set is resident, used on every expert miss when it is not. On the Qwen run, weights and Q8 KV sat in the hot pool, RAM held the OS and llama.cpp’s host bookkeeping, PCIe stayed quiet, NVMe was involved at mmap. On the gpt-oss-120b run the same four lines inverted: most of the weights sat in RAM, every token crossed the pipe, decode fell to 5 tok/s. I would rather have the 27B I can answer with than the 120B I have to babysit. The map told me that before the model card did.
The unified-memory variant is the same hierarchy with different labels. There is no discrete VRAM wall. There is still a capacity budget and a bandwidth budget. An M4 Max at the full 16-core, 40-core bin is 546 GB/s of unified memory, 410 GB/s on the binned 32-core part, up to 128 GB of addressable capacity. Strix Halo (Ryzen AI Max+ 395) is 256 GB/s on paper and about 215 GB/s measured under mixed load, with the same 64 or 128 GB class of LPDDR5X. Subtract 8 to 10 GB for the OS and the runtime before you start packing weights. You do not “offload to CPU” on these boxes in the discrete-GPU sense. CPU and GPU share the pool. That is the advantage: a 70B at a usable quant can sit where a 24 GB card would spill. The cost is bandwidth. 215 to 546 GB/s is faster than DDR4 and slower than GDDR6X. A model that fits on a 128 GB Mac and crawls is not a win over a 24 GB card that holds a smaller working set at 99 tok/s. Capacity bought the fit. Bandwidth still prices the token.
Where this flexes is the parts of the working set that do not have to be hot at the same time. Dense weights want the hot pool for the whole run, or you accept the pipe. MoE experts can live warm or cold if the router is right and the prompt is short; long-prompt prefill will not let you hide that. Hybrid attention, Gated DeltaNet plus sparse full attention in Qwen3.6 and Qwen3.8, is why a 35B-A3B can hold 128k on 24 GB when a dense 32B dies on KV. Speculative decoding and MTP add a draft that occupies the hot pool in exchange for decode. Image and video graphs make the same trade in sequence: text encoder, denoiser, VAE, latents. The graph is a memory scheduler. Two unmatched GPUs flex by role, not by addition. A 24 GB card running the long-context model and a 16 GB card running a short grunt model is two annotated boxes. --split-mode tensor on a model that already fits one card is how you buy the pipe you did not need.
What it costs is honesty about the first number you write down. nvidia-smi after load is the cheapest number in the stack. The expensive numbers are time to first token on the second turn, decode you would actually wait for, and a quality gate that still retrieves. I have the 3090 version of that sentence. Fully resident Qwen at 99 tok/s. Offloaded gpt-oss-120b at 5. Same card. The difference was placement, not intelligence. A 128 GB unified box that holds a 235B MoE at a low quant still has to move the active slice at LPDDR speed. Skip the annotation and you will spend the month arguing about engines that are all obeying the same map.
The starting point is a card, not a purchase. Write four numbers: usable hot-pool capacity after the OS cut, host RAM after the OS cut, the PCIe generation and lane count of the slot the GPU actually occupies (or “unified” if there is no slot), and sequential NVMe read. Then write the working set on top of it: weight file size, KV at the context you will actually set, a 1 to 2 GB compute reserve, draft tensors if you will use them, vision projector if you will load it. If the hot-pool line still has room, the first run is fully resident. If it does not, you are choosing which tier pays. Measure time to first token and decode on a prompt that looks like your job, including a second turn, before you keep the layout. llama.cpp is the boring instrument for that first run. Thursday is that tour.
The rest of the month is engines that move the working set on purpose. They do not cancel the map. They pick a tier and send you the bill. Annotate the box you already own. The largest model that prints one token is still a screenshot. The useful model is the one whose next token never leaves a pool you can feed.
If this was useful, forward it to one engineer who needs less noise in their feed.