Loading is a screenshot. A workflow needs latency you can live with and quality that survives the packing.
A screenshot of a loaded model is not a workflow. nvidia-smi at 22 GB tells you the weights landed. It does not tell you whether the second agent turn still has a tolerable time to first token, whether a 100k prompt still retrieves, or whether decode is fast enough that you stay in the thought.
August spent thirty-one days on whether the agent is still there after you close the terminal. The model underneath that agent has a quieter failure. The process is up. The conversation is waiting. Local inference culture treats that gap as a VRAM number. Discord asks “does it fit?” and a green memory bar answers. Local AI spent the last year celebrating packing victories that nobody would use twice. Fit is the wrong first question. The working set is the right one.
Qwen3.8-27B at Unsloth UD-Q4_K_M is 15.33 GB on disk. On a 24 GB RTX 3090 that looks like headroom. I measured the rest of the bill on August 22. The same file at 64k context with FP16 KV and MTP already sat around 21.8 GB. Pushing the context to 131,072 with FP16 KV is the OOM path. Q8 KV and a single slot made 128k fit at 22.5 GB. Same weights. Different working set. The context length printed on the model card did not make 128k free.
The working set is whatever has to be hot for the next operation. Dense weights, or the experts the router actually picked. The KV cache for the live context. Activations. In an image graph, the text encoder, the VAE, and the latents that are in flight. Those pieces do not all need the same pool at the same time, and they do not all need to be resident for the whole run. Treating them as one blob is how a “fits” screenshot becomes a forty-second prefill and a decode that feels like a demo.
An agent makes this worse, because the expensive phase is often the one people stop measuring. Prefill is prompt processing. Decode is generation. A chat demo is mostly decode. A tool-using agent edits the context after every call. If the prefix cannot be reused, you pay prefill again. On that 3090, a 99,226-token prompt took 131 seconds to prefill at 761 tok/s. An agent that does that on every turn is not local intelligence. It is a wait with a model attached.
People screenshot nvidia-smi because it is the only number the stack makes easy. llama-server reports slots and context, not whether the last tool turn reused the prefix. Ollama reports that the model is running, not the KV type that made 128k possible. Local tooling still reports capacity. Capacity is the least interesting tier once the process starts.
The obvious answers look like they dissolve the constraint. They mostly move it.
llama.cpp will offload leftover layers into system RAM. That is a real feature, and it is also a bandwidth tax. Every token that hits a CPU-resident layer pays host memory and the PCIe pipe instead of staying on the card. I already have the ugly version of that sentence on this hardware. gpt-oss-120b is about 46 GB at every GGUF quant I tried, and offload is the 5 tok/s path. The process starts. Nobody would pair it with a runtime that expects a reply.
Mixture of Experts makes the same mistake in marketing copy. “3B active” sounds like a 35B model occupies 3B of memory. Active parameters are compute per token. The expert pool still has to live somewhere. Decode may touch a small routed set. Long-prompt prefill can touch nearly every expert. Colibrì is the honest version of that idea, not the slogan. GLM-5.3-Flash is 321B total and can sit around 12 GB resident. The project’s own reference SSD still prices decode at roughly 44 seconds per token cold, because one token moves about 4.8 GB of experts. That is a systems result worth studying. It is not a daily driver. Loading is not success.
Two cards invite the same arithmetic error. A 16 GB card next to a 24 GB card is not a 40 GB GPU. Capacity can be partitioned. Transfer still happens. Apple unified memory removes the discrete VRAM wall and still has a capacity budget and a bandwidth budget. The number on the product page is not the working set, and adding two product pages does not merge the pools.
The useful frame is four tiers, not one number.
VRAM, or the GPU side of unified memory, is the hot pool. Compute lives here. If the working set for the current phase fits here, decode can be interactive. System RAM, or the host side of unified memory, is the warm pool. Weights can live here. Experts can live here. The tax is the pipe into the compute unit. PCIe is not storage. It is a clocked pipe. Every layer, expert, or KV page that crosses it costs latency you cannot recover in a better quant. NVMe is the cold pool. Treating the SSD as an expert store is a legitimate architecture. It is also why “runs in 25 GB of RAM” and “usable chat” are different sentences.
On the 3090 run, the 15.33 GB of Q4 weights and the Q8 KV lived in VRAM. RAM held the OS and llama.cpp’s host-side bookkeeping. PCIe stayed quiet because nothing was spilling. NVMe was involved once, at load. That is why 99 tok/s was possible. The gpt-oss-120b offload run inverted the map: most of the weights sat in RAM, every token crossed PCIe, and 5 tok/s was the honest number. Same machine. Different placement. Bandwidth is the part the screenshot never shows. Once the working set spills out of the hot pool, you are no longer running at GPU memory speed. You are running at whatever pipe you spilled onto. A clever placement policy can hide that on decode if the hot slice is small. Prefill, long context, and a cache miss will not let you hide it. The engine that looks magical on a short prompt is often the engine that falls over on the second agent turn.
Concurrency is part of the working set too. The 64k llama-server I ran advertised four slots. VRAM will not hold four concurrent 64k fills. One user at 128k is a different machine than four users at 8k. Desktop VRAM is also not a dedicated inference allocation. The OS, a compositor, and whatever else is using the card already took a cut. “Fits in 24 GB” always meant “fits in whatever is left.” The same hierarchy returns in image and video graphs later this month. A denoiser file that fits is still not a workflow if the VAE and the latent at your resolution do not.
The test for the next thirty days is three questions, and a recommendation has to answer all three. Fit: does the working set for this phase land in a pool that can feed the compute? Speed: is time to first token and decode tolerable for the job, including the second turn? Quality: did the packing, the KV type, and the offload preserve retrieval and answers you would actually ship?
I ran that test on the 3090. Qwen3.8-27B UD-Q4_K_M, 128k context, Q8 KV, one slot, MTP draft 5: 22,541 MiB used, 99 tok/s decode, 761 tok/s prefill on that 99k prompt, and a needle buried at the midpoint that came back as ORANGE-PLUM-917 exactly. Drop MTP and decode halves to 42 tok/s. Same file. Same card. The working set changed. Q4 KV saved about 2 GB at the same draft-2 speed. I have not treated that cache type as proven for the 100k retrieval. Memory you saved in the cache is memory you may have spent in accuracy. Unsloth UD-Q6_K is 20.47 GB on disk and will not fit the 64k-plus-MTP layout I actually ran. The quality ceiling on 24 GB is not the highest quant filename. It is the highest quant that still leaves room for the cache and the draft. Fit without the quality gate is packing.
llama.cpp is the boring baseline every exotic claim this month has to beat. Thursday is that tour: GGUF, partial offload, quantized KV, MTP, the OpenAI-compatible server, and why the first run still belongs there even when a specialized engine may eventually win. Tomorrow is the map. Where weights, KV, activations, experts, and checkpoints can live, what moves during prefill versus decode, and how a discrete GPU box and a unified-memory box draw the same hierarchy with different labels. Friday collects the claims that only proved fit and the ones that proved a usable workflow.
The rest of September is engines that move the working set on purpose. Colibrì streams routed experts from disk. FreeToken treats the whole PC as one elastic system. KTransformers puts dense work on GPU and experts where they belong. MoE-Infinity treats expert caching as a serving problem. Later weeks pick a runtime for the box you own, spend the leftover bytes on KV and speculative decoding, then watch the same hierarchy show up in image and video graphs. None of that cancels physics. Each engine chooses which tier pays.
August asked whether the agent survives the terminal. The model underneath that agent has the same problem one layer down. A runtime that cannot get a token back in time looks idle. Pick the strongest model whose working set meets the latency and quality floor of the job. The largest model that prints one token is a screenshot.
If this was useful, forward it to one engineer who needs less noise in their feed.