Context length, cache precision, and concurrency belong in the budget before the weights do.
A model that loads at 8K has made no promise about surviving a 128K conversation. The weights can remain exactly where you put them while the memory needed to use those weights pushes the process off the card. Yesterday’s quantization discussion ended with a reservation: the space left after loading still has work to do. The KV cache is a large part of that work, and treating it as leftover overhead is how a successful download becomes a failed deployment.
The cache holds attention keys and values from tokens the model has already processed. Keeping them avoids recomputing those projections for every generated token. In ordinary full attention, the history grows as the conversation grows, layer by layer. The cache therefore belongs in the workload budget from the beginning. Hugging Face’s explanation describes both the stored tensors and their sequence-length dimension. A small weight file says nothing about how large that dimension will become.
You can make the cost concrete without a benchmark. Imagine a full-attention model with 32 layers, eight key-value heads per layer, and 128 values per head for each of K and V. At two bytes per value, each cached token needs 131,072 bytes across those layers. That is one gibibyte at 8,192 tokens and sixteen gibibytes at 131,072 tokens, for one sequence, before allocation overhead. This is illustrative arithmetic from the tensor dimensions, not a measurement of a particular checkpoint. The sixteenfold increase is enough to explain why the same weights can fit in the morning and fail against a longer document that afternoon.
Parameter count does not supply those dimensions. Grouped-query attention shares key-value heads across query heads, so counting every query head would overstate the cache. Sliding-window layers can stop growing once their window fills. A static cache can reserve its capacity before the conversation reaches it, while a dynamic cache grows with use. These distinctions change both the estimate and when an out-of-memory failure appears. The Transformers cache guide makes those allocation differences explicit.
Qwen3.8-27B is a useful correction to the simple example. Its model card describes 64 layers arranged as sixteen groups of three Gated DeltaNet layers followed by one gated-attention layer. The full-attention layers have four KV heads with a head dimension of 256. Applying ordinary attention-cache arithmetic to those sixteen layers gives eight gibibytes at 131,072 tokens with sixteen-bit keys and values. That calculation excludes recurrent state, weights, compute buffers, and runtime overhead. Multiplying the same growing-cache formula across all 64 layers would describe the wrong model.
Hybrid architecture reduces one bill and leaves others on the table. Gated DeltaNet maintains recurrent state rather than a full token history in every layer. That state still occupies memory, and its precision deserves separate treatment. The August DAMP paper studies precisely this problem: its authors report serious reasoning degradation when they uniformly compress recurrent states, then propose a selective mixed-precision approach. Those are research results on their evaluated models, not proof that a cache flag safely compresses every state tensor in yours.
The August 22 setup record behind this month’s 3090 example documents Qwen3.8-27B with UD-Q4_K_M weights, a 131,072-token context, Q8 keys and values, and one server slot on the 24GB card. It identifies sixteen-bit cache at that context as the configuration that would not fit. The original raw benchmark files were unavailable for this article’s review, so this is a historical configuration record, not a newly verified benchmark or a controlled proof that each changed setting was necessary. It explains the intended trade: buy context by reducing cache storage and limiting concurrency. The context label did not buy those bytes.
Even Q8 needs a more careful estimate than dividing everything by two. In llama.cpp’s Q8_0 block layout, 32 signed eight-bit values share a sixteen-bit scale. That is 34 bytes per block, rather than 32. Applied to the full-attention portion calculated above, the storage becomes roughly 4.25 gibibytes instead of eight, before runtime padding and other allocations. Weight quantization and cache quantization are separate decisions; a Q4 weight filename does not establish either key or value precision.
Concurrency is the other decision that disappears behind a comfortable default. Four unrelated conversations need storage for four histories, but four advertised slots do not automatically mean four full advertised context windows. Servers can divide or share a pool. The llama.cpp server documentation exposes separate controls for context, parallel slots, and K/V types. Read the startup allocation and per-slot limits for the build you run. Do not multiply capacity twice, and do not promise four users the space that only one can occupy.
A one-slot configuration can be the right answer for a personal coding assistant. It can also move the problem from memory into a queue when another job arrives. A background extraction task and an interactive conversation become competitors even if both fit separately. I would rather specify that constraint in the service contract than discover it while somebody waits. The useful capacity number is the context each admitted request can actually use, with room left for its answer.
The strongest objection is reasonable: compress the cache harder and keep both the history and the concurrency. Low-bit methods can work. KIVI reports near-baseline quality for its evaluated models with an asymmetric two-bit method that treats keys and values differently. That is evidence for a particular method and evaluation. It does not make an arbitrary four-bit runtime setting equivalent, or establish that a long tool-use trace will survive because a short completion did.
I would make cache precision earn its place with the task that motivated the longer window. Put a required fact early in a long input, add distracting material, and require the answer to use that fact correctly. Move it to other positions. Include exact tool arguments and a sequence of decisions if the service runs an agent. Compare with the higher-precision cache while holding weights, prompts, runtime, and sampling settings constant. A single retrieved phrase is a useful gate; it is too narrow to certify the whole workflow.
Fit and speed need their own checks alongside that quality gate. Fill the intended context, generate a meaningful continuation, and repeat at the intended concurrency. Record peak memory and time to first token as well as generation speed. A health endpoint establishes that the server started. It cannot establish how the server behaves when the expensive request arrives. If the configuration survives only after reducing the context or queuing additional users, write down that operating limit before calling the service usable.
Runtime changes can invalidate a tidy comparison too. The current llama.cpp v0.4.1 release, published September 14, includes fixes to recurrent-model normalization and to speculative-decoding cache allocation for named model families. Those fixes do not prove an older Qwen run was wrong. They do establish why the runtime version belongs beside cache precision in the record. An August setup note and a September binary are different experimental conditions.
My starting decision would be the longest context the job needs, the concurrency the service owes its users, and the quality floor it must retain. Reserve memory for that combination, plus the runtime’s other allocations, before choosing how much to spend on weights. If the budget fails, shorten the context, admit fewer requests, choose a smaller model, or evaluate a different cache format deliberately. Each option changes what the service can promise.
The next gigabyte can buy better weights or more history. Decide which one the job needs before the allocator makes the decision for you.
If this was useful, forward it to one engineer who needs less noise in their feed.