The exotic engine is allowed to win. It is not allowed to skip this measurement.
The exotic engine is allowed to win. It is not allowed to skip the baseline.
llama.cpp is that baseline. GGUF weights, partial GPU offload, layer split across unequal cards, quantized KV, multi-token prediction, and an OpenAI-compatible server. If a specialized stack cannot beat those numbers on the same file and the same card, it is not a better engine. It is a different placement policy with a press release.
Yesterday I annotated the box. Hot pool, warm pool, pipe, cold pool. Today is the instrument that turns that map into a command line. This is a consumer-inference tour, not a production-serving recap. vLLM and SGLang still matter in the cluster. On a 24 GB card under a desk, the first run still belongs to ggml-org/llama.cpp.
The current tagged release is v0.3.0, published August 25. This morning the repo sits at 126,851 stars, MIT, last push a few hours ago. Master is still shipping b builds; b10776 landed today. I am writing against the tag. The nightlies are the development stream, not the binary I would drop into systemd until I have measured them.
GGUF is why this engine became the default. One file. Quant in the filename. mmap at load, so “loaded” can mean the pages are reachable, not that they are resident. Hugging Face is full of them. Unsloth’s Qwen3.8-27B UD-Q4_K_M is 15.33 GB on disk. You do not convert at 2 a.m. unless you have to. The first smoke test can be llama-server -hf against a GGUF repo. The measured run is a file on local disk. Pin the file. Then change one knob at a time. The format is the product as much as the runtime is. Every more exotic engine this month either reads GGUF, wraps it, or asks you to abandon it. Abandon it after you have a number, not before.
-ngl is the hot-pool gate. all or 99 means try to keep the layers in VRAM. A smaller integer spills the rest into system RAM. That is a real feature. It is also the bandwidth tax I already paid. gpt-oss-120b is about 46 GB at every GGUF quant I tried on this box. The process starts because 30 GB of DDR4 can hold the spill. Every token then crosses PCIe 3. Decode falls to 5 tok/s. The README calls this “CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity.” Partial acceleration is the honest phrase. On an 8700K it is not a workflow. If the working set does not fit the hot pool, drop the quant or the context before you celebrate the spill.
--split-mode layer is the default when two cards are visible. Each GPU holds a contiguous slice of layers, and that layer’s KV lives on the same device. --tensor-split 3,1 is how a 24 GB card and a 16 GB card stop pretending they are equal. --split-mode tensor splits weights and KV across the cards. It is experimental. It refuses quantized KV. Auto-fit is disabled in that mode, so you size context yourself. It is unimplemented for a long list of MoE and hybrid architectures, including DeepSeek2, GLM-DSA, Nemotron-H, Mamba, and BitNet. Ampere next to Blackwell typically has no peer-to-peer path unless you opt into GGML_CUDA_P2P, which the docs warn can crash or corrupt on some boards. Two cards can partition capacity. They do not become one pool. A Thursday later this month is that argument. The first run should not start there.
-ctk and -ctv are the KV budget. Default is f16. q8_0 is what made 128k fit on the 3090. q4_0 saves more and still needs a retrieval test before you trust it. Flash attention has to be on for quantized V. --spec-type draft-mtp spends leftover VRAM on multi-token prediction. Drop it and decode halves. The working set changed. The model file did not.
Those are the knobs. The server is why they matter for an agent.
llama-server speaks OpenAI chat completions, embeddings, and tool use on /v1. Anthropic Messages too, if that is what the client already sends. Parallel slots, continuous batching, a web UI that is not a production console. The agent runtime does not need a special adapter. It needs a base URL. On this box that URL is http://127.0.0.1:9931/v1. Port 8080 is retired here. The command that won the August 22 sweep is the one still worth running:
llama-server \
-m Qwen3.8-27B-UD-Q4_K_M.gguf \
-ngl 99 -c 131072 -fa on -np 1 \
-ctk q8_0 -ctv q8_0 \
--spec-type draft-mtp --spec-draft-n-max 5 \
--host 127.0.0.1 --port 9931 -t 6Fully resident. One slot. Q8 KV. MTP draft 5. nvidia-smi showed 22,541 MiB. Decode ran at 99 tok/s. Prefill on a 99,226-token prompt ran at 761 tok/s. A needle buried at the midpoint came back as ORANGE-PLUM-917 exactly. Same file, no MTP: 42 tok/s. Same file, Q4 KV, draft 2: 79 tok/s and 2 GB back, with no 100k needle yet. Fit, speed, quality. That layout answered all three. Q4 KV has only answered the first two.
An agent makes the server the product rather than the CLI. A chat demo is mostly decode. A tool-using agent edits the context after every call. If the prefix cannot be reused, you pay prefill again. 131 seconds on that 99k prompt is the bill the screenshot never shows. -np 1 at 128k is not a concurrency brag. It is how the slot stays large enough to hold the job. The 64k process advertised four slots. VRAM will not hold four concurrent 64k fills. Desktop inference is one user who expects the second turn to come back.
The protocol is three prompts, not a screenshot. A short completion to confirm decode. A prompt that looks like your job, including a tool-shaped second turn, to confirm time to first token when the prefix should reuse. A needle at the context you actually set, to confirm the packing did not throw retrieval away. nvidia-smi after load is the cheapest number in the stack. I already had it. The expensive numbers were 99 tok/s, 761 tok/s on a 99k prefill, and ORANGE-PLUM-917 at the midpoint.
The reputation problem is that llama.cpp is treated as the slow portable option, the thing you run until the “real” engine is installed. Sometimes that is true. ExLlamaV3 will likely win on a CUDA-resident model that already fits. MLX-LM will likely win on a 128 GB Mac. Colibrì, FreeToken, and KTransformers exist because static layer offload is a blunt instrument for sparse models. They get their own days this month. They still have to beat this number, on this file, or admit they are solving a different placement problem.
The first run belongs here because the comparison is otherwise fake. A specialized engine that looks magical on a short prompt and falls over on the second agent turn is not a winner. It is an unmeasured claim. llama.cpp gives you the boring number: time to first token on turn two, decode you would wait for, a retrieval check at the context you actually set. GGUF is everywhere. The server already speaks the protocol your agent uses. CUDA, Metal, Vulkan, HIP, and CPU are all first-class enough that the same command shape works when the backend changes. Start underneath the convenience wrappers when you need the knobs. Ollama is the right first answer for most people. It is not the right instrument for this measurement.
v0.3.0 added MTP for GLM-4.5-Air and tensor-split fixes for DeepSeek 4. Master is still moving. That cadence is the other reason this is the baseline. The format and the flags stay stable enough to measure against while the model zoo keeps changing. You do not rebuild the instrument every week. You rebuild the working set.
Friday collects the claims that only proved fit and the ones that proved a usable workflow. Next week the MoE engines move the working set on purpose: disk-streamed experts, elastic CPU/GPU caches, heterogeneous scheduling. They are allowed to win.
The baseline is not the engine you settle for. It is the number the next engine has to beat.
If this was useful, forward it to one engineer who needs less noise in their feed.