The placement decision has to survive the tool call, not only the first answer.
The best engine for your first answer can be the wrong engine for your second turn. An oversized mixture-of-experts model might generate smoothly after its weights and caches settle, then spend the next request rebuilding context that the agent edited. Choosing a runtime from its warm token rate misses the part of the workflow where you are waiting for it to think again.
This week’s tool assessments approached that problem from different directions. Following one workload through llama.cpp, Colibrì, KTransformers, and FreeToken puts their placement decisions under the same constraint: the agent has to finish the next turn. This is a reference architecture, not a benchmark leaderboard. I have not run a matched four-engine comparison, and the documentation cannot supply a universal winner.
Use DeepSeek-V4-Flash-0731 as the common model target. Imagine a Linux workstation with an RTX 5090, 256GB of system RAM, and local NVMe storage. Those are illustrative design inputs, not a certified configuration. The workload is one coding agent: a substantial initial prompt, a generated tool call, a test result appended to the conversation, then another answer. A separate variant edits earlier context before that second request.
The model name alone does not make those runs equivalent. llama.cpp uses GGUF; the other paths can read supported safetensors checkpoints, with backend-specific packing and precision choices. Keep the source checkpoint revision fixed, record every conversion, and reject comparisons that silently substitute a pruned model or a lower-quality quantization. DeepSeek also supplies a dedicated message encoder rather than a Jinja chat template. Matching the rendered prompt and reasoning settings belongs beside matching the weights.
Start the clock before the endpoint says it is ready. Cold start includes reading weights, allocating state, preparing kernels, and any graph capture the selected path performs. First-request prefill can add work that startup deferred. Restarting a process does not necessarily empty the operating system’s file cache, so label that run process-cold rather than storage-cold. Otherwise, the second engine tested may inherit a storage advantage the first one paid for.
The llama.cpp server controls expose a useful baseline: place layers on GPU, keep MoE weights on CPU, or override individual tensor placement. In this configuration, CPU-resident expert weights support CPU computation rather than automatically moving wholesale to the GPU whenever routing selects them. The placement is mostly a startup decision. Leave room for context state and compute buffers instead of using the entire card as a weight shelf.
Its first prefill then follows that chosen split. CPU expert execution can constrain prompt processing even when attention runs on GPU; decode may favor a different balance because each step touches a smaller routed set. This baseline is not a claim that llama.cpp lacks caching or newer storage features. Version 0.4.0, released September 4, adds on-demand tensor reading and per-slot context limits. The comparison here isolates configured placement, not every experimental path in the repository.
Colibrì changes the capacity boundary. Its DeepSeek engine keeps canonical state in host memory, streams routed experts from storage, and grows its RAM expert cache within a planner budget. An optional CUDA tier accelerates supported stages. On our large-memory example, many experts may remain cached; constrain available RAM and disk traffic becomes part of inference rather than merely startup. The experiment must record that budget instead of treating all Colibrì runs as equally disk-bound.
The first long prompt is where streaming earns or loses its place. Across many prompt tokens, routing can touch a broad union of experts, forcing reads that a short greeting never requested. During decode, recurring experts can hit the cache, while misses still wait on storage and staging. A fast NVMe label cannot predict that wait without the access pattern, filesystem, and cache state. Colibrì’s documented Windows and WSL results explicitly involve different storage paths; they are not a clean operating-system speed contest.
The version matters here too. Colibrì 1.10.2, released September 6, fixes a memory guard that counted reclaimable page cache and could evict experts unnecessarily. It also fixes cancellation during serving. Neither change grants the machine more bandwidth, but both affect whether an experiment measures the intended policy. I would choose this path when the model exceeds affordable RAM and the task can tolerate streaming, not assume that proving fit establishes interactive speed.
KTransformers makes host RAM an execution resource rather than only an expert staging area. Its current DeepSeek recipe splits routed experts between CPU kernels and GPU execution through KT-Kernel and the project’s SGLang fork. An expert computed on CPU avoids transferring its full weights for that use, although activations, results, and synchronization still cross the boundary. Memory-channel bandwidth and CPU kernel support therefore belong in the machine description alongside VRAM.
Prefill need not use the same policy as decode. The recipe documents layerwise GPU prefill, including allocations deferred until the first qualifying request. The scheduling guide also permits expert redistribution after sufficiently long prefill. Calling KTransformers permanently static would erase that behavior. Its attraction is strongest when the host can hold the expert pool and compute against it efficiently; buying a server platform to make that true is a different cost decision from reusing a gaming PC.
FreeToken asks which misses should move and which should compute locally at each step. Its current strategy documentation distinguishes offload, CPU, and hybrid execution. Offload fetches missing experts into a GPU cache. Hybrid overlaps some transfers with CPU computation of other misses, using a machine-specific bandwidth profile to guide the split. The automatic setting does not mean hybrid everywhere: it selects offload unless a suitable calibration recommends hybrid.
That gives FreeToken two different jobs during our first request. Double-buffered prefill overlaps weight movement with prompt computation; decode uses the expert cache and selected miss policy. Its documented live memory controls can also resize expert and context-cache pools without restarting the engine. More context space can leave fewer resident experts, so a longer conversation changes the placement budget. These are current-source capabilities; the latest tagged engine release remains v0.1.2, while September’s main branch has changed configuration names and quantization internals.
The tool result now arrives. An append-only continuation may reuse an unchanged cached prefix, provided the engine retained compatible state and the request reaches it correctly. Editing earlier context is different: computation after the first changed token may need to run again. Warm expert weights do not make that old context state valid. The engine can avoid fetching the same weights while still paying to process thousands of tokens again.
This is where feature lists become dangerous. KTransformers’ cited native DeepSeek launch example explicitly disables its radix prefix cache; copying that command does not establish prefix reuse merely because the serving framework supports caching elsewhere. FreeToken documents semantic checkpoints for agentic edits, but that is not permission to reuse state across arbitrary changes. Colibrì’s DeepSeek path documents one serving slot and greedy generation. Each engine needs a second-turn correctness and latency check using the exact client behavior, not a generic statement that it supports agents.
I would keep the starting implementation small: one local endpoint, one request at a time, bounded context, and speculation disabled. Save the rendered requests and returned tool arguments, check a coding task with executable tests, then measure startup, first prefill, decode, and both second-turn variants separately. Record peak RAM and VRAM, storage reads, transfer activity, and reused prompt tokens where available. PCIe generation and negotiated link width matter when misses travel to the GPU; populated memory channels matter when they compute on CPU.
My default is llama.cpp if its supported configuration already clears the task’s quality and latency floor. If host capacity is the blocker, Colibrì makes storage-backed execution worth testing. If RAM holds the model but CPU/GPU balance limits useful speed, compare KTransformers with calibrated FreeToken on that same workload. Include installation time, duplicate weight storage, dependency maintenance, and failed-task reruns in the cost. None of these engines earns a recommendation from fit alone.
Before replacing the endpoint, send the tool result back through it. The next purchase should solve the wait you measured on that turn.
If this was useful, forward it to one engineer who needs less noise in their feed.