A consumer GPU does not make the server underneath it a consumer machine.
The most misleading part of a KTransformers benchmark can be the GPU model. An RTX 4090 sounds like the card in a gaming desktop. Put it beside two Xeons and hundreds of gigabytes of memory, and you are looking at a different inference machine. The CPU is doing model work, not waiting for the graphics card to finish.
That distinction gives KTransformers its place in this week’s MoE arc. Colibrì made the SSD an expert store. FreeToken treated expert placement as a bandwidth decision that can change during execution. KTransformers puts optimized CPU expert computation alongside GPU execution, with explicit control over which experts run where. It deserves a look when the model exceeds VRAM, but enough host memory can hold the weights and feed useful computation.
I have not benchmarked KTransformers on my own hardware. This is an assessment of its current documentation, distribution packages, and reported results, checked September 8. The latest GitHub release remains v0.7.0, published August 17. PyPI has moved to 0.7.0.post3, uploaded August 27. The tagged release emphasizes fine-tuning, including native FP8 LoRA and broader AVX512 CPU support; those training improvements should not become inference speed claims by proximity.
The inference path now lives in KT-Kernel, paired with the project’s SGLang fork. The original integrated framework sits in an archive directory. That matters if your search results land on an old installation guide: the project name stayed the same while the recommended components changed underneath it.
The architectural split follows the workload. Attention and other dense operations stay on the GPU, where the engine can use GPU kernels and tensor parallelism. Routed experts can execute on CPU or GPU according to placement. A CPU-resident expert does not need to cross PCIe as a complete weight tensor every time it is selected. The CPU computes against local weights, although activations, results, and synchronization still have to cross the boundary.
This is why host memory bandwidth belongs near the top of the hardware requirements. During sparse, low-batch decode, expert computation can spend much of its time reading weights. More CPU cores help until the memory channels cannot supply them. Adding threads beyond that point cannot manufacture bandwidth, and remote memory on another socket introduces another path to manage. KTransformers exposes physical-core thread counts and NUMA-aware thread pools because locality affects the work, not because server tuning needs more knobs.
Prefill changes the balance. A long prompt can activate much of the expert pool across its tokens, creating more opportunity for batched GPU computation but also more weight movement. The native backend offers layerwise GPU prefill above a configurable token threshold. The useful split is therefore not “experts always belong on CPU.” It depends on whether the engine is processing a prompt or producing the next token, and which backend supports that path.
Yesterday’s shorthand about KTransformers pinning experts at prefill needs qualification. It is not limited to a permanently frozen placement map. The current expert scheduling guide documents uniform and frequency-based placement, along with dynamic expert updates triggered by sufficiently long prefill. That differs from the per-decode-step decision described in FreeToken’s paper, but calling one engine dynamic and the other static hides the decision that matters: when the placement can change.
The project’s own scheduling table shows why that distinction is worth testing. On Qwen3-Next-80B-A3B-Instruct-FP8, with four RTX 4090s, a Xeon Gold 6454S, and ShareGPT traffic, the reported throughput at a 10 percent GPU expert ratio is 56.57 tokens per second for uniform placement and 70.22 with dynamic updates. At 90 percent residency, frequency placement reports 107.15 while dynamic updates report 95.04. Dynamic redistribution does not win every row, even in the project’s benchmark.
Those are project-reported throughput figures, not my measurements or a single-user decode promise. The tutorial does not specify enough prompt-length, output-length, and concurrency detail beside that table to turn it into a reproducible desktop forecast. It demonstrates that placement policy affects the result. It does not establish which policy wins on your coding session, especially after a tool response changes the prompt.
Model support needs the same precision. The current documentation includes DeepSeek-V4-Flash, Qwen MoE variants, GLM-5.2, and MiniMax-M3. Newer GLM-5.3-flash support, announced August 26, reads native FP8 weights and documents text, image, and video inputs. That recipe lists roughly 306 GiB of model weights and asks for at least 350 GB of available system memory. Its specified GPU path is RTX 40/50-series hardware with an AVX-512 FP8 CPU kernel.
A 24GB card is not the full bill. The GLM guide also distinguishes the advertised million-token context from its validated 501,025-token launch examples. Its layerwise-prefill implementation normalizes the resident GPU expert count to zero, even though one example passes a nonzero setting. Video uses a different prefill path. The model guide, rather than the generic feature list, has to be the authority for the configuration you intend to run.
Installation is another place where the generic guide and the current artifact disagree. The kernel README describes wheels for Python 3.10 through 3.12 and manylinux_2_17. The current KT-Kernel 0.7.0.post2 upload contains a CPython 3.12 wheel tagged manylinux_2_35. The latest KTransformers package pins that kernel and, through its SGLang extra, sglang-kt==0.7.0.post3. I would check the actual wheel against the target interpreter and Linux environment before treating a one-line install as a deployment plan.
The fork distinction is explicit: the inference guide calls for sglang-kt, not an arbitrary upstream sglang installation. Backend choice also controls weight preparation. AMX INT4/INT8 paths use converted CPU expert weights; the LLAMAFILE backend reads GGUF on the CPU side; native-precision paths have their own instruction-set requirements. Compatibility is a model, weight format, CPU kernel, GPU path, and package combination. A successful Python import verifies only a small part of it.
Open reports expose where that combination gets narrow. Issue #2172 describes an Ampere capability gate excluding dual A10s from the DeepSeek-V4-Flash MXFP4 layerwise-prefill pipeline. It remains a reported configuration limitation, not proof that all Ampere inference fails. A broad “Ampere supported” statement cannot tell a 3090 owner whether a particular model’s faster prefill path works.
I would evaluate KTransformers first on an existing large-memory Linux workstation or server, with a supported CPU backend and a model whose exact recipe matches the GPUs. Fast, well-populated memory channels make the CPU a credible execution resource. Buying hundreds of gigabytes of RAM, a different motherboard, and server-class cooling to rescue one oversized model is a separate decision. Compare that cost with using a smaller model that already meets the task’s quality floor.
On a gaming desktop where a useful model fits in VRAM, I would keep llama.cpp as the starting point. On a supported NVIDIA machine with an oversized MoE and enough RAM, FreeToken remains a candidate for adaptive placement. Neither recommendation establishes a speed ranking against KTransformers without a matched run. The setup cost is justified only if the resulting endpoint does useful work faster, or enables a quality level the simpler engine cannot deliver within the same latency budget.
My acceptance test would include a cold prompt, warm decode, and a second turn after a substantial tool result. Record the exact weights, quantization, package versions, CPU topology, context budget, and concurrency. Check an answer with a known result before interpreting throughput, then measure time to first token separately from generation speed. A server that starts has established fit; it has not established speed or preserved quality.
KTransformers makes the host platform part of the inference engine. Before you buy another GPU, find out whether the next token is waiting on the card or on the memory channels feeding its experts.
If this was useful, forward it to one engineer who needs less noise in their feed.