Static offload is a load-time guess. A miss is a bandwidth decision.
llama.cpp decides where the experts live when the file loads. FreeToken decides again on every decode step.
Yesterday Colibrì treated the SSD as the expert store. Dense slice hot, routed experts mmap’d, a miss costing disk. FreeToken’s warehouse is host RAM. The GPU is a cache that can be resized without a reload. The CPU is a miss path that can beat a blocking PCIe fill on the right box. Same month, same sparse-pool problem, a different opinion about which pipe the experts should cross.
The paper is arXiv 2608.16157, dated August 17. Yang, Fan, Pan, Xi, Wang, Sun, Keutzer, Han, Zaharia, Xu, Stoica. Berkeley and UT Austin. The tagged engine is v0.1.2, published August 19, matching PyPI freetoken 0.1.2 the same day. That is the only Git tag. Main has moved 28 commits past it. Last push on this writing is September 4: nightly wheels, per-platform engine manifests, GLM-5.3-Flash on the family list. The desktop app lives in a second repo, FreeToken-Web, currently v0.2.0-beta.17. Apache-2.0. About 11,900 stars this morning, up from roughly 8,300 two weeks ago. I am writing against the tag and the paper. I have not run this engine on the 3090. Every speed number below is project-reported unless I say otherwise.
The claim is a scheduler, not a new quant.
Prefill on a long MoE prompt destroys sparsity. Each token still routes to a handful of experts. The union across thousands of tokens covers nearly the whole pool at every layer. DeepSeek-V4-Flash in FP4 is about 140 GB of experts walking the bus. The paper’s measured transfer: roughly two seconds on a 5090 with PCIe 5.0 x16, five seconds on a 4090 or 3090 with 4.0 x16, ten or more on a laptop x8. An engine that fetches on demand sits idle for that window. Decode is the opposite regime. One token, a sparse set, and a cache that either hits or does not. Static layer offload, llama.cpp’s -ngl and --n-cpu-moe, freezes the answer at load. KTransformers pins a hot subset at prefill. Routing shifts on the next token. The frozen map is already wrong.
FreeToken keeps the complete routed-expert pool in host RAM as the source of truth. Non-expert weights stay on the GPU. Remaining VRAM becomes one shared LRU cache of layer-expert slots, not a per-layer pin list. Hits execute on the GPU. Misses split. The split is the product.
They call it q*. Measure two bandwidths on the deployed tensor shapes: PCIe expert-transfer and host-side expert-kernel. ft bench bw writes that profile per GPU. A residual miss can fill a cache slot over the bus and run on the GPU, or execute in place on the CPU where the weights already live. Neither is always cheaper. A 4060 laptop on LPDDR5 and a 5090 desktop on DDR5 sit at opposite ends of that balance. Spec sheets will not tell you which side you are on. The paper’s Table 1 is measured, not advertised. A 4090 on a dual-socket rental is not your 4090.
Prefill hides the 140 GB walk with two full-layer buffers. While the GPU computes layer n, a transfer stream loads every expert of layer n+1. Routing for n+1 is not known yet. Loading the whole layer lets the copy start anyway. Overlap on, an 8,192-token chunk on their 5090 finishes in 1.19 to 1.22 seconds, the time to stream 64.4 GB at 52.7 GB/s. Disable the second buffer and you pay 19 to 26 percent as the prompt grows. When VRAM cannot spare two full layers, it falls back to on-demand rather than oversubscribe.
The second-turn problem is the one people stop measuring. Hybrid-attention models compress the past into a recurrent state. Agent harnesses then edit history at thinking blocks, tool calls, and turn boundaries. A checkpoint taken after the edit is dead. FreeToken parks those checkpoints at the same special-token boundaries the harness already cuts on. OpenClaw strips thinking. OpenCode replaces old tool output. The surviving prefix ends at a mark. Restore from the mark, re-prefill only the new suffix. ft ctl cache can rebuild the expert pool and the KV pool at a scheduler safe point without restarting the engine. Context grows. The expert working set does not. A split chosen on turn one is wrong on turn twelve.
uv pip install "freetoken[accel]"
ft bench bw
ft serve --model ~/models/Qwen3.6-35B-A3BServer comes up on 127.0.0.1:1919. OpenAI /v1/chat/completions and Anthropic /v1/messages on the same process. ft launch will point Claude Code, Codex, OpenCode, OpenClaw, DeepSeek Harness, or Hermes at that port and strip the cloud API keys from the child so the agent cannot silently fall back to a paid endpoint. MoE backend is fused if the experts fit, offload if they live in RAM and miss over PCIe, cpu if misses run in place, hybrid if the bandwidth profile says to overlap both. auto picks offload, then hybrid once a ft bench bw file exists.
The complication is the screenshot.
“35B on an 8 GB laptop” is the tweet. The paper cell is Qwen3.6-35B-A3B NVFP4 on an RTX 4060 Laptop with 32 GB of LPDDR5, 39.3 tok/s. It is not BF16. It is not 284B. “284B on a gaming desktop” is DeepSeek-V4-Flash, 13B active, native MXFP4, on a 5090 with 192 GB of DDR5. “753B on a workstation GPU” is GLM-5.2 NVFP4, a 433 GB checkpoint, on a single RTX PRO 6000 with 512 GB of host RAM. The 96 GB card is not housing 433 GB. Host RAM is. Colibrì already taught that lesson from the other direction.
On their 5090, FreeToken reports 77 to 83 tok/s on Qwen3.6 BF16 and 22 to 25 tok/s on DeepSeek-V4-Flash. 1.5 to 2.3 times the strongest baseline in each of four workloads: AIME, OpenCode on a SWE-bench issue, Claude Code on the same issue with concurrent subagents at 56 to 65k tokens, and thirteen OpenClaw mail turns. Decode stays within 12 percent of the single-turn rate as the session goes agentic. KTransformers on DeepSeek-V4-Flash has already lost 31 percent by the second coding turn. Tail TTFT is the number I would actually A/B. Their worst turn stays under 44 seconds in every cell. llama.cpp crosses 232. Ollama 179. KTransformers 946. OpenClaw ships a 120 second idle watchdog. That is an availability boundary, not a latency stat.
Those are author numbers from a v1 arXiv preprint. The 3090, 4090, and 5090 rows in Table 1 are rented dual-socket servers with CPU threads capped to emulate a consumer host. The two machines that are actually edge boxes are the 5090 desktop and the 4060 laptop. Wavect reviewed the paper on September 2 and did not reproduce the GPU table. I have not reproduced it either. Community clips citing 100 tok/s on a 5080 or 51 tok/s on a 3090 are not the paper and are not a matched llama.cpp run on the same file. Wait for a third-party table, or run yours, before you rip out the engine that already posted 99 tok/s on Qwen3.8-27B on this desk.
The install is narrower than the star count. Linux x86_64, NVIDIA, driver r580 or newer, CUDA 13, Python 3.10. Mac and AMD are open issues, not a weekend compile flag. Qwen3.8-Flash-Next keeps a 47.7 GiB PLE n-gram table pinned in host RAM. Multimodal checkpoints serve text only. fused is never auto-selected; you ask for it when the experts actually fit. Nightly wheels on main already carry GLM-5.3-Flash and Qwen3.8-Flash-Next. The tag does not. Pin the commit if you care which family you got.
Use this when you have an NVIDIA box, a supported MoE larger than VRAM, host RAM that can hold the pool, and an agent harness that edits at marked boundaries. The cell worth measuring is p99 TTFT on your gold-patch or mail-kit trace against llama.cpp on the same weights, plus whether the 120 second watchdog fired. Stay with llama.cpp when the job is an agent loop on a model that already fits. Apple Silicon is someone else’s engine. An RX 580 is Colibrì’s problem, not this one.
Tuesday is KTransformers. Dense work on the GPU, experts where host bandwidth can feed them, SGLang in the stack, installation cost as part of the product. A 128 GB workstation with fast DDR5 is a different machine than a 32 GB gaming box. That engine will say so. FreeToken’s opinion is that the split should be measured on the box you own, then allowed to move as KV grows. Static offload is still the baseline. Dynamic scheduling is allowed to win. It still has to beat the number.
A 284B screenshot proves the pool was addressable. A second turn that stayed under the watchdog is the working set with the costume off.
If this was useful, forward it to one engineer who needs less noise in their feed.