A cache hit is useful only if it changes how long the next token waits.
A higher expert-cache hit rate can leave your model slower. The cache may hold more of the right weights while speculative transfers occupy the link that an urgent miss needs. MoE-Infinity is worth studying because its source makes that distinction concrete: predicting useful experts and delivering them without delaying generation are separate engineering problems.
That gives it a different place in this week’s oversized-MoE discussion. Yesterday’s KTransformers assessment followed expert computation onto the CPU. MoE-Infinity’s central offloading path keeps expert weights in host memory and SSD storage, then brings needed experts onto GPUs for execution. Its activation-aware cache tries to keep frequently useful experts close to compute. The interesting question is how that policy behaves once several requests want different experts at the same time.
This is a source and documentation assessment, not a benchmark I ran on my hardware. I checked the repository on September 9 and pinned this article to commit 02a4afebffa45ac00815d24671969027c4bbe0ce, committed September 6. The calendar’s August 21 activity cutoff was stale. There are still no Git tags or GitHub releases, and the PyPI package remains a placeholder rather than the runtime. An evaluation starts with a pinned source build, not a reassuring package name.
The README at that commit also warns that this HuggingFace-friendly implementation differs from the performance-focused paper version. That boundary matters before importing any published speedup into a deployment forecast. The current source documents an OpenAI-compatible HTTP server, continuous batching, paged KV cache, streaming, and request preemption. Those are implemented serving components, not proof that every registered model works correctly through every component.
Expert caching earns its memory budget when routing has reuse. A weight tensor already on the GPU avoids a host-to-device transfer; an expert missing from that tier requires more work before execution can proceed. Host RAM and SSD are not interchangeable backing stores. A host-memory hit still needs a GPU transfer, while a disk-backed miss can add storage access and staging. Counting both as simply “offloaded” conceals the latency difference the operator needs to see.
The useful measurement is exposed fetch time: the part of expert retrieval that computation cannot hide. A transfer completed while useful work continues may consume bandwidth without extending the request’s critical path. A shorter transfer started too late can stall the next layer. Cache hit rate helps explain residency, but it does not capture when a miss arrives, what else occupies the transfer queue, or whether the GPU has anything useful left to do.
This distinction becomes sharper under continuous batching. The scheduler can admit new requests while others decode, rather than waiting for one fixed batch to finish. That improves its opportunity to use the device, but unrelated prompts can broaden the expert working set. A long prefill can touch many experts across its tokens while ongoing decodes need their own small routed sets. Raising concurrency therefore changes the cache workload as well as the compute workload.
MoE-Infinity exposes speculative expert prefetch as an option, with the configuration defaulting it off. In the ordinary predictive path, the prefetcher averages current-layer router logits and chooses up to two expert IDs to warm in the next layer. That is a prediction about weight access, not speculative decoding of output tokens. It can also accept an explicit expert set through a separate route-ahead integration.
Correct predictions still have a cost. Early weights occupy cache space, mistaken predictions consume bandwidth, and an aggressive transfer can delay a real miss. The separate overlap flag tries to launch prefetch before the current layer’s barrier, but its configuration help warns about cache pressure and locked-expert warnings. I would leave both options at their defaults for the first run. An optimization needs a clean comparison before it needs a more elaborate explanation.
The repository contains an unusually useful failed experiment. Its BM3 priority benchmark evaluates giving route-ahead transfers a dedicated priority above background prefetch but below on-demand misses. The candidate must lower exposed fetch time, preserve token throughput, and leave on-demand work the fastest service class. The gate must hold across both required targets, rather than whichever model makes the change look good.
The native enqueue code records a NO-SHIP verdict for that dedicated band: across two seeds, the exposed-fetch effect changed sign and did not preserve throughput at the same time. That is a project-recorded finding, not an independent reproduction, and it rejects that priority policy rather than prefetching as a whole. I trust that boundary more than a generic statement that tracing hides transfer cost. The implementation keeps the failed idea from becoming a default on the strength of its explanation.
September’s changes add another competing use for memory. The pinned commit introduces opt-in INT8 KV-cache storage with explicit effective-format and fallback reporting. Its commit message reports a Qwen3-30B-A3B run on SM120 hardware, context 1024, batch four, and 32 decode tokens: native storage reached 5.65 tokens per second, while the INT8 path reached 4.40. These are project-reported results from one configuration, not a general ranking. Smaller KV storage can make room for something else without making the measured request faster.
Model support is where I would slow the evaluation down. The compatibility matrix separates sync/offload evidence from continuous serving. It labels Qwen3 offload validated, Qwen3.5 tiny-fixture validated, and their continuous serving implemented or experimental. DeepSeek-V4 has a validated official offload path but unvalidated continuous serving. GPT-OSS-20B has validated serving evidence, yet its resident expert implementation cannot establish that offloaded expert caching works well.
Open reports make the distinction more than documentation etiquette. Issue #191 reports DeepSeek-V2/V3 continuous-batching output degenerating after the first token because of missing production paged-attention wiring; a proposed fix remains open as PR #195. Other current documentation describes MLA fallback support, so the repository’s support story is not fully consistent. I would require coherent multi-token output from the exact pinned server path before treating either the README example or a throughput number as evidence of usable serving.
My first comparison would keep the checkpoint, precision, memory ratios, prompt corpus, and concurrency fixed, then vary prefetch alone. Cold start, warm decode, and a second turn after a substantial tool result need separate measurements. Record time to first token, inter-token tail latency, exposed fetch time, transfer bytes, and peak host and device memory alongside throughput. Check known-answer tasks before accepting the speed result, then repeat with long prefill arriving during decode to expose contention that a warm singleton hides.
I would use MoE-Infinity as a pinned research runtime for a supported NVIDIA configuration, especially when the purpose is understanding expert placement. I would not choose it as the default endpoint merely because it speaks an API my agent already understands. Before exposing the server, check its documented authentication settings: it defaults to listening on all interfaces without authentication unless a key is configured. The cache policy is only one part of owning the service.
MoE-Infinity earns attention by making failed scheduling ideas inspectable. The next optimization deserves a place in your server only when the request stops waiting sooner.
If this was useful, forward it to one engineer who needs less noise in their feed.