The fastest single-user stack on a modern NVIDIA card is also the one with the most conditions attached.
ExLlamaV3 earns its place only after another engine has already answered the question that decides whether it belongs on your machine.
That question is narrow: does a supported quantized model fit, with its cache and its context, inside modern NVIDIA VRAM? When the answer is yes, ExLlamaV3 stops being one option among several and becomes the specialist that extracts decode speed the portable engines leave unclaimed. When the answer is no, the stack still runs through CPU offload, but the reason to prefer it over llama.cpp weakens with every layer that leaves the card. This is the CUDA-resident speed path. Its value and its limits come from the same design decision.
The project moves fast enough to make version pinning a habit rather than a courtesy. ExLlamaV3 v1.5.0 shipped September 13, 2026, the latest of five stable releases in the preceding eleven days. PyPI carries the same 1.5.0, uploaded the same day. The repository holds 1,395 stars, and its official server, TabbyAPI, holds 1,387. Those are small numbers next to Ollama’s 180,000 or llama.cpp’s 126,000, and the gap is information: this is a specialist tool for an engaged minority, not a mass-market runtime. The people who run it tend to know exactly why.
The speed comes from committing to one hardware story instead of every hardware story. EXL3 is the library’s own quantization format, based on QTIP, paired with 2-bit through 8-bit cache quantization, dynamic batching, speculative decoding, and tensor or expert parallel execution across consumer cards. TabbyAPI wraps that backend in a FastAPI server with an OpenAI-compatible surface, Hugging Face model downloading, embedding support, and Jinja2 chat templates. The README names TabbyAPI the official and recommended backend, and its startup script installs the prerequisites that otherwise turn a first run into an afternoon. On a single modern NVIDIA GPU holding a quantized model in the hot pool, that combination is a coherent, fast, single-user endpoint.
TabbyAPI’s server surface is more capable than the hobby-project label suggests. It exposes model loading and swapping, tool calling, structured-output constraints, paged attention, and concurrent inference, and it ships a CUDA container on GitHub’s registry with tags for CUDA 12.8 and CUDA 13. One operational detail matters more than it looks: the Docker default of 64MiB shared memory is too small, because ExLlamaV3 uses POSIX shared memory for tensor parallelism and CPU MoE offload, so the image needs a raised --shm-size or those paths fail. The capability is there. The framing around it is a warning that you own the uptime.
The first cost is the format itself. EXL3 is not GGUF. A model has to exist as an EXL3 quantization or be converted, and that quant does not move to a Mac, an AMD card through Vulkan, or a CPU-only box the way a GGUF file does. llama.cpp’s portability is the thing you give up to gain the speed. On a machine that will only ever run CUDA, that trade is easy. On a machine whose next model might not have an EXL3 build, or whose next GPU might not be NVIDIA, the portable format keeps options the specialist closes.
The second cost is setup and support posture. ExLlamaV3 does not manage its own PyTorch dependency. You install a CUDA 12.4-or-later Torch build first, match a prebuilt wheel to your CUDA and Python versions, or accept a source build with the toolkit present. The README’s own install example still points at an old v0.0.6 wheel URL, a reminder that documentation on a fast-moving project drifts behind the releases. TabbyAPI is blunter about what it is: a rolling-release hobby project made for a small number of users, explicitly not meant for production servers. That warning is not modesty. It is an operating fact that belongs in the decision next to the throughput.
The third cost hides inside a genuine improvement. v1.5.0 added an experimental zero-copy pinned arena mode for CPU offload on Linux, alongside faster MoE prefill and decode and better speculative decoding for MoE models. ExLlamaV3 can now run large MoE checkpoints with limited GPU memory, which retires the old claim that it only handles fully resident models. The catch is that the moment the working set leaves the card, you are back in the regime llama.cpp was built for, where transfer cost and format breadth start to outweigh the last increment of CUDA speed. The offload path is real. It is not the reason to choose this stack over the baseline.
That returns the decision to the only test that pays: does the migration cut the phase that limits the request I actually send? On the 24GB RTX 3090 under my desk, llama.cpp already runs Qwen3.8-27B at a 15.33GB Q4 quant, decoding at 99 tokens per second with MTP speculative decoding, Q8 KV cache, one slot, and a 128K context that leaves about 2GB of headroom. A 99,000-token prompt processes at 761 tokens per second and still imposes roughly 131 seconds before the first generated token. If my complaint is that a coding agent waits two minutes on prefill, a specialist that posts a higher decode number optimizes the wrong stage.
ExLlamaV3 earns the switch when the workload rewards what it is tuned to do. Single-user decode on a fully resident model is its strongest case, and its cache quantization gives a lever llama.cpp’s Q8 KV does not match: pushing the cache to lower bit depths to buy context or a larger weight quant in the same VRAM, with a quality risk you have to measure rather than assume. Speculative decoding and the MoE prefill improvements matter most on the architectures the release notes name, and the supported list is broad, covering Llama, Qwen, GLM, Gemma, DeepSeek V4, Mistral, Nemotron, and GPT-OSS families. The honest form of the recommendation is conditional: bring a representative prompt, run the second turn after a tool result, and keep the engine only if it cuts the wait you can feel or preserves more quality in the memory you have.
Fit, speed, and quality stay separate claims here too. EXL3 establishes fit the same way every engine does, by loading the model with its cache and context and leaving room for the desktop. It establishes speed only against a recorded baseline on the same card, not against a chart from a Pro 6000 in the release notes. It establishes quality only when a task gate passes: a tool call that parses, a repository edit that survives its tests, an answer checked against source. A 3-bit cache that doubles context and quietly corrupts a long retrieval has traded a number you can see for a failure you cannot.
The specialist is not the engine that wins the benchmark. It is the engine that was built for exactly one machine, and the discipline is refusing to install it until that machine is the one you have.
If this was useful, forward it to one engineer who needs less noise in their feed.