Choose for the model, the machine, and the request path you need to survive.
The best local inference engine is usually the one benchmarked on somebody else’s machine.
A chart shows the highest decode rate for one quantized model on one GPU. A README shows a larger checkpoint loading through CPU offload. A video shows an agent answering one prompt from a 128GB Mac. Each result can be accurate, and none of them tells me what to install on the machine under my desk. The useful answer changes when I swap a 24GB NVIDIA card for Apple unified memory, add a second unmatched GPU, or ask the endpoint to serve three coding sessions instead of one chat window.
There is no best local inference engine. There is an engine whose hardware assumptions happen to match the job.
That sounds like a soft answer until the assumptions are written down. Model format decides which runtimes can load the file. Architecture support decides whether loading it produces correct output. VRAM decides whether the hot path stays on the GPU. System memory bandwidth decides how painful a spill becomes. Request concurrency changes the value of continuous batching. Tool calling turns chat-template fidelity into a production requirement. Setup time determines whether a faster engine pays back its installation cost before the next model release moves the target again.
Most comparisons flatten those constraints into tokens per second. Decode rate matters, but it is one stage of one workload. A coding agent may spend two minutes processing a repository before it generates a 20-token tool call. A shared endpoint may deliver lower single-request speed while completing more work across four simultaneous sessions. A desktop chat user may care more about pulling a model with one command and unloading it cleanly than gaining another 15 percent in a controlled benchmark. The winner changes before the first token is measured.
The one-user CUDA box makes the specialist look good. If a quantized model fits entirely in VRAM on a modern NVIDIA GPU, ExLlamaV3 can optimize around that fact. Its current project documentation describes EXL3 quantization, 2-bit through 8-bit cache quantization, dynamic batching, speculative decoding, and tensor or expert parallel execution. TabbyAPI wraps that backend in an OpenAI-compatible server with model loading, tool calling, structured-output constraints, paged attention, and concurrent inference. That is a coherent stack when CUDA is the machine and the model belongs in the hot pool.
The same recommendation gets weaker as soon as the model spills. ExLlamaV3 now documents CPU offload for large MoE models, so the old claim that it is only a full-GPU engine is stale. The decision still depends on whether its supported architectures, quant format, CUDA and PyTorch build, and offload path line up with the checkpoint in front of me. TabbyAPI labels itself a rolling-release hobby project intended for a small number of users, not a production server. That warning belongs in the architecture decision alongside the speed numbers. A fast backend does not erase its operating model.
A mixed or constrained machine pulls toward llama.cpp. Its advantage is not that it wins every kernel race. Its advantage is the number of bad hardware conversations it is willing to have: CUDA, HIP, Metal, Vulkan, SYCL, CPU-only execution, partial GPU offload, and model placement across more than one device. GGUF is also the broadest practical exchange format in local inference. When I do not know whether a checkpoint will fit, llama.cpp is where I want the first honest measurement because it can keep running after the answer becomes no.
Portability has a bill. A model split across a 24GB card and host RAM does not inherit full-GPU speed. Two unmatched cards do not become one allocation because a layer split uses both. Every boundary adds transfer, synchronization, or a model-specific unsupported path. The baseline is valuable because it exposes those costs without pretending the machine changed. A specialized engine only earns the migration when it improves the phase that limits the real request.
Ollama optimizes a different constraint: the time between deciding to try a model and having an endpoint that another application can reach. Its current documentation covers macOS, Windows, Linux, a local REST API, partial OpenAI compatibility, model lifecycle controls, and direct launches for coding agents. The project identifies llama.cpp as a supported backend. That relationship explains both the reach and the trade. Ollama packages a complicated local stack into a product-shaped experience, while some of the low-level placement and memory controls remain easier to reach in llama.cpp itself.
I would still put Ollama first on many developer machines. The first answer should minimize the chance that the user spends an evening resolving CUDA wheels before learning whether the model can perform the task. Convenience is not a beginner feature. It is an operating-cost decision. The recommendation changes when exact GPU splits, unusual expert placement, quantized cache experiments, or peak single-user CUDA throughput become the work rather than the means.
Apple Silicon removes one familiar wall and introduces a different decision. A 128GB Mac does not have a 24GB VRAM pool beside a much larger system-memory pool. CPU and GPU share unified memory, which lets MLX-LM run model shapes that become awkward across discrete cards. Its documentation includes Hugging Face conversion, quantization, rotating KV caches, prompt-cache files, and configurable prefill steps. Those are not convenience details on a large-context workflow. They decide how much of the shared pool the weights, cache, and macOS can occupy together.
Unified does not mean unlimited or uniformly fast. MLX-LM warns that models large relative to installed RAM can slow down, even when they technically fit, and its rotating cache explicitly trades memory for quality. Smaller prefill steps reduce peak memory at a speed cost. Prompt caching can avoid recomputing a stable prefix, but a changing agent transcript may not preserve enough of that prefix to pay. The Mac avoids PCIe transfers between separate CPU and GPU pools. It does not avoid bandwidth, cache growth, swap, or architecture support.
The API boundary can invalidate a good hardware choice. “OpenAI-compatible” rarely means every client assumption survives unchanged. Tool schemas, parallel tool calls, reasoning fields, JSON constraints, stop behavior, chat templates, token accounting, and model lifecycle controls vary by server and checkpoint. The engine can generate text at 80 tokens per second and still fail the task by emitting a tool call the harness cannot parse. I have seen the same local model fail to read a file in one coding harness and complete the work in another. That is not a kernel benchmark. It is still an inference-stack result.
Concurrency creates another fork. A single person watching one terminal rewards low latency and aggressive use of the whole GPU. Three agents sharing one endpoint reward scheduling, paged attention, cache management, cancellation, and isolation between requests. The engine with the best one-stream decode number may produce the worst afternoon once long prompts arrive together. Local does not always mean single user. One engineer can create a multi-tenant workload without inviting anybody else.
Setup tolerance deserves a line in the benchmark sheet because it is part of cost. A prebuilt Ollama installer and a curated model pull can be working in minutes. A pinned ExLlamaV3 environment may require matching Python, PyTorch, CUDA, wheels, quant conversion, and server configuration. llama.cpp can arrive as a binary, but exploiting an unusual split may still require flags, repeated measurements, and a current build. MLX-LM narrows the platform choice, then asks the model conversion and cache questions inside that boundary. None of this makes the specialist wrong. It sets the amount of performance the specialist must return before the switch pays.
The 24GB 3090 on my own bench makes that payback test concrete. llama.cpp already ran a 15.33GB Qwen3.8-27B Q4 file at 99 tokens per second in decode, with Q8 KV, one slot, MTP, and a 128K context allocation. A 99,226-token prompt processed at 761 tokens per second and still imposed about 131 seconds before generation. If my complaint is slow prefill on a coding session, moving to a specialist because it posts a higher decode rate may optimize the wrong number. I would test ExLlamaV3 there, but the migration has to cut the wait I can feel or preserve more quality in the same memory. Another decode trophy does not pay the bill.
The 128GB Mac and the mixed 16GB plus 24GB Windows workstation invert different parts of that test. The Mac makes one large allocation possible, then asks whether memory bandwidth, cache growth, and macOS headroom keep it useful. The Windows box offers two fast pools separated by software support and transfer cost. llama.cpp may split a supported model across them, while two independent services may produce the better system: one card for the text endpoint and one for embeddings, vision, or a second agent. Selecting the engine before selecting that topology locks the decision at the wrong layer.
My decision order is workload, model, hardware, then engine. Workload tells me whether I care about first-token delay, sustained decode, repeated prefix reuse, concurrency, structured output, or unattended recovery. Model tells me the architecture, available quant formats, and quality floor. Hardware tells me which memory pool holds the weights and where the cache has room to grow. Only then does an engine comparison become more than a borrowed leaderboard.
The first benchmark should follow the same order. I want the actual system prompt, one representative input, one tool call, the returned tool payload, and the second turn. I record cold and warm time to first token, prefill rate, decode rate, peak VRAM and RAM, cache settings, and whether the answer passes one task-specific quality check. A chat demo proves chat. An agent endpoint has to survive the request after the tool result changes its context.
This week will put names against those decision points. Ollama gets the convenience path. ExLlamaV3 with TabbyAPI gets the CUDA-resident specialist path. MLX-LM gets the unified-memory path. llama.cpp remains the portable baseline, especially when the model does not fit neatly or the hardware refuses to be uniform. The mixed-GPU case gets its own correction because 16GB plus 24GB is useful capacity, not a transparent 40GB card. The final decision tree will begin with the box and the workload instead of ending with a logo.
I do not want one runtime installed everywhere. I want each endpoint to have a reason for existing and a test that can make us replace it. The best engine is not a product name. It is the one that fails last on the request your machine actually has to serve.
If this was useful, forward it to one engineer who needs less noise in their feed.