The leaves of the tree are not engine names. They are the one benchmark you run before you commit.
The decision tree most people want is a lookup table from GPU model to engine name, and that table gives the wrong answer more often than it helps. Your hardware is not the root of the tree. It sits somewhere in the middle, and putting it first is why so many local setups end up on the wrong runtime.
This week walked through the pieces separately. Ollama for time-to-working. ExLlamaV3 and TabbyAPI for a CUDA-resident speed path when the quantized model fits in VRAM. MLX-LM for Apple unified memory. The mixed-GPU correction that two cards are two pools with a slow wall between them, not one bigger card. Each post answered a narrow question. The reader with a box in front of them still has to assemble those answers into a single decision, and that assembly is the actual skill.
The root question is not which GPU you own. It is two questions asked in order. Does the model fit in the fastest memory the machine has, and what is the workload going to do to it. Hardware becomes decisive only after those two answers narrow the field. A 24GB card serving one person and the same card serving four concurrent agents route to different runtimes. The card did not change. The workload did.
Start with fit, because fit determines which branch of the tree you are even allowed to walk. Fit does not mean the weight file is smaller than the card. It means the quantized weights, plus the KV cache for the context you will actually use, plus compute buffers, plus whatever the desktop already took, all live inside VRAM or unified memory at once. A 24GB card does not fit a file that reports 22GB, because the cache and the buffers were not in that number. This is the trap the whole month keeps naming. The weight file is the down payment, not the price.
When the model fits in VRAM and the card is NVIDIA, the branch splits on what you value. If you want the model running in ten minutes with an OpenAI-compatible endpoint and a curated pull, Ollama is the correct first answer and there is no shame in stopping there. If you want the fastest single-user decode that card can produce and you are willing to spend an evening on setup, ExLlamaV3 behind TabbyAPI takes an EXL3 quantization and stays resident on the GPU, which is where its speed comes from. If you need portability across mixed hardware, quantized KV cache, or an architecture the fast engines have not implemented yet, llama.cpp is the baseline that runs almost everywhere. The same fit, three different runtimes, chosen by whether your scarce resource is time, tokens per second, or compatibility.
When the model fits in unified memory on Apple Silicon, the branch is short. MLX-LM is the native path, and a high-memory Mac runs models that are awkward on a consumer multi-GPU PC because the CPU and GPU share one pool with no discrete VRAM wall to cross. What unified memory removes is the wall. What it does not remove is the capacity ceiling or the bandwidth budget. A 128GB Mac still has to reserve memory for the operating system and the cache, and its memory bandwidth, while generous, is not a 3090’s 940 gigabytes per second. Route here for capacity and quiet operation, not for peak CUDA throughput, and the machine will not disappoint you.
The interesting branch is the one where the model does not fit, because that is where most of the month’s engines live and where the wrong choice costs the most. When a dense model spills, the honest first move is llama.cpp with partial GPU offload, placing as many layers on the card as fit and running the rest on CPU. That path is slower than full residency and faster than pretending the model was never too big. When a sparse mixture-of-experts model spills, the question is not whether it fits but where the experts go, and the answer depends on a resource most people forget to check: system RAM bandwidth and NVMe speed. If host RAM can hold the expert pool, KTransformers and FreeToken make the CPU an execution resource rather than a staging shelf. If RAM cannot hold it either, Colibrì streams routed experts from storage and turns a feasibility result into something you can measure, at token rates that are a systems achievement before they are an interactive workflow.
Now the hardware finally enters, and it enters as a filter on the branches above rather than as the root. An 8GB card lives almost entirely in the fit-first world. Run models that genuinely fit, an SDXL-class visual workflow, a 9B-class text model at Q4, and treat any 35B MoE as a streaming experiment you are conducting on purpose, not a daily driver you were promised. A 12GB to 16GB card opens up GPT-OSS-20B in native MXFP4, a 14B-class model at Q4, or an A3B-style MoE at an exact quant the runtime actually supports, and here the choice between Ollama and llama.cpp comes down to whether you want the knobs or the guardrails.
The 24GB card is where the tree is most rewarding and most tempting to overreach. It comfortably runs a Qwen3.8-27B at Q5 or Q6, and it is the card where the ExLlamaV3 speed path pays for its setup cost. This is also where I have real numbers instead of projections. On my own bench a 24GB 3090 runs a 15.33GB Qwen3.8-27B Q4 file at 99 tokens per second in decode with a Q8 KV cache and a 128K context allocation, and it processes a 99,000-token prompt at 761 tokens per second before generation begins. That is the bar any more exotic configuration on the same card has to clear. Reaching for a larger quant that spills, and loses more speed to the spill than it gains in quality, is the classic 24GB mistake.
Above a single card, the branches stop being about one model. Two mismatched cards, a 16GB next to a 24GB, are not a 40GB card, and yesterday’s post spent its whole length on why the wall between them is real. The design that wins there is usually two services, the text model on the 24GB card with room for its cache to grow and a second job on the 16GB card, rather than one model split across a slow link. A 24GB to 32GB card paired with 128GB or more of fast system RAM is the natural home for KTransformers or FreeToken co-execution, where the GPU does the dense work and the host holds and computes the experts. A machine with 25GB to 32GB of RAM and fast NVMe, and no room to hold the experts anywhere but disk, is Colibrì territory, and the thing to measure there is not whether it runs but whether the prefill on a real prompt finishes before you lose patience.
What flexes across every branch is the workload, and it flexes hard enough to change the engine on identical hardware. A single person watching one terminal wants the fastest engine for that box and nothing else. The moment concurrency enters, an engine built for serving and continuous batching beats a faster single-stream runtime that queues requests behind each other. An agentic multi-turn workload adds a concern neither of those captures: prefix reuse and re-prefill. An agent that edits earlier context pays to reprocess thousands of tokens on its next turn, and an engine that reuses an unchanged prefix will feel faster than a quicker decoder that rebuilds the prompt every time. The fixed part of the tree is the physics of fit and bandwidth. The configurable part is the workload sitting on top of it, and it deserves an honest answer before you pick a leaf.
The cost of walking the wrong branch is not only slower tokens. EXL3 and GGUF are different quantization formats, so committing to the ExLlamaV3 path means maintaining EXL3 files and giving up the portability of GGUF across your other machines. Choosing the fast engine for an architecture it has not implemented sends you back to llama.cpp after you already spent the setup time. A specialized MoE engine buys capacity with installation complexity, duplicate weight storage, and a dependency chain that breaks on the next driver update. None of these engines earns its place from fit alone, and the setup hours are part of the price whether or not you counted them.
Do not end any branch of this tree at an engine name. End it at a benchmark. Before you commit to a runtime, load the exact quant at the exact context you need, then measure time to first token, prefill tokens per second, decode tokens per second, and peak VRAM on your machine, not the project’s. If those numbers clear your task’s latency and quality floor, the search is over and the rest of the tree is noise. If they do not, you learned which branch to walk next for the price of one run.
The leaves were never the engines. The leaves are the numbers, and the machine you already own is closer to a good answer than another spec sheet will ever get you.
If this was useful, forward it to one engineer who needs less noise in their feed.