vLLM at Scale
The De Facto Standard for Production Inference
At 86,000 GitHub stars and version 0.25.1 released this week, vLLM is not the fastest inference engine or the most specialized one. It is the one you can bet your production deployment on, and for most enterprise teams, that is a better criterion.
The numbers tell the story that the architecture posts do not. vLLM crossed 86,000 GitHub stars this week, up from 55,000 at the start of this month. Version 0.25.1 shipped on July 14 with Model Runner V2 as the default execution path for all dense models, the removal of legacy PagedAttention, and a Transformers modeling backend that now matches native vLLM performance. That growth rate and release cadence are not marketing signals. They are a direct measure of how many production deployments depend on this engine and how many contributors are actively improving it.
There are inference engines that are faster on specific hardware. There are engines that handle structured output more efficiently. There are frameworks that offer deeper integration with specific model families. But there is no inference engine that more production deployments trust at scale, and the gap between vLLM and every other option in that dimension is widening with every release cycle.
The architecture is worth understanding because the design decisions are what make it the safe choice. PagedAttention, which vLLM introduced and which has since been adopted or replicated by every other inference engine, solves the problem that defines production inference at scale: the KV cache. When a transformer model processes a sequence, it computes key and value tensors for every token. Those tensors are large, they grow with sequence length, and they fragment in memory the same way that application memory fragments over time. PagedAttention treats the KV cache as a set of fixed-size blocks mapped to non-contiguous physical memory, analogous to how virtual memory pages map to physical frames in an operating system. The result is near-zero memory waste from fragmentation and the ability to pack more concurrent requests onto the same GPU.
That single design choice is responsible for the throughput advantage that vLLM has maintained through multiple generations of hardware and model architectures. It is also the reason vLLM handles long context windows better than engines that allocate contiguous KV cache buffers. When your workload has variable-length sequences, which is every production workload, the difference is not incremental. It is the difference between utilizing 95 percent of your GPU memory versus watching 30 percent sit idle because of fragmentation.
Continuous batching, the second major architectural feature, eliminates the idle GPU time that batch-based inference incurs. In a traditional batched approach, the engine collects requests until it has enough to fill a batch, runs them together, waits for the slowest request to finish, and then starts collecting the next batch. The gaps between batches are wasted compute. vLLM processes tokens on a per-request basis within a batch, adding new requests and removing completed ones continuously. The GPU stays fed, and the latency profile flattens. For interactive workloads where users expect sub-second response times, continuous batching is not a nice-to-have. It is the feature that makes the difference between a usable product and one that feels like submitting a batch job.
Version 0.25.0, released July 11 with 558 commits from 232 contributors, made Model Runner V2 the default execution path for all dense models. MRv2 is not a new feature in the traditional sense. It is a re-architecture of how vLLM executes model forward passes, separating the model definition from the execution strategy. The practical effect is that model support accelerates because contributors do not have to understand the full inference pipeline to add a new architecture. They define the model graph, and MRv2 handles the optimized execution. This is the same pattern that made PyTorch successful versus earlier frameworks that required deep framework knowledge to add a new operator. The standardization of the execution path also makes feature development faster because new capabilities like prefix caching, speculative decoding, and multimodal support benefit from a single optimization target rather than needing separate implementations for each model backend.
The same release removed legacy PagedAttention entirely. This is worth pausing on because it signals something important about the project’s maturity. vLLM is old enough and confident enough in its V1 backend to delete the code path that made it famous. Legacy PagedAttention was the foundation that every subsequent optimization built on, and the team decided that maintaining backward compatibility with the original implementation was no longer worth the cost. That is the kind of decision that a project makes when it has enough production users who have already migrated and enough confidence that the new implementation handles every edge case. It is also the kind of decision that a project in a less mature ecosystem cannot make because it cannot risk fragmenting its user base.
The Transformers modeling backend reaching parity with native vLLM is the third signal in this release that matters for enterprise deployments. The Transformers backend allows vLLM to run models that do not have a custom vLLM implementation, using the Hugging Face Transformers library as the model definition layer. Historically, this path was slower than the native implementation. With version 0.25.0, the performance gap is closed for most model architectures, and the Transformers backend gained FP8 MoE support, CUDA graph compatibility, and expanded coverage for GPTBigCode, Starcoder2, and RoBERTa. The practical effect for enterprise teams is that the set of deployable models expanded without requiring custom vLLM integration work for each one.
Where vLLM falls short is where its design philosophy becomes a limitation. vLLM optimizes for throughput and broad compatibility, not for specialized workloads. If your primary inference pattern is structured output generation with strict schema enforcement, SGLang’s grammar-constrained decoding will beat vLLM on both accuracy and latency because it operates at the token selection level rather than as a post-processing step. If your workload is MoE models at extreme scale with a narrow, well-understood traffic pattern, TensorRT-LLM on optimized NVIDIA hardware will deliver higher throughput. If your deployment is on AMD hardware, which is increasingly common in cost-conscious enterprise environments, neither engine supports it natively and you are evaluating ROCm or other alternatives.
The right way to evaluate vLLM for your deployment is to ask a different question than which engine is fastest. The question is which engine you can bet your production uptime on. vLLM wins that question on three dimensions. The community is large enough that bugs get found and fixed within hours of a release, not weeks. The release cadence has been steady through multiple major version upgrades, model architecture shifts, and hardware generations. The documentation, deployment guides, and Helm charts for Kubernetes deployment are maintained by the same team that writes the code, not by a separate documentation team that lags the releases.
The practical implication for an enterprise team evaluating private inference today is that vLLM should be the default choice unless you have a specific workload that requires a specialized engine. Start with vLLM on your VPC-deployed Kubernetes cluster with NVIDIA’s GPU operator handling device allocation. Benchmark your actual traffic pattern, not a synthetic load test. If the throughput meets your requirements, and for most workloads it will, you have eliminated a major architectural risk by choosing the engine with the largest production footprint in the ecosystem. If you hit the edge case where structured output performance or MoE throughput falls short, SGLang or TensorRT-LLM are proven alternatives that integrate into the same architecture. But start with vLLM. The safe choice is also the smart choice when the safe choice has this many production hours behind it.
If this was useful, forward it to one engineer who needs less noise in their feed.