The VPC Boundary
Why Your Most Sensitive Inference Will Never Hit a Public API
Private inference is not a niche. It is the default for anyone processing sensitive data, and the architecture decision that determines whether your enterprise AI deployment survives security review.
The most valuable AI workloads in the enterprise run on data that cannot leave the VPC. Healthcare claims adjudication processes protected health information under HIPAA. Financial fraud detection models score transactions that contain personally identifiable financial data. Government intelligence analysis ingests classified material that cannot touch infrastructure outside a controlled boundary. These are not edge cases. They are the core use cases that justify enterprise AI investment at the scale where the budget reaches seven figures, and every single one of them has a hard architectural constraint: the inference request cannot cross a public network.
Most of the AI content published this year skips this reality. It assumes a world where the model lives behind an API key and the data traveling to it is just text. In that world, you call GPT-4o or Claude Sonnet 4, you pay per token, and the security conversation starts and ends with API key rotation and data retention policies. That world exists and covers a meaningful range of use cases. But it is not the world where the most expensive and most impactful enterprise AI deployments live. Those deployments live in the world where the model has to come to the data, not the other way around.
The architectural implication is straightforward. If your inference cannot cross the VPC boundary, you are running the model yourself. You are provisioning GPUs, managing inference infrastructure, handling model updates, and absorbing the capital cost of hardware that runs at utilization rates that make a finance team nervous. You are accepting operational complexity as the price of data sovereignty, and the question is not whether that tradeoff is worth it. It is whether you can execute it without creating a cost explosion or a maintenance nightmare.
The decision framework starts with the data classification. There are three tiers. Tier one is data that can leave the VPC under contractual terms. Customer support transcripts, product documentation queries, marketing content generation. These go to public API endpoints with appropriate data processing agreements, and they represent the bulk of AI usage in most organizations. Tier two is data that cannot leave the VPC but can be processed on dedicated infrastructure within a trusted cloud boundary. Financial transactions, HR records, internal knowledge bases that contain business-sensitive but not regulated content. These need a VPC-deployed inference endpoint on the organization’s cloud tenant. Tier three is data that cannot leave a controlled facility. Classified government material, certain healthcare data under the strictest interpretations, proprietary ML training data that represents competitive advantage. These need on-premises inference with air-gapped deployment.
Most organizations have all three tiers. The mistake I see most often is designing for the lowest tier and hoping the architecture stretches. It does not. A deployment pattern that works for tier one content breaks on tier two the first time the security team audits the data flow and discovers that something classified as internal-only is crossing a public network boundary. The fix is not a better VPN. It is an architecture that treats the VPC boundary as a design constraint from the start.
The tools available for private inference have matured significantly in the last eighteen months. vLLM, at 86,000 GitHub stars and with version 0.25.1 released yesterday, is the de facto standard. PagedAttention, continuous batching, tensor parallelism, and broad GPU support make it the safest choice for any team deploying private inference on NVIDIA hardware. The version cadence has been steady for years, and the community is large enough that if you hit a bug, someone has already hit it and either fixed it or documented the workaround. vLLM handles dense models well, supports a wide range of GPU architectures, and integrates with every major orchestration framework.
SGLang, at 30,000 stars and version 0.5.15 released yesterday, has a narrower focus that gives it an edge for specific workloads. Its native support for grammar-constrained decoding means that if your private inference pipeline produces structured outputs that feed into line-of-business systems, SGLang handles that path more efficiently than vLLM. The constraint-guided generation operates at the token selection level rather than as a post-processing step, which means the output is guaranteed to match the schema without retries. For any workload where structured output is the primary integration pattern, SGLang is worth evaluating as the primary engine rather than the secondary one.
TensorRT-LLM from NVIDIA offers the highest throughput on NVIDIA hardware but at the cost of full stack lock-in. If your organization is already NVIDIA-native across the ML stack and has the engineering depth to manage NVIDIA’s toolchain, TensorRT-LLM delivers the best raw performance. For most organizations, the vLLM path is better because it is more portable and has a larger community. TensorRT-LLM makes sense when you have a stable workload at high utilization and the team to tune it. It does not make sense as the default choice for a team that is still figuring out its inference pattern.
The deployment architecture for these engines follows a consistent pattern regardless of which engine you choose. The models live in a container registry within the VPC, pulled from a private registry that mirrors approved model weights from Hugging Face or another upstream source. The inference server runs on Kubernetes with NVIDIA’s GPU operator managing device allocation. A routing layer sits in front of the inference endpoints and decides, based on the request’s data sensitivity classification, which engine serves it and whether any intermediate processing steps are required. The routing layer is where PII redaction, prompt injection detection, and content filtering happen before the request reaches the model. It is also where the logging happens that the compliance team needs to verify that no sensitive data left the VPC.
The part of this architecture that most documentation does not tell you about is the cost. Private inference is more expensive than API-based inference for most workloads, and the gap widens at low utilization. A single H100 node running vLLM with a 70B parameter model costs somewhere between 10,000 and 15,000 dollars per month in cloud GPU spend, depending on the provider and the commitment level. That node handles a meaningful number of concurrent requests, but if your workload is a few hundred requests per day, the per-request cost of private inference is dramatically higher than a pay-per-token API. The economics flip when the workload reaches scale or when the data sensitivity rules out the API option entirely. At that point, the cost of private inference is the cost of doing business, and the comparison is not against API pricing but against the cost of not deploying the capability at all.
The tooling for managing private inference at scale is still evolving. NVIDIA’s GPU operator handles device allocation on Kubernetes but does not address multi-tenant scheduling across teams. Run:ai, now part of NVIDIA, adds a scheduling layer but introduces its own operational overhead. The orchestration gap is real and is the subject of a later post in this arc. For now, the honest assessment is that private inference at enterprise scale requires Kubernetes expertise, GPU operations experience, and a willingness to manage infrastructure that is more complex than a serverless function call.
If you are evaluating whether your organization needs private inference, the decision rule is simple. If your most sensitive AI workload processes data that your security policy, compliance framework, or legal agreement prohibits from crossing a public network boundary, you need private inference. The question is not whether it is worth the operational cost. It is whether you have the team and the budget to execute it reliably. The organizations that can answer yes to both questions will run inference workloads that their competitors cannot touch, and that advantage compounds over time.
The decision that separates the teams that execute private inference well from the teams that struggle is the same decision that separates most successful infrastructure investments from unsuccessful ones. It is not about picking the right inference engine. It is about accepting the operational complexity as a permanent feature of the architecture rather than a temporary phase that will go away when the technology matures. Private inference is not getting simpler. The models are getting larger, the hardware is getting more specialized, and the security requirements are getting stricter. The teams that plan for that trajectory are the ones that will still be running production AI five years from now.
If this was useful, forward it to one engineer who needs less noise in their feed.