Designing a Private Inference Stack for a Regulated Enterprise
The reference architecture for serving sensitive workloads that never leave your VPC.
The private inference arc this week has covered the tools: vLLM for throughput, SGLang for structured output, RunPod for burst capacity. Each makes a standalone case. What I have not done yet is describe how they fit together in a single architecture that meets the requirements of a regulated enterprise. That is the gap this post fills.
Here is the scenario. You work for a healthcare payer, a regional bank, or a government agency. Your most valuable AI workload operates on data that cannot leave the VPC. Protected health information. PII from financial transactions. Classified operational data. The model itself, fine-tuned on proprietary data, is an asset you cannot expose to a public API. The architecture I am going to describe was designed for this scenario, and it answers the question that every private inference deployment must answer: how do you serve multiple model types, multiple latency profiles, and multiple data sensitivity tiers from a single cost-effective infrastructure footprint?
The architecture has four layers that form a request pipeline. Each layer has a specific job, a specific tool, and a specific decision rule for when to use it versus skip it. The layers are layered in the network sense: traffic passes through them in order, and each layer adds a capability that the layer before it could not provide.
The first layer is the routing layer, and it is the most important because it decides which model serves each request. This is not a load balancer in the traditional sense, though it sits at the same point in the network topology. The routing layer inspects each incoming request and decides three things: which model should serve it, which inference engine should run that model, and what data sensitivity tier the request belongs to. The sensitivity tier determines which VPC subnet the request traverses and whether the response gets logged with full content or with PII redaction applied in transit.
A request asking for a claims processing summary on a named patient hits the high-sensitivity tier. A request asking for a coding suggestion on de-identified synthetic test data hits the low-sensitivity tier. Both hit the same cluster, but they follow different data paths and different retention policies. The routing layer makes that decision in under five milliseconds with a lightweight rules engine running in a sidecar container. A simple YAML config defines the sensitivity tiers, the model-to-engine mappings, and the fallback chain when the primary inference path is saturated.
The second layer is the high-throughput inference tier powered by vLLM. This is where the fine-tuned production models live. For a regulated enterprise, the most common setup is a fine-tuned Llama 3.1 70B that has been adapted on organization-specific data through QLoRA or full fine-tuning. The model runs on a cluster of NVIDIA H100 or A100 GPUs distributed across multiple nodes in the same VPC, connected by high-bandwidth interconnects. vLLM’s tensor parallelism distributes the 70B parameter model across GPUs within a node, and its pipeline parallelism distributes layers across nodes when a single node does not have enough GPU memory.
The key architectural decision at this layer is how to colocate multiple fine-tuned models on the same GPU cluster without running out of memory. Most regulated enterprises have more than one fine-tuned model. A claims processing model. A prior authorization model. A customer service escalation model. Running each on its own dedicated GPU cluster is cost-prohibitive. vLLM supports model colocation through automatic prefix caching and LoRA adapter swapping, which means a single GPU can serve multiple fine-tuned adapters from the same base model by swapping the LoRA weights on each request. The latency impact of swapping is measurable but small, typically twenty to fifty milliseconds per swap, which is acceptable for workloads where the model response time is measured in seconds anyway.
The third layer is the structured output tier powered by SGLang. Not every inference request produces free text. Some of the most valuable workloads in a regulated enterprise produce structured outputs that feed directly into line-of-business systems. A claims adjudication output needs to match a predefined schema with specific fields for diagnosis codes, procedure codes, and dollar amounts. A compliance check needs to return a structured assessment with pass-fail determinations per regulation. These workloads benefit from SGLang’s grammar-constrained decoding because the output must conform to a schema, not just roughly match one.
The architectural choice between vLLM and SGLang for a given workload depends on a single question: does the output need to be machine-consumed by an existing system that expects a specific schema? If yes, route to SGLang. If the output is consumed by a human or by a downstream process that can handle natural language, route to vLLM. The routing layer makes this decision per request based on the endpoint the request arrived at, not on any analysis of the request content. The enterprise operator defines which endpoints produce structured outputs and which produce free text at deployment time, and the routing decision is deterministic from that point forward.
The fourth layer is the batch processing tier powered by RunPod spot instances. Not all inference happens in real time. Batch workloads like monthly claims reprocessing, bulk document classification, and periodic model evaluation generate thousands or millions of inference requests that do not need sub-second latency. These workloads can run on preemptible spot instances at a fraction of the cost of on-demand reserved capacity. RunPod spot pricing for H100 instances currently sits around fifty cents per hour, compared to two dollars and thirty-nine cents for on-demand. The savings on a batch workload that runs for a hundred hours per month cover the cost of the entire routing layer infrastructure.
The architectural pattern for batch processing is to decouple the batch workload from the real-time inference cluster. A queue-based ingestion pipeline collects batch requests in a message queue or object store bucket. A batch orchestrator, which can be a simple cron job or a Kubernetes Job, provisions RunPod spot instances, submits the batch work, collects the results, and tears down the instances. If a spot instance gets preempted mid-batch, the work it was processing is still in the queue and gets reassigned to the next available instance. This is the same pattern that data engineering teams have used for years with Spark and EMR. It works identically for inference workloads.
The data path between layers is worth describing because it is where most architectures leak data. Every request that enters the routing layer carries a sensitivity tag. That tag propagates through every downstream call. When vLLM processes a request tagged high-sensitivity, it writes the full prompt and response to an encrypted log that goes to the compliance audit store. When it processes a request tagged low-sensitivity, it writes a metadata-only log to the operations monitoring store. The same SGLang instance handles requests from both tiers because the model is the same, but the logging and retention policy differ based on the tag. This is not a security boundary. The VPC boundary itself is the security boundary. The sensitivity tag controls the data governance layer, not the access control layer.
The cost of this architecture varies dramatically by scale. At the low end, a single-node cluster with four A100 GPUs running vLLM and a separate SGLang instance on CPU can serve a small enterprise for roughly three thousand dollars per month in GPU compute. At the high end, a multi-node cluster with thirty-two H100 GPUs, a dedicated SGLang inference node, and active batch processing on spot instances can run fifteen to twenty thousand dollars per month. The batch workloads on spot instances represent the largest variable cost because they scale with the volume of offline inference, but they also represent the largest savings opportunity because running them on spot vs. on-demand can cut the batch compute bill by seventy to eighty percent.
The architecture composes because each layer handles a distinct workload pattern. The routing layer does not know about GPU scheduling. The vLLM cluster does not know about structured output schemas. The SGLang instance does not know about spot instance preemption. The batch tier does not know about request sensitivity tiers. Each layer has a narrow contract with the layers around it: receive a request with a routing key, process it, and return the result with the same routing key attached. That is the pattern that makes the architecture composable rather than a monolith, and it is the pattern that lets a regulated enterprise add new model types, new sensitivity tiers, and new consumption patterns without rebuilding the infrastructure from scratch.
The small start for any team building this today is the routing layer and a single vLLM cluster with one fine-tuned model. Run that for a month. Observe the traffic patterns. Add SGLang when the first structured output workload arrives. Add the batch tier with RunPod spot instances when the offline processing volume justifies the orchestration overhead. The architecture does not require all four layers to deliver value. The routing layer and vLLM alone cover the majority of enterprise inference workloads. The additional layers extend the architecture into the edge cases where most teams discover, six months in, that they need capabilities they did not design for.
If you are building a private inference stack for a regulated enterprise right now, the decision that will define the next six months is not which inference engine to pick or which GPU SKU to reserve. It is whether you design the routing layer before you need it or after. The teams that design it first have a growth path. The teams that design it after have a migration.
If this was useful, forward it to one engineer who needs less noise in their feed.