Architecture of an Enterprise AI Gateway in Production
The composable pattern that works across healthcare, insurance, and government.
Every enterprise AI deployment I have seen in the last year follows the same architecture even when the teams building them do not realize it. There is a gateway at the edge, a guardrail layer in the middle, and an inference endpoint at the back. The layers are consistent. What varies is where each team draws the boundaries between them, which decisions they push into the gateway versus the application layer, and what they leave out. The teams that handle all three layers survive the security review. The teams that skip a layer find out why six months later when an incident forces the question.
The reference architecture I am going to walk through is not specific to any one client or deployment. It is the pattern that I have seen work across healthcare claims processing, insurance underwriting, and government intelligence analysis three different regulatory regimes with different threat models, and the same stack survived in all of them with only the configuration layer changing. That is the test of a composable architecture: the structure stays, the tuning varies.
The default choice for the gateway layer in 2026 is LiteLLM. I will say why directly because a lot of teams spend weeks evaluating alternatives when the answer is already clear. It covers 100-plus provider integrations, which means you do not rewrite your gateway when you switch from OpenAI to Bedrock or add a self-hosted vLLM endpoint. It has built-in rate limiting that operates per virtual key, which maps directly to per-team and per-user cost attribution. It supports model fallback, so when your primary provider is degraded the gateway routes to a secondary without the application knowing. And it exposes an OpenAI-compatible API, which means any SDK or agent framework that speaks the OpenAI protocol already works with it. LiteLLM v1.91.0 shipped July 4. The version that was current at planning time v1.90.2 moved to 1.91.0 in three days. That release cadence is a signal: the project is actively maintained and responsive to production issues.
The virtual key system is the most underrated feature in LiteLLM and the one that makes the biggest difference in an enterprise deployment. Each team gets a virtual key. Each key has its own rate limit, its own model access list, its own spend cap. When the finance team asks who is spending what on AI inference, the answer is in the LiteLLM proxy logs. No custom instrumentation, no per-request metadata tagging. The keys do the attribution. I have seen teams run for six months with no cost attribution at all, and the cleanup effort to reconstruct it from raw logs always takes longer than setting up virtual keys would have taken on day one.
The guardrail layer is where the honest answer requires a caveat. You need NeMo Guardrails for configurability and you need to plan for its complexity. NeMo Guardrails v0.23.0 shipped July 1. It runs dialog flows defined in the Colang configuration language, and those flows give you something that simple classification filters cannot: conditional logic about what happens when a guardrail fires. A classifier says block or allow. A Colang dialog says block the request, log the event to the SOC pipeline, and surface it for human review only if the confidence score is above 0.95, otherwise let it through with a warning attached to the audit log. That conditional behavior is what makes the guardrail layer survivable in production. Without it, you either block too much and degrade the user experience, or you block too little and discover the gap during an incident.
The Colang DSL is the friction point that teams underestimate. It is a custom language with its own syntax, its own debugging workflow, and a learning curve that is steeper than any team budgets for. I have seen teams adopt NeMo Guardrails, write three dialog flows in the first sprint, and then never add another because the maintenance cost of the Colang code exceeds the perceived benefit. The fix is not to avoid NeMo Guardrails. The fix is to budget for the learning curve upfront and treat the guardrail layer as a maintained codebase, not a configuration file. If you cannot commit to maintaining the Colang flows, you are better off with Portkey’s classification-based guardrails, which trade configurability for zero maintenance overhead and may be the right choice for a team of five.
The inference endpoint is the part of the stack that teams overthink the least because the decision framework is straightforward. If you can send data to a public API, use the model provider directly through LiteLLM’s provider routing. If the data cannot leave your network use vLLM v0.24.0 self-hosted on your own GPU infrastructure. If you need structured output guarantees that free text generation cannot provide route those specific requests through SGLang, which handles grammar-constrained decoding natively. The three-tier inference model is not new. What most teams miss is that the gateway and guardrail layers need to know which inference path each request is on, because the guardrail rules are different for each. A request routed to a public API gets content filtering and PII redaction at the gateway. A request routed to a self-hosted vLLM endpoint in the same VPC gets less aggressive input filtering because the data never leaves the trust boundary, but it gets stricter output filtering because the model is a fine-tuned internal model with access to sensitive data.
Where PII redaction lives in this stack is one of the most common design mistakes I see. Most teams put it in the application layer, which means every service that calls the LLM has its own redaction logic, its own list of patterns, and its own failure modes. The redaction layer belongs in the gateway, before the request reaches the guardrail layer and after the response comes back from the inference endpoint. That gives you a single point of configuration for PII patterns, a single audit trail for what was redacted, and a single place to update when the compliance team adds a new data classification. LiteLLM supports input and output filtering through its custom hook mechanism. You wire a function that scans for PII patterns, and it runs on every request at the gateway boundary. The application layer never needs to know PII redaction exists.
Rate limiting cascades in a predictable pattern that most teams design backward. The standard mistake is to set a single rate limit at the gateway and call it done. The right pattern is three layers of rate limiting that operate at different granularities. The first layer is at the gateway per virtual key, which limits how many requests a team can send to the proxy. The second layer is at the provider level, which limits how many requests the gateway sends to the upstream inference API. The third layer is at the model level, which limits how many requests hit the inference endpoint for a specific model. The cascade matters because a single team should not be able to exhaust the provider quota for every other team. The cascade also means that rate limiting errors return different status codes at each layer, and your application needs to handle all three. A 429 from the virtual key limit means your team is over budget. A 429 from the provider limit means the whole organization is hitting the upstream rate ceiling. A 429 from the model limit means the inference endpoint is saturated. Your error handling should treat each one differently because the remediation is different.
Cost attribution per team is the feature that turns the gateway from a security tool into a business tool. LiteLLM logs every request with the virtual key ID, model, input tokens, output tokens, latency, and cost. The cost is calculated from the provider’s pricing for that model at that time. A weekly report goes to each team lead showing spend by model, cost per request, and trends. The aggregate report goes to the finance team. After six months of data, you can answer the question that every enterprise eventually asks: are we spending more on inference than we expected, and which team is driving the increase? In practice, the answer is always one team running a high-volume batch workload on an expensive model that should have been switched to a cheaper alternative. The cost attribution data makes that conversation a data-driven decision rather than a guess.
The logging layer that makes audit easy is the part of this architecture that teams skip most often because it does not look like it belongs in the gateway. The gateway logs request metadata not response content. The response content from the model and the guardrail decisions go to a separate observability stack Arize Phoenix for deep tracing, Langfuse for prompt management and compliance. The gateway logs the routing decisions, the virtual key used, the rate limit state, the latency breakdown, and the cost. That separation matters because the gateway log is what you produce during a security audit, and it should not contain any model outputs that might contain sensitive data. The observability stack is what you use for debugging, and it should contain everything. The boundary between the two is defined by what data leaves the trust boundary and what stays inside.
The architecture composes. LiteLLM handles the edge, NeMo Guardrails handles the policy decisions, vLLM or the model provider handles the inference. PII redaction lives at the gateway boundary. Rate limiting cascades in three layers. Cost attribution falls out of the virtual keys. Audit logs stay clean by separating metadata from content. None of this is novel. What is novel is how few teams actually build it this way. The most common pattern in the wild is LiteLLM as a thin proxy with no guardrail layer, no PII redaction, no cost attribution, and a single rate limit that everyone shares. That pattern works until it does not, and the failure mode is always the same: an incident that forces the security team to mandate the architecture that should have been there from the start.
The smallest viable implementation of this stack is LiteLLM in Docker on a single VM, one Colang dialog flow in NeMo Guardrails that blocks common injection patterns, and a single inference endpoint from a provider API. That is running in less than a day. From there, you add the virtual keys, the PII redaction hook, the second rate limit layer, and the cost reporting. Each addition takes an afternoon. The architecture does not require a platform team or a six-month initiative. It requires knowing which layer does which job and committing to the maintenance of the configuration between them.
If you are building an enterprise AI gateway right now, that is the decision that will define the next six months. Not which gateway tool you pick. The decision is whether you build all three layers before the security review forces the question or after.
If this was useful, forward it to one engineer who needs less noise in their feed.