The Multi-Tenancy Problem Nobody Solved
AI Inference at Org Scale
The question every enterprise AI platform eventually faces: how do you serve multiple teams from a shared inference cluster without it becoming a free-for-all?
The private inference arc this week has given you the tools. vLLM for dense model throughput. SGLang for structured output. RunPod for burst capacity. A reference architecture showing how they fit together at the VPC level. What none of these tools solve, and what no tool I have found solves well, is the question that hits every enterprise about four months after the first model goes to production: how do you serve multiple internal teams from the same GPU cluster without it turning into a shared resource tragedy?
The problem is not technical in the way most people frame it. Most people frame it as a GPU scheduling problem. Should I use Kubernetes with the NVIDIA GPU operator? Should I write a custom scheduler? Should I allocate GPUs per team and let them fight over the allocation? Those are real questions, but they are not the hard question. The hard question is that every team has different traffic patterns, different latency requirements, different model sizes, and a different definition of what fair access looks like. And the cost of getting multi-tenancy wrong is not a slow cluster. It is a slow cluster that everyone blames on everyone else, with no way to prove who is consuming what.
The common pattern I see in practice is the carve-up approach. The infrastructure team allocates a fixed number of GPUs per business unit. Team A gets four H100s. Team B gets two. Team C gets one. This feels fair because it is simple and visible. Everyone knows their allocation. No team can claim they are being starved by another team’s workload.
The carve-up approach fails because inference demand is not steady-state. Team A runs a real-time customer-facing agent that needs capacity during business hours. Team B runs a batch claims-processing pipeline that runs overnight. Team C runs sporadic research experiments that spike unpredictably. With fixed allocations, Team A’s GPUs sit idle at midnight while Team B’s batch queue backs up because they only have two GPUs. Team C’s research spike hits their single GPU allocation and the experiment takes four times longer than expected. Nobody is happy, and nobody used less total compute than they would have in a pooled model.
The carve-up approach also fails on model colocation. Not every model needs a full GPU. A fine-tuned 8B parameter model uses a fraction of an H100 at inference time. But if it is allocated to a fixed GPU, that GPU is wasted the rest of the time. The fixed-allocation model optimizes for accounting simplicity at the expense of utilization. The more teams you have, the worse this gets.
The opposite extreme is the shared pool. All GPUs in a single pool. All teams submit to the same queue. The scheduler handles fairness. This is the dream of GPU virtualization that every hardware vendor sells, and it works fine until the first incident. The first incident is always the same: Team A deploys a new model that is slightly too large for efficient inference on their preferred GPU count. The scheduler, trying to be fair, preempts Team B’s running inference to make room. Team B’s users see timeout errors. Team B’s manager escalates to the VP. The VP asks why Team A’s experiment gets priority over Team B’s production workload. The answer is that the scheduler does not know the difference between a production workload and an experimental one. Nobody told it.
This is the gap. Not GPU scheduling. Not fair-share allocation in the abstract sense. The gap is that no tool in the current stack has a concept of workload priority that maps to organizational reality. Kubernetes has pod priorities and preemption policies, but configuring them correctly for inference workloads is harder than most teams realize, and the default behavior almost never matches what the business actually needs. The NVIDIA GPU operator gives you device-level isolation but no workload-level awareness. vLLM and SGLang do not even see the concept of a tenant. They see requests. A request from a production customer-facing agent and a request from a data scientist running a weekend experiment look identical to the inference engine.
I have seen teams solve this in three ways, and none of them is a tool. The first is the over-provisioning approach. Buy enough GPU capacity so that contention never happens. This works if your budget is effectively unlimited, which describes almost no enterprise. The second is the manual gating approach. Designate one person as the GPU allocator. Every team submits a capacity request. The allocator reviews it, queues it, and schedules it. This works for batch workloads with predictable cadence. It breaks immediately for real-time inference where demand varies by hour of day. The third is the chargeback approach. Track GPU consumption per team and make the cost visible in each team’s budget. When teams see that their experimental workloads cost the organization twelve thousand dollars a month in idle GPU time, they self-regulate. This works better than the other two, but it requires a cost attribution system that most enterprise AI platforms do not have.
The chargeback approach is the closest thing to a solution I have seen, and it is the pattern I recommend to most teams. You do not need a fancy scheduling algorithm to solve multi-tenancy. You need a billing system. When every team can see exactly how many GPU-hours they consumed, at what model size, at what latency tier, and at what dollar cost, the contention problem mostly solves itself. Teams optimize their own usage because the cost is visible and attributable. The data science team that was running four concurrent model evaluations realizes they could run them sequentially on one GPU and cut their bill by seventy-five percent. The infrastructure team does not need to be the GPU police.
The tools for this exist, but they are not packaged as a multi-tenancy solution. LiteLLM tracks spend per API key, which maps naturally to per-team cost attribution. The OpenAI-compatible endpoints that vLLM and SGLang expose all carry the user and team identifiers from the original request. The challenge is wiring those identifiers through the inference stack consistently and building the reporting layer that makes the costs visible. That is an organizational project, not a code project. The tools provide the data. The team provides the discipline to collect it and the organizational will to use it.
The layer between your inference engine and your internal customers is a routing layer that knows about teams, not just models. It inspects the incoming request, identifies the team, checks the team’s usage against its allocation and budget, decides whether to serve the request from the pooled cluster or queue it for off-peak processing, and logs every decision with enough detail to answer the question that will come up in the first week: why did Team B’s request get queued while Team A’s got served? The answer should be a single query against the routing layer’s logs, not a meeting over a shared spreadsheet.
If you are building an enterprise inference platform right now, the multi-tenancy question will hit you around month four. Plan for it by designing the routing and attribution layer before you need it. The teams that design it first have a clear growth path. The teams that skip it have a migration, a blame culture, and a VP asking questions nobody can answer.
If this was useful, forward it to one engineer who needs less noise in their feed.