Recovery should preserve the evidence of what failed before the answer arrived.
A fallback can keep the service working while the router gets worse. The user receives an acceptable answer, the availability chart stays green, and the primary destination keeps failing underneath. Automatic recovery has done useful work. It has also removed the symptom that would have forced somebody to investigate the first choice.
Yesterday’s scorecard asked whether a route deserves traffic. Once it has traffic, the fallback path can change what the scorecard appears to measure. A request configured for an economical model may finish on a more capable replacement. If the report credits the requested model with the delivered result, the operator learns that the economical route is reliable while paying for a different service. The unit of observation needs to survive the switch.
Consider a hypothetical assistant that answers engineers’ questions from approved manuals. Its primary deployment handles routine lookups; a second approved deployment can recover transient failures. The acceptance check requires the applicable document revision, supported claims, and delivery before a deadline. These are proposed operating conditions, not a deployment or benchmark performed here. Recovery is valuable only while those conditions remain intact.
LiteLLM’s fallback documentation gives the mechanism a concrete shape: a call that fails after configured retries can move to another model group. It also documents separate context-window and content-policy fallback settings. Those branches deserve separate review because their causes differ. A larger context window may address an oversized evidence packet; an alternate provider does not automatically resolve a policy restriction or an answer that lacks support.
The gateway sees a provider error more readily than a plausible wrong answer. An answer with unsupported citations can return successfully and require application-level rejection. Record that rejection as a quality failure before choosing a remedy. Missing source material should lead to an evidence request or owned hold; a stronger model cannot retrieve a document the application never supplied. Treating every failure as an invitation to spend more conceals defects outside inference.
Start with one request identifier and attach each attempt to it. Preserve the requested route, policy version, attempted destination, selection reason, attempt number, failure class, fallback destination, and final serving destination. Record acceptance separately, including the rubric version and whether correction was required. Timing and cost belong to each attempt and to the completed request. A final success event should close the history, not replace it.
This does not require keeping every prompt in an analytics database. Use protected artifact references and bounded error codes where those provide enough diagnostic evidence. Limit access and retention for sensitive content. Keep request identifiers in traces or detailed records rather than turning each identifier into a metric label. The useful aggregate answers are how often a route needed rescue, what caused it, and whether the rescue preserved the task contract.
Cost attribution needs both destinations. Charge the primary policy for the complete sequence it initiated, while retaining each deployment’s actual consumption for billing and diagnosis. A timed-out attempt may have incurred charges even when no answer arrived; reconcile known usage with billing and label unresolved charges. If the dashboard counts only the final model’s tokens, it loses wasted work. If it assigns every charge to the requested model, it hides where the money went.
The latency distortion is easier to see with constructed timing. Suppose a primary attempt times out after four seconds, backoff takes one second, and the fallback finishes in two seconds. With no other work or overlap, the user waited seven seconds. Reporting the final call as a two-second response removes the delay that triggered it. These numbers illustrate accounting, not measured model performance, and actual end-to-end timing must include retrieval and validation too.
Build latency distributions from completed request durations rather than adding provider percentiles. A percentile from one population cannot be added to a percentile from another to recover the combined request tail. Separate direct completions, fallback completions, failed requests, and deadline misses, then show the overall workload distribution as well. Google’s SRE guidance distinguishes successful and failed request latency and warns that averages conceal slow tails. A fallback that eventually returns can still violate the deadline.
Provider comparisons need the same care. The fallback population contains requests that already failed somewhere else; it may carry longer inputs, prior delay, or harder questions. Comparing its acceptance rate with the primary’s easy direct completions confounds the destination with the work it received. Retain those operational cohorts, but use controlled paired cases with the same permitted evidence and rubric when comparing capabilities. A rescue rate is not a clean model ranking.
An increase in fallbacks is therefore a diagnostic question, not an automatic indictment. A brief provider outage, a changed traffic mix, a bad context limit, and a poorly chosen primary route can all raise it. Inspect failure classes and task slices before changing the router. Compare with a baseline that accounts for time of day and load when those matter. Tie an investigation threshold to the workflow’s tolerable cost and delay rather than copying a universal percentage.
Retry ownership matters when several layers can recover. The client, application, gateway, and provider SDK may each retry without seeing the others’ budgets. Amazon’s guidance on timeouts and retries explains how layered retries multiply load and how backoff and jitter reduce synchronized pressure. Give the request an overall deadline and a bounded recovery budget, with one layer responsible for coordinating attempts. Do not begin another call when too little time remains to complete and validate it.
A timeout also leaves an uncertain outcome when the route can execute tools. The first attempt may have performed an operation before its response disappeared. A fallback must reconcile that state or use an idempotency mechanism before repeating the operation. Otherwise, the final answer can look successful while two changes occurred. Keep dispatch authority and side-effect verification outside a model’s assumption that the previous attempt failed.
The strongest objection is sensible: users should not absorb a provider incident when an approved backup can finish their work. Keep that recovery path. Visibility need not mean exposing an implementation log in the interface or paging somebody for every retry. Page on user impact or threatened capacity; use route-level records to investigate concealed degradation. Google explicitly identifies internal monitoring as a way to detect failures masked by retries. Recovery can protect the user while still leaving the operator evidence.
Exercise that evidence path in a controlled environment. Make the primary return a transient error, exceed its context limit, produce an unsupported answer, and time out after a simulated side effect. State the expected fallback, hold, or denial for each case, then check both the final result and the complete attempt history. These are proposed fixtures, not tests executed here. Verify that a recovered request retains its original reason and failure, and that an exhausted budget leaves an owner and next state.
Repeated rescue should eventually force a routing decision. Repair the configuration when the evidence points to a configuration defect; move an eligible task slice directly to the replacement when comparative evaluation justifies it. Hold work when no permitted destination clears the requirements. A backup that carries the service indefinitely may have become the real primary, and the policy should acknowledge that change. A fallback earns the name resilience when the operator can see what it rescued and fix what keeps needing rescue.
Subscribe to Signal Over Noise for practical analysis of AI agents and automation: what to test, which constraints matter, and when a tool is worth using. Bring the next idea to one bounded workflow and check the result before expanding it.
If this was useful, forward it to one engineer who needs less noise in their feed.