Observe the decision before handing it authority.
A decision gate can spend a week in production without preventing a single bad action, then get promoted because nobody reported an incident. That is not evidence of readiness if the gate was only observing. The useful question is what it would have changed, which of those changes were justified, and what the observation system failed to capture. Shadow mode should create an answerable comparison before a score acquires authority. It should not become a waiting period that ends when the dashboard looks calm.
The distinction is concrete in SageMaker’s production-validation contract. Requests can be replicated from a production variant to a shadow variant, whose responses are logged rather than returned to the caller. A production split instead returns the selected production variant’s response. That contract supplies a useful boundary, not a complete deployment design for an action-taking agent. Copying an inference request is different from replaying the workflow that could charge a customer, send a message, or modify a record.
Consider a hypothetical support gate proposing whether a ticket should go directly to billing or remain in general triage. This is a proposed architecture, not a service I deployed or measured. The existing routing process remains authoritative. A separate observer receives permitted decision-time inputs and records its own proposed destination without moving the ticket. It receives no write credential and cannot enqueue real billing work. Existing deterministic prohibitions remain enforced; shadowing a candidate probabilistic rule never suspends a known security boundary.
Isolation must include dependencies, not just the returned answer. Use an asynchronous bounded copy, separate resource limits, and an explicit rule for dropping shadow work under pressure rather than delaying the live service. Record those drops. Google’s SRE discussion of traffic teeing warns that shared state can distort comparisons, including performance measurements through a shared cache. My design inference is to inspect every tool reachable from the shadow worker and replace effectful operations with a nonexecuting proposal sink. Turning off the last response does not neutralize an earlier side effect.
Create the observation at the decision boundary. In this design, retain an operation identity, decision timestamp, approved input reference, proposed action, predicted event, probability or raw score with its exact meaning, threshold revision, threshold result, abstention reason, and model or prompt revision. Keep probability unavailable when the system supplies only an uncalibrated score. Record inference latency, queue delay, timeout, and whether the result arrived before the live decision. A judgment made with information that appeared later cannot be presented as a timely alternative to the original action.
Join that record to what production actually did and to an independently defined outcome. The human’s selected queue is an observed decision, not automatically the correct label. A reviewer can be wrong, policy can be ambiguous, and a ticket can legitimately involve more than one team. Establish a labeling rubric and an adjudication owner before counting agreement. Preserve label revisions rather than replacing yesterday’s disputed answer with today’s settled one and hiding the change. The observer’s recommendation should not be shown to ordinary operators during this stage, or their decisions no longer provide an unaffected comparison.
Delayed outcomes need their own clock. Record decision time, outcome time, label-availability time, and the join identity separately. AWS’s ground-truth merge documentation illustrates matching predictions to later labels by an identifier; its label-collection date need not equal inference date. The page also limits Model Monitor to existing customers, so this is an evidence-design reference, not an adoption recommendation. In the proposed gate, report mature labeled cohorts separately from recent arrivals awaiting resolution. Missing labels remain missing, not correct by default, and the report needs the fraction and slices for which outcomes are still unknown.
Sampling changes the question the report can answer. Keep a representative traffic sample to estimate ordinary behavior and a separately identified challenge sample for rare, costly, ambiguous, or near-threshold cases. Preserve selection rules and inclusion probabilities where applicable. Oversampling disagreements is useful for diagnosis; treating that set as the natural workload distorts the denominator. Compare eligible arrivals with copied, completed, timed-out, dropped, and labeled observations. A quiet report assembled only from easy requests completed during low load cannot justify enforcement across every request that reached the live service.
Privacy belongs at capture time. The comparison does not require an unrestricted second copy of every ticket, credential, attachment, or private conversation. In this design, store the minimum permitted features, scoped references to protected evidence, access roles, and a retention expiry. OPA’s decision-log documentation explicitly supports removing or modifying sensitive input and result fields before log upload. It also documents dropped events under configured rate limits. The practical lesson is to verify the saved record and its completeness counters, not assume that enabling logging produced a safe, exhaustive audit trail.
A counterfactual field must not manufacture a counterfactual outcome. The record can say that the candidate would have routed a ticket to billing while production retained it in triage. It can compare that proposed destination with an adjudicated ownership label. It cannot directly observe how long that unchosen billing path would have taken or whether the customer would have responded differently. Mark those outcomes unobserved. Any estimated benefit needs stated assumptions and a suitable evaluation design; agreement alone does not establish saved time, prevented harm, or causal improvement. A later controlled rollout may answer some of those questions, but only within its authorized scope.
Review disagreements in both directions. Candidate allows while production holds, candidate holds while production allows, and both agree on a wrong disposition are different findings. Sample agreements as well as exceptions so the review is not blind to shared mistakes. Inspect error cost, accepted-case coverage, unresolved labels, and relevant traffic slices beside the totals. Support a probability claim with labeled evidence rather than operator agreement: scikit-learn’s calibration guide describes comparing predicted probabilities with observed positive frequencies and retaining bin support. Shadow traffic can supply evaluation evidence; continuously tuning on it does not leave the same traffic untouched for final acceptance.
Promotion should change one authority at a time. My proposed sequence begins with invisible observation, then a separately evaluated warning that operators may dismiss, then a bounded confirmation requirement with an owned queue, and only then blocking for supported cases. Warnings can influence people; confirmation can delay legitimate work. Each stage therefore needs its own success and rollback conditions. Do not require every operation to graduate to blocking. Some decisions should remain advisory, and some prohibitions should stay deterministic regardless of the model’s result.
Before granting the next mode, freeze the candidate and require an explicit review of representative labeled evidence, critical disagreements, missingness, latency under the intended load, privacy controls, and the capacity of the nonautomatic path. Name the person authorized to promote it and the person able to disable it. Keep current permissions and safe fallback behavior outside the observer. A model, prompt, threshold, or workload change should reopen the relevant comparison. A week with no visible incidents can justify continuing the observation; it cannot answer the promotion question by itself. Ask which decisions would have changed, what supports those changes, and which consequences remain unobserved before handing the gate a key.
Subscribe to Signal Over Noise for practical analysis of AI agents and automation: what to test, which constraints matter, and when a tool is worth using. Bring the next idea to one bounded workflow and check the result before expanding it.
If this was useful, forward it to one engineer who needs less noise in their feed.