What Your AI Observability Stack Should Look Like
A Production Architecture
The composable pattern that ties Phoenix depth, Langfuse compliance, and WhyLabs reporting into a single operating picture.
Over the last week, I covered three observability tools in this publication. Arize Phoenix for deep agent tracing. Langfuse for prompt management and GDPR compliance. WhyLabs for the executive reporting layer that makes quarterly reviews survivable. Each article made a standalone argument for the tool. What I did not do in any of them is describe how the three fit together in the same enterprise.
The honest starting point is that most teams should not run all three. Running three observability platforms creates its own operational burden, and the integration tax of keeping them in sync often cancels the benefit of having each tool specialize. But for the enterprise that has grown past the point where a single tool covers the observability surface, the three-layer architecture I am going to describe answers questions that no single tool answers alone.
The architecture has three layers, and they form a pipeline that data flows through from left to right.
The first layer is deep tracing with Arize Phoenix. Phoenix sits closest to the inference path. It receives OpenTelemetry spans from every LLM call, every tool invocation, every retriever step, and every agent decision that your system produces. LiteLLM, LangChain, LlamaIndex, the OpenAI Agents SDK, the Claude Agent SDK, and every framework that speaks OpenInference or OTLP pipes directly into Phoenix through the OpenTelemetry collector. The spans form a tree that mirrors the agent’s execution structure, and that tree is what the engineering team uses when a production agent produces an unexpected output and they need to reconstruct the decision chain. Phoenix is the tool for debugging the edge case that happens once and costs a thousand dollars in compute to reproduce.
The Phoenix instance runs self-hosted on a single VM or Kubernetes cluster, backed by a Postgres database for trace storage. For enterprises with data residency requirements, this is the layer that never leaves the VPC. The raw model outputs stay inside the trust boundary. The traces contain the actual prompts and responses because you cannot debug an agent trajectory without seeing what the agent said and received.
The second layer is prompt management and compliance with Langfuse. Langfuse receives a subset of the data that Phoenix receives. It does not need every span. It needs the LLM call metadata, the prompt version identifiers, the response content for compliance logging, and the evaluator scores. The integration pattern is a dual instrumentation where your application code sends spans to both the Phoenix OpenTelemetry collector and the Langfuse Python SDK. For most frameworks, this is two import statements and two initialization calls. The observation overhead is negligible because both instrumentations piggyback on the same request path.
Langfuse serves a different audience than Phoenix. The engineering team uses Phoenix for deep debugging. The platform team and the compliance officer use Langfuse for prompt version tracking, A/B experiment comparison, and the audit trail of what model configuration was in production at any point in time. The prompt management workflow in Langfuse is deeper than what Phoenix provides, and the compliance posture matters for any team operating under GDPR or similar frameworks. Langfuse’s EU-hosted SaaS option means that European enterprises can keep compliance-sensitive trace data within EU jurisdiction without self-hosting another infrastructure component.
The third layer is executive reporting with WhyLabs. WhyLabs receives the most abstracted view of the system. It does not need individual traces or prompt versions. It needs statistical profiles of model inputs and outputs, cost aggregates per team and per model, and drift metrics that indicate whether the system’s behavior is shifting over time. The whylogs library generates compact statistical sketches from the data streams that pass through your pipeline. Those sketches are privacy-preserving by design they contain distributional information but not individual records. They upload to the WhyLabs platform, which generates the dashboards that the CFO and CISO read before the quarterly review.
WhyLabs is the layer that answers the questions nobody in engineering wants to answer. How much did we spend on inference last quarter, broken down by team and by model? Is any of our model deployments drifting away from the validation distribution? Has data quality degraded in any pipeline that feeds a regulated decision? These questions do not require span-level traces. They require trend data, aggregate statistics, and a view that abstracts away the individual request and shows the system’s health as a business asset.
The split is not ideal. Nobody wants to maintain three observability platforms. But the specialization across these three tools reflects something real about the enterprise AI observability problem: there is no single tool that does deep tracing, compliance-grade prompt management, and executive reporting equally well. The market has not converged on a unified platform, and the tools that try to cover all three usually compromise on at least one dimension. A team that tries to use Phoenix for everything ends up building its own prompt management workflow because the Phoenix implementation, while functional, was added recently and lacks the maturity of Langfuse’s dedicated approach. A team that tries to use Langfuse for everything discovers that the flat trace view cannot reconstruct complex agent trajectories. A team that tries to use WhyLabs for everything finds that the profile-based approach does not provide the per-request fidelity needed for debugging edge cases. Each tool has a core strength, and the architecture works because it routes each observability use case to the tool that handles it best.
The integration that most teams skip is the one that ties all three layers back to the business metrics that already exist in the organization. Phoenix produces traces. Langfuse maintains prompt versions. WhyLabs generates drift and cost reports. None of them, by default, connects those observability metrics to the business outcomes the AI system is supposed to drive. The engineering team sees that latency increased by 200 milliseconds. The finance team sees that spend went up by fifteen percent. Neither view answers the question of whether the latency increase or the cost increase was worth the measured improvement in the business metric the system was built to optimize. That connection requires a custom integration: exporting the observability metrics from Phoenix, Langfuse, or WhyLabs into the organization’s existing analytics or BI tooling, where they sit alongside revenue data, customer satisfaction scores, or operational efficiency metrics. Datadog, Grafana, Snowflake, Tableau, whatever the organization already runs for business intelligence is the right home for the combined view.
The teams that build this integration are the ones that can answer the hardest question in enterprise AI: is this system delivering measurable business value, or is it just running. Without the connection to business metrics, each layer of the observability stack tells you whether the system is healthy but not whether it is working. The two are not the same thing, and the difference determines whether AI observability is a cost center or a strategic function within the organization.
The smallest viable version of this three-layer architecture is Phoenix self-hosted on a single VM for deep tracing and Langfuse cloud for prompt management and compliance. That covers debugging and governance with two tools and a few hours of setup. WhyLabs enters the picture when the organization reaches the point where someone in finance or compliance asks for a report that spans multiple teams and multiple models. That question usually arrives between month three and month six of a production AI deployment, and having WhyLabs already instrumented with whylogs profiles means the answer is ready before the question is asked.
The architecture composes because each layer receives a different grain of data. Phoenix gets the raw traces. Langfuse gets the structured metadata and prompt versions. WhyLabs gets the statistical aggregates. The pipeline from raw trace to executive dashboard is a data flow where each stage reduces detail and increases abstraction. That is not a weakness. It is the pattern that makes the architecture work across audiences that have fundamentally different information needs. The engineer debugging an agent trajectory needs the raw trace. The platform team managing prompt rollouts needs the version history. The executive approving the next quarter’s AI budget needs the trend line and the cost breakdown. The three needs do not conflict. They operate on the same data at different levels of resolution.
If you are building your AI observability stack right now and wondering whether you need all three, the honest answer is probably not. Phoenix alone covers most teams through the first year of production. Add Langfuse when the compliance requirement or the prompt management workload justifies the second tool. Add WhyLabs when the business case requires dashboards that speak finance’s language. The architecture I described is not a shopping list. It is a growth path that maps to the stages an enterprise observability practice passes through as the system scales.
The decision that will define the next six months is not which observability tool to pick. It is whether you build the connection between your AI metrics and your business metrics before someone asks for it or after. Most teams build that connection after the question arrives and the data to answer it is incomplete because nobody instrumented it. The teams that build it before the question arrives are the ones that run AI like a business function, not a science project.
If this was useful, forward it to one engineer who needs less noise in their feed.