Context, Not Computation: The Shifting Bottleneck in AI-Assisted Root Cause Analysis

A growing consensus among observability engineers indicates that the reasoning capabilities of large language models (LLMs) are no longer the primary impediment in AI-assisted root cause analysis (RCA). Instead, the more formidable challenge now resides in the intricate data pipelines responsible for delivering relevant information to these advanced models. This paradigm shift underscores a critical practical takeaway for teams integrating LLMs into incident response workflows: strategic investment in preparing and curating contextual data may yield more significant returns than merely pursuing larger or more sophisticated models.

The Evolving Landscape of Observability and Root Cause Analysis

Observability, a cornerstone of modern software operations, refers to the ability to infer the internal state of a system by examining its external outputs, such as logs, metrics, and traces. In today’s complex, distributed microservices architectures, ensuring high availability and performance hinges on effective observability. However, even with comprehensive monitoring, pinpointing the precise cause of an outage or performance degradation—a process known as root cause analysis—remains notoriously challenging. Traditional RCA often involves manual correlation of disparate data points, expert intuition, and time-consuming investigations, leading to prolonged mean time to resolution (MTTR) and significant business impact.

The advent of artificial intelligence, particularly machine learning (ML), began transforming observability by enabling automated anomaly detection, predictive analytics, and basic correlation. Yet, the promise of truly autonomous, intelligent RCA remained elusive. The emergence of large language models, with their remarkable ability to understand, summarize, and reason over vast amounts of textual data, ignited fresh enthusiasm. Engineers envisioned LLMs acting as digital detectives, sifting through logs, metrics, and alerts to identify fault patterns and suggest remediation steps with unprecedented speed and accuracy.

Two Dominant Approaches to AI-Driven RCA

Within the nascent field of AI-assisted RCA, two primary architectural philosophies have emerged: agent-based designs and deterministic designs.

Agent-based designs typically empower an LLM with a suite of tools and an investigative mandate. In this model, the LLM acts as an autonomous agent, reasoning about a problem, deciding what additional telemetry or diagnostic information to fetch, executing those queries or commands, and iteratively refining its understanding. This approach mirrors a human engineer’s investigative process, allowing the model to dynamically explore the data space. The allure of agent-based systems lies in their potential flexibility and ability to handle novel, unforeseen incidents by dynamically requesting information not explicitly provided initially.

Conversely, deterministic designs prioritize pre-correlation and context preparation. In this camp, signals from various sources—logs, metrics, traces, topology maps—are correlated upfront by a specialized "harness" or pipeline. This pipeline then synthesizes a single, focused, and highly curated context, which is subsequently presented to the LLM. The model’s role is primarily to reason over this pre-digested information, rather than actively fetching it. This method aims to simplify the LLM’s task by ensuring it receives only the most relevant and processed data, in a structured and digestible format.

The Industry Shift: Embracing Deterministic Context

Recent developments and practical experiences suggest a broader industry gravitation towards the deterministic, context-centric approach. Observability vendor Coroot’s work, for instance, distinctly reflects this shift. Their research posits that the fundamental question "can AI do RCA?" is misdirected, arguing instead that the two distinct jobs—reasoning over data and preparing that data—must be evaluated independently.

Coroot’s pipeline exemplifies this deterministic philosophy: it correlates signals into actionable "findings" and presents them to the LLM within a focused context, devoid of an agent loop. This design offers a crucial advantage: if the LLM provides an incorrect diagnosis, the blame can be squarely attributed to the model’s reasoning deficiency, rather than ambiguous evidence or the failure of an agent to retrieve critical information. This separation of concerns significantly aids in debugging and performance evaluation.

This trend is not isolated. Dynatrace’s highly regarded Davis AI, a veteran in the causal AI space, also leans heavily on deterministic, topology-based causal analysis. Davis AI constructs and traverses a real-time dependency map of the entire IT landscape to pinpoint root causes, leveraging its understanding of system relationships rather than unleashing an LLM in an open-ended investigatory loop. This approach, which predates the widespread adoption of LLMs, validates the power of structured, pre-correlated context in achieving reliable RCA. The inherent challenge with both approaches, however, is discerning whether a diagnostic failure stems from the model’s weak reasoning or from a flawed data harness that supplied inadequate or incorrect evidence.

Coroot’s Empirical Validation: The NetworkChaos Experiment

To rigorously test their hypothesis and isolate the LLM’s reasoning ability, Coroot engineer Nikolay Sivko devised a specific scenario. Utilizing Chaos Mesh, a cloud-native chaos engineering platform, Sivko injected a NetworkChaos experiment. This experiment deliberately introduced network delay between a catalogue service and its PostgreSQL database, leading to slow queries and cascading 502 errors on the front end.

Crucially, the context provided to the LLMs was designed to be challenging. It included not only direct symptoms but also deliberately misleading signals, such as query timings inflated by network round-trip time, which could potentially distract or misguide an LLM. The same prompt, approximately 9,800 tokens in length, was then presented to eleven different LLMs. Each model was tasked with identifying the root cause, outlining the cause-and-effect chain, and suggesting an immediate fix.

The results of this controlled experiment were illuminating. Leading frontier models, including Claude Opus 4.8, GPT-5.5 (presumably GPT-4.5 or a similar iteration, given the common naming conventions, though GPT-4 is the widely recognized current frontier model), and Gemini 3.1 Pro, all successfully passed the test. They accurately identified the NetworkChaos experiment as the root cause and correctly flagged the need to delete both the experiment and its schedule. Notably, several larger open-weight models also performed comparably well. Among self-hostable options, Gemma 4 31B emerged as a standout, successfully identifying the root cause. In contrast, other substantial open-weight models, such as Qwen3.6 35B and Qwen3 Coder Next, failed to pinpoint the issue.

These findings strongly suggest that, when presented with a well-prepared and focused context—even one containing distracting elements—leading LLMs possess sufficient reasoning capabilities to perform complex RCA. The implication is profound: the bottleneck is not in the LLM’s cognitive capacity, but in the preceding steps of data ingestion, correlation, and context distillation.

Operational Realities and Practitioner Perspectives

While agent-based approaches theoretically offer superior flexibility by allowing models to fetch unanticipated signals, practical operational challenges have tempered their enthusiasm. Reports from platforms like ZenML and Incident.io, which have explored multi-agent LLM investigations, highlight significant debugging difficulties in production environments. A failed run in an agentic system rarely produces a clean stack trace; instead, it manifests as unpredictable prompt interactions, emergent and often opaque coordination failures between agents, and a general lack of determinism. This fragility makes troubleshooting and improving such systems exceedingly complex.

This operational friction has led many practitioners to reconsider purely agentic designs. Discussions on platforms like Reddit among experienced developers reveal a common sentiment: many teams are scrapping fully agentic architectures in favor of mostly deterministic workflows, where LLMs are integrated as a narrow, specific step. The rationale is clear: deterministic pipelines offer superior reliability, predictability, and significantly lower token costs because the LLM is only invoked with pre-processed, compact data. This trade-off of some flexibility for enhanced repeatability and cleaner evaluation further solidifies the industry’s view that the "harness"—the data pipeline—is now the hardest and most critical part of AI RCA.

The Economic and Engineering Implications

The shift towards context engineering also carries significant economic advantages. Sivko’s research noted that a single, short LLM call, even to a frontier model, typically costs only a few cents because the intensive correlation and context preparation work occurs before the model is invoked. This cost efficiency further strengthens the case for optimizing the data pipeline rather than solely focusing on acquiring ever-larger or more expensive LLMs.

This conclusion redirects engineering efforts. Instead of pouring resources into model selection, fine-tuning, or the development of more sophisticated agentic frameworks, the emphasis shifts to building robust, intelligent data pipelines. These pipelines must excel at collecting, correlating, filtering, and summarizing vast streams of observability data into compact, high-signal contexts. This framing echoes a wider movement around "context engineering" or "prompt engineering" but applied at a systemic level rather than just individual prompts. Guidance from AI leaders like Anthropic (on effective context engineering for AI agents) and LLM framework providers like LangChain (on context engineering for agents), along with observability vendors like Mezmo (on delivering the right data to LLMs), all converge on this fundamental idea: curating the smallest, most relevant set of high-signal, compact context is rapidly becoming a core discipline for ensuring the reliability and effectiveness of LLM-based reasoning in observability.

The Future of AI-Assisted RCA: A Human-Centric Design

While the technical focus has shifted, the ultimate goal remains the same: to empower human engineers with faster, more accurate insights during incidents. The future of AI-assisted RCA likely involves a sophisticated interplay between intelligent data pipelines and human expertise.

The human element remains critical in designing these pipelines, defining what constitutes "relevant" context, and interpreting the LLM’s output, especially in ambiguous or novel situations. AI acts as a force multiplier, reducing the cognitive load and accelerating the diagnostic process, but it does not fully replace human judgment.

The ongoing challenge will be to build data pipelines that are not only deterministic and reliable but also adaptive. While predefined correlation sets work well for known incident patterns, true resilience requires systems that can still identify and contextualize signals from novel incidents that fall outside any pre-established schema. This may involve hybrid approaches where deterministic pipelines handle the majority of cases, with provisions for human-in-the-loop intervention or carefully constrained agentic components for extreme outliers.

In conclusion, the journey of AI-assisted root cause analysis has reached a significant inflection point. The initial fascination with raw LLM power has matured into a pragmatic understanding: the intelligence of the system is ultimately constrained by the quality and relevance of the information it receives. The true frontier of innovation now lies not in making models smarter, but in making the data smarter—by engineering sophisticated, reliable, and cost-effective pipelines that deliver precisely the right context, at the right time, to the right model. This emphasis on the "harness" over the "horse" marks a pivotal evolution in how organizations will leverage AI to maintain the health and performance of their increasingly complex digital infrastructures.