RAG · Walmart Global Tech
RAG Diagnostics Pipeline
Problem
Monitoring agents could see telemetry but lacked the operational and historical context to diagnose why something was failing.
Approach
Engineered retrieval-augmented pipelines integrating ServiceNow and internal APIs as LLM knowledge sources, surfacing real-time operational context and prior incident signal at inference time.
Impact
Agents reason over live operational state and institutional memory — producing diagnoses grounded in what's actually happening, not just raw metrics.
Detection without diagnosis is half a system
The agentic monitoring layer could spot anomalies, but an anomaly without context is just a louder alarm. Diagnosing why a host is failing requires what experienced SREs carry in their heads: historical incidents, known failure patterns, runbooks. That institutional memory had to become queryable.
ServiceNow as a knowledge source
The pipeline ingests ServiceNow ticket history and internal API data, chunks it — by ticket, then by resolution notes and symptom description — embeds it, and makes it retrievable at inference time. When an agent detects an anomaly, it queries with the current context (host, metrics, error signature) and retrieves the most relevant prior incidents and runbook snippets.
Retrieval quality is the product
Top-k retrieval gated by a similarity threshold, validated against known historical incidents — if retrieval is wrong, the diagnosis is confidently wrong, which is worse than no diagnosis. RAG was the right call over fine-tuning here: incident data changes constantly, and re-indexing is cheap while re-training is not.
Outcome
The LLM produces grounded, pre-analyzed diagnostic notes instead of generic summaries. Agents surface actionable findings with citations into the incident history — SREs arrive knowing what's wrong and what worked last time.