Observability for RAG: Logging, Tracing, and Catching Failures in Production

A golden test set tells you whether a RAG system is good before it ships. It does not tell you whether the system is healthy right now, for the actual users hitting it in production. Those are two different problems, and most teams only solve the first one.

A RAG pipeline has more moving parts than a typical request: embedding calls, vector search, filtering, reranking, prompt assembly, and generation, often across multiple external services. When an answer is slow, wrong, or empty, "check the logs" usually means digging through unrelated lines with no way to tell which stage actually failed.

Observability fixes that by giving every query a trace, and every stage a span.

Why Generic APM Is Not Enough

Standard application monitoring tells you a request took 1.3 seconds and returned a 200. It does not tell you that vector search returned zero candidates, or that the reranker silently dropped the one relevant chunk, or that the model generated an answer with no supporting context at all.

RAG-specific observability needs to capture pipeline-specific signal:

• how many candidates came back from retrieval
• which chunks survived filtering and reranking
• how many tokens the prompt used and from which sources
• whether the model had to fall back to "I don't have enough information"

None of that is visible in a request duration and a status code.

One Trace, One Span Per Stage

The right unit of observability for RAG is a trace per query, with one span per pipeline stage, all correlated by a single request ID.

trace: req_4471
  span: query_embedding        12ms
  span: vector_search           34ms   candidates=20
  span: metadata_filter          3ms   remaining=14
  span: rerank                  81ms   selected=6
  span: prompt_assembly          4ms   tokens=2140
  span: llm_generation        1180ms   input=2140 output=310
  span: total                 1314ms

This structure turns "the answer was slow" into "reranking took 81ms and generation took 1180ms" and turns "the answer was wrong" into "only 14 candidates survived filtering, and the correct chunk was not among them."

What Belongs in Each Span

Every span should record enough to diagnose a problem without re-running the query:

• stage name and duration
• input and output size (candidate counts, token counts)
• the model or component version used
• errors or fallbacks triggered

For vector_search and rerank specifically, record the actual chunk IDs and their scores. That is usually the single most useful piece of debug data in the whole trace — it turns "retrieval seems off" into "chunk 00291 scored 0.71 and was dropped in favor of three unrelated chunks from a different document."

Per-Query Summary Logging

Alongside spans, log one summary record per query:

• tenant and category
• final chunk IDs used in the prompt
• prompt version and model name
• answer length and citation count
• a fallback flag, set when the model had to answer with insufficient context
• total latency and token cost

This summary is what powers dashboards. The spans are for deep debugging one query; the summary is for trends across thousands of them.

Building the Dashboard

A useful RAG dashboard answers operational questions at a glance, not just "is the server up":

Fallback rate deserves special attention. A rising fallback rate is often the earliest signal that something upstream broke — a source went stale, an index rebuild silently failed, or a metadata filter got too strict — well before users start complaining.

Alerting Without Noise

Threshold-based alerts on rolling windows cover most of what matters, without needing an anomaly-detection model:

• fallback rate above a configured threshold
• p95 latency above a configured threshold
• error rate for any single stage above a configured threshold
• a sampled faithfulness pass rate (from ongoing evaluation) dropping below threshold

Keep thresholds tenant-aware where usage patterns vary widely. A fallback rate that is normal for a sparse knowledge base looks alarming for a dense one, and a single global threshold will either miss real problems or page someone every night for nothing.

Observability Must Never Break the Pipeline

Logging and tracing are diagnostic infrastructure, not core functionality. They must degrade gracefully:

• a logging failure must never fail or slow down the actual chat response
• tracing writes should be asynchronous and best-effort
• a full observability outage should be visible in its own right, not silently swallowed

The chat response path stays the priority. Observability watches it; it never gates it.

From Traces to Action

Traces and dashboards are only useful if they connect to something an operator can act on. In practice, the same fields that make a trace useful for debugging are exactly what should route into a review queue when something goes wrong: the query, the retrieved chunks, the assembled prompt, and the final answer.

That connection — from an observed failure to a corrected system — is the last piece of the loop.

Final Takeaway

A RAG pipeline has too many stages and too many external dependencies to debug from generic request logs. Per-stage tracing turns a vague "the answer was bad" into a specific, attributable cause, and per-query summaries turn thousands of individual traces into trends an operator can act on before users notice.

Knowing something is wrong is still not the same as fixing it. The next article covers the last piece: turning user feedback and flagged conversations into a review workflow that continuously improves the system.

Previous: Evaluating RAG Systems: Metrics for Retrieval and Answer Quality

Continue with: Closing the Loop: Feedback, Human Review, and Continuous Improvement for RAG