Closing the Loop: Feedback, Human Review, and Continuous Improvement for RAG

A golden test set catches regressions before release. Tracing and dashboards show whether the system is healthy right now. Neither one tells you what to actually fix next, and neither one gets better on its own.

A golden test set built once at launch goes stale. User questions drift, new documents get added, and the exact failure cases that matter in production are, by definition, not the ones anyone thought to write down in advance. Without a path from real usage back into the evaluation set, quality work stops the day the system ships.

That path is the feedback loop, and it's the piece most RAG systems skip entirely.

The Loop, End to End

production conversation
  -> user downvote or low judge score
  -> review queue
  -> reviewer confirms correct source
  -> promoted to golden test set
  -> included in every future evaluation run

Each stage is simple on its own. The value is in connecting them so a bad real-world answer becomes a permanent regression test, not a one-time apology.

Capturing Feedback Without Getting in the Way

Feedback capture has to be nearly free for the user, or it won't happen:

• thumbs up / down on every chat response, no required comment
• an optional free-text reason on a downvote, never mandatory
• a "flag for review" action available to operators reviewing conversations directly
• an automatic flag when a sampled judge score (from the evaluation layer) falls below threshold

That last one matters as much as explicit user feedback. Most users don't downvote a subtly wrong answer — they just stop trusting the system. Automatic flagging from evaluation scoring catches cases silent users would never report.

The Review Queue

Flagged conversations land in a queue, not directly in the golden set. A human still confirms the failure before it becomes a permanent test case — an unconfirmed downvote might just mean the user asked an unanswerable question, not that the system was wrong.

A useful review queue shows, for each item:

• the question, the retrieved chunks, and the generated answer
• the signal that raised it: downvote, low judge score, or manual flag
• reviewer actions: mark acceptable, mark defective, add a note

This is the same information the tracing layer already captures per query. Observability and feedback share a data model on purpose — a flagged conversation should open directly into its trace, with no separate lookup required.

Reviewer Actions and What They Mean

That third row matters. Not every bad answer is a retrieval or prompt bug. Sometimes the correct information was never uploaded in the first place, and no amount of pipeline tuning will fix a gap in the underlying documents. A good review workflow routes that case differently instead of quietly closing it.

Promotion to the Golden Test Set

Once a reviewer confirms both the failure and the correct source, promotion is mechanical:

{
  "id": "gq_0143",
  "question": "What is the refund window for annual plans?",
  "expected_document_id": "doc_8831",
  "expected_chunk_ids": ["chk_00291"],
  "category": "billing",
  "source": "promoted_from_feedback",
  "created_at": "2026-09-19T00:00:00Z"
}

From this point on, every future evaluation run — including the CI regression gate covered earlier in this series — includes this exact case. The fix isn't just applied once; it's protected against ever silently regressing again.

Keeping the Golden Set Honest

A test set that only grows eventually gets slow to run and full of duplicates. A few simple rules keep it useful:

• deduplicate near-identical questions before promoting
• retire entries tied to documents that no longer exist or were superseded
• periodically re-review a sample of older entries — what counted as a correct answer six months ago may no longer match the current product

The golden set is a living dataset, not an archive. Treat changes to it with the same review discipline as changes to code, since a bad entry silently sets the wrong bar for every future evaluation run.

What "Done" Looks Like

The loop is working when a bad answer, once found, cannot happen unnoticed again:

1. a user or the evaluation layer flags a conversation
2. a reviewer resolves it in minutes, not days, because the trace and context are already attached
3. a confirmed defect becomes a permanent regression test
4. the next pipeline change is checked against it automatically, in CI, before it ships

That is the difference between a RAG system that was good on launch day and one that stays good as content, usage, and models keep changing underneath it.

Final Takeaway

Evaluation catches known problems before release. Observability surfaces unknown problems in production. The feedback loop is what turns those unknown problems into known ones — permanently. Without it, every hard-won fix is temporary, and quality quietly erodes as the system evolves.

This closes the arc that began with getting documents into the system at all: ingest it, retrieve it, answer with it, measure it, watch it, and keep it honest as real usage keeps teaching the system what it got wrong.

For the full implementation specification behind this three-part arc, see the separate project recipe in the repository docs folder at docs/RAG_QUALITY_SPEC.md.

Previous: Observability for RAG: Logging, Tracing, and Catching Failures in Production