This content originally appeared on HackerNoon and was authored by Tahir Nawaz
Most teams evaluating agentic RAG stop at one number. They run a faithfulness check against the final answer, see 0.83, and move on.
That number does not tell them which retrieval round produced the bad chunk, whether the agent’s own critic flagged it before generation, or how many tool calls were wasted on queries that returned nothing useful. The single score smooths over every interesting failure mode in the pipeline.
I have spent a fair amount of time on RAG eval and the gap is clear: evaluation is the part most teams build last, and it is the part that decides whether the system holds up in production. A demo agentic RAG can be evaluated with vibes, but a production one cannot.
I used to think evaluation was mostly a tooling problem. Most of the time it turned out to be a measurement problem. We were measuring the wrong unit, on the wrong cadence, with judges we had not calibrated. My opinions on which signals matter have shifted more than once, and I expect them to keep shifting.
This article walks through the metrics and tools I currently trust once your RAG is doing more than a single retrieve then answer step. It covers RAGAS for component scores, LangSmith and Langfuse for node-level traces, critic scores for the agent’s self-judgement, retrieval round behavior, latency and cost as eval metrics, and how to build an eval set that catches real regressions. Some sections go deep because I have opinions; others stay short because the topic is mostly obvious once you see it.
1. Why agentic RAG breaks single-score evaluation
A vanilla RAG pipeline has two parts to score: the retriever and the generator. End-to-end correctness is hard, but at least the shape is well defined.
One query, one retrieval, one answer.
Agentic RAG does not look like that. The agent can decide to retrieve once, twice, or not at all. It can rewrite the query, run a web search, call a tool, escalate to a human, or refuse to answer. A single trace may contain four retrieval rounds, two critic checks, one fallback to web search, and a final answer. Scoring that trace with one faithfulness number throws away most of the information.
I kept seeing teams report 0.85+ faithfulness on agentic systems where, when I actually opened the trace store, a meaningful share of queries had three or four wasted retrieval rounds before the system stumbled onto something workable. The metric was happy. Nobody who looked at the traces was.
A few specific shapes of the same problem:
- Example 1: false pass on a long trace. A user asks “Which customers had failed payments last month?” The agent makes three retrieval rounds across payment data and support tickets. The final answer is well-cited and faithful to the retrieved context. The score is 0.91. What the score does not show is that the first two retrieval rounds returned nothing useful and added four seconds of latency. The system passed the metric and the user got a slow answer.
\
- Example 2: false fail on a short trace. A different user asks the same kind of question. The agent retrieves once, decides the results are weak, and answers “I don’t know which customers had failed payments in that period; the data I have access to only covers the last seven days.” This is the correct behavior. End-to-end faithfulness scores it low because the answer does not match the ground truth. The eval punishes the agent for refusing to hallucinate.
\
- Example 3: faithful answer to outdated information. A user asks “What is our refund policy for EU customers?” The retriever finds nothing EU-specific in the corpus and the agent escalates to web search. The web search returns a cached marketing page from 2023, before the policy was updated in 2025. The agent generates an answer that is fully grounded in the retrieved page. Faithfulness is 0.94. The user gets a confidently wrong answer. The metric did not catch it because faithfulness only asks “is the answer grounded in the retrieved context?”, not “was the retrieved context the right context?”
\
- Example 4: the agent talking to itself. A user asks a complex multi-hop question. The agent’s query rewriter is broken: every rewrite preserves the same misspelled keyword the user typed. The critic keeps rejecting the retrieval. The agent keeps retrying with the same effective query. After five rounds it hits the retry limit, generates a fallback “I cannot find this information” answer, and stops. End-to-end faithfulness either skips the trace or returns a vacuous high score because the fallback answer has no factual claims to verify. The pipeline burned roughly eight LLM calls and produced nothing useful. No single-score metric catches this.
All four failures share a root cause: the score was computed over the wrong unit. Nothing about that is unique to agentic RAG. What’s different is the surface area, and the way these failures hide inside loops that produce a passable final string.
The rest of this article is mostly about what to measure instead, with some opinions about which signals I trust more than others.
2. RAGAS for component metrics
I default to RAGAS for component scoring because the metrics it computes match how RAG actually fails. It is built around the same separation that matters for any RAG system: score the retriever and the generator independently before looking at the final answer.
The four metrics that most teams actually use:
- Context precision. Of the chunks you retrieved, how many were actually relevant to the question? High precision means the retriever is not flooding the model with noise.
- Context recall. Of the relevant chunks in your corpus, how many did you retrieve? Recall requires a ground-truth answer to compare against, so it is more expensive to build but more informative.
- Faithfulness. Of the claims in the generated answer, how many are grounded in the retrieved context? A low faithfulness score means the generator is making things up that are not in the chunks.
- Answer relevancy. Does the answer actually address the question that was asked? You can be faithful to the context and still answer a slightly different question.
The reason these four matter, and why I would resist collapsing them into a single “RAG quality” score, is that they fail in different ways and need different fixes. I kept seeing teams mistake retrieval problems for generation problems because the answer looked correct at first glance. Weeks would get spent on prompt engineering when the actual fix was in the retriever.
Example 1: low precision, high faithfulness. The retriever pulled 10 chunks, 2 are relevant, 8 are noise. The generator was disciplined and only used the 2 relevant ones, so faithfulness is 0.95. Context precision is 0.20. The fix is in retrieval, not generation, and a reranker would do more for it than any prompt change.
Example 2: high precision, low faithfulness. The retriever pulled 5 relevant chunks. Precision is 1.0. The generator wrote an answer with claims that are not in any of them. Faithfulness drops to 0.6. The fix is in generation. The model may be reaching for prior knowledge, or the prompt may not be strict enough about staying inside the context. Tightening the system prompt with “answer only from the provided context; if not present, say I don’t know” often picks this up.
Example 3: high faithfulness, low answer relevancy. The generator is honest about the context but is answering the wrong question. The user asked “what is the refund window for enterprise customers?” The answer faithfully describes the general refund policy without mentioning enterprise. The fix is in the prompt or the routing, not the retrieval.
Example 4: high faithfulness, high relevancy, low correctness. This is the dangerous one. The retriever pulled 5 chunks, all from the same outdated document. Precision is 1.0. The generator faithfully summarized them, so faithfulness is 0.98. The answer is on-topic, so relevancy is 0.92. But the document was superseded last quarter by a newer policy in the same corpus that the retriever missed. The only metric that catches this is context recall against a ground-truth set that lists the newer document as the relevant source. Without recall, you have no way to know whether the retrieved chunks were the right ones in the first place.
~~I am not fully convinced that answer relevancy as RAGAS computes it is the right signal for every workload. It correlates well with what users want on Q&A. On open-ended summarization or comparison queries, it sometimes punishes answers that are doing the right thing because the user’s question is too open to score against. Treat it as one data point.~~
You can run RAGAS in two modes. With a ground-truth answer set, you get context recall and end-to-end correctness alongside faithfulness and precision. Without one, you get the reference-free metrics (faithfulness, answer relevancy, context precision via LLM judge), which are useful for monitoring in production where no ground truth exists.
The thing I wish I had known earlier: calibrate the judge model against human labels on at least 50 samples before trusting any LLM-as-a-judge score. RAGAS uses an LLM judge for most metrics, and judges have biases. They tend to reward longer answers, formal phrasing, and certain citation styles. If you do not calibrate, you end up optimizing for the judge’s biases instead of for your users.
Example: judge bias on length. Two answers, same content. The short answer says “Refund window is 30 days for standard plans, 90 for enterprise.” The long answer wraps the same facts in three paragraphs of context. A naive judge often scores the long answer higher on “answer relevancy” because it appears more thorough, even though the short answer is what the user wanted. If you calibrate on a sample where humans labeled both as equally good, you can detect this bias and either pick a different judge or write a more specific judge prompt.
In code, a basic RAGAS run looks like this:
from ragas import evaluate
from ragas.metrics import (
Faithfulness,
ResponseRelevancy,
LLMContextPrecisionWithoutReference,
LLMContextRecall,
)
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI
evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))
results = evaluate(
dataset=eval_dataset,
metrics=[
Faithfulness(),
ResponseRelevancy(),
LLMContextPrecisionWithoutReference(),
LLMContextRecall(),
],
llm=evaluator_llm,
)
The output is a per-row score for each metric, which you can then average or slice. A typical row looks like:
{
"question": "What is the refund window for enterprise?",
"answer": "Enterprise plans have a 90-day refund window.",
"contexts": ["...", "..."],
"ground_truth": "Enterprise: 90-day refund window.",
"faithfulness": 1.0,
"answer_relevancy": 0.94,
"context_precision": 0.80,
"context_recall": 1.0
}
The interesting analysis is not the average score across the set. It is the slice. Average faithfulness over 500 cases tells you nothing actionable. Faithfulness sliced by intent (“billing” 0.94, “technical” 0.71, “legal” 0.62) tells you exactly where the retriever or the generator is breaking down.
RAGAS gets you component scores. It does not tell you what happened inside the agent’s loop. That is what observability is for.
3. Node-level observability with LangSmith and Langfuse
Once an agent starts chaining retrieval and tools together, evaluating only the final output stops being useful. You need to see what happened at each node in the trace: which tool was called, what arguments it got, how long it took, what came back, and whether the agent’s next decision made sense given that output.
I have used both LangSmith and Langfuse in production. They cover similar ground.
LangSmith is tighter with LangChain and LangGraph. If your agent is already built on those, it is the obvious choice. Langfuse is open source, self-hostable, and works with any framework or raw SDK calls.
If your stack is mixed, Langfuse usually needs less wiring. If you need to self-host the trace store without an enterprise license, or your compliance posture rules out sending production traces to managed SaaS at all, Langfuse is often the more practical option.
What both of them give you that print debugging cannot:
A trace tree. One user query expands into a tree of LLM calls, tool calls, retrievals, and critic checks. You can collapse and expand nodes, see latencies per node, and jump to inputs and outputs at any level. For an agent with even modest branching, this is the difference between five minutes of debugging and an hour of grep.
Per-node metrics. You can attach RAGAS faithfulness to the generation node, retrieval precision to the retrieval node, and a custom critic score to the critic node. The platform aggregates these across runs so you can see drift over time on each component independently.
Tagging and filtering. You can tag traces by user segment, query type, or experiment variant, then filter scores by tag. When the eng team asks “did the new reranker help on technical queries?”, you can answer it without exporting to a spreadsheet.
What a useful trace tree looks like
A real trace from an agentic RAG run, drawn as ASCII:
[user_query: "what is our refund policy for EU customers?"] t=0
├── [router: classify_intent] 12ms
│ └── intent = "policy_lookup", confidence = 0.91
├── [retrieval_round_1] 340ms
│ ├── embed_query 90ms
│ ├── vector_search (top 50) 180ms
│ └── rerank (top 5) 70ms
│ └── rerank_score_top1 = 0.42
├── [critic_round_1] 420ms
│ └── score = 0.38, decision = "retry"
├── [query_rewrite] 280ms
│ └── rewritten = "refund policy european union region"
├── [retrieval_round_2] 310ms
│ └── rerank_score_top1 = 0.78
├── [critic_round_2] 440ms
│ └── score = 0.71, decision = "accept"
└── [generation] 1.8s
└── tokens_out = 142, finish_reason = "stop"
─────
total: 3.6s
Two failure modes jump out from this tree. The first round was a 760-millisecond waste because the original query did not contain “EU” or “European Union”, so the retriever scored a generic refund-policy chunk highest. The query rewriter saved the trace; it would have been faster to route on intent first and add the EU filter at retrieval time.
Without the trace tree, you would see only a 3.6-second query with no obvious reason it was slow. From the production dashboard, traces like this look indistinguishable from a normal slow generation, which makes them surprisingly annoying to debug. I have spent more time staring at trace trees than I would like to admit.
Per-node metrics in practice
The metrics worth tracking at the node level:
Retrieval node. Number of chunks returned, average rerank score, fraction of chunks with metadata match, retrieval latency. The latency one matters more than people expect; a retrieval round that takes 2 seconds because of a poorly tuned BM25 index, an unindexed metadata filter, or a vector store under load is invisible in end-to-end metrics until users complain.
Tool call nodes. Success rate, error rate by tool, average duration, retry count. If your web search tool returns a 429 one in twenty calls, you want to see that on a dashboard, not as a single user complaint. The aggregate end-to-end success rate stays high because the agent retries, but each retry costs latency and tokens.
Generation node. Token count, time to first token, and finish reason. Long generations with finish_reason: length mean the model ran out of output space and the answer is truncated. End-to-end faithfulness will not catch this because the visible part of the answer is faithful; the user just got cut off.
Critic node. Critic score distribution, fraction of traces where the critic triggered a retry, average improvement after a retry-triggered round.
Instrumenting a span
In Langfuse, attaching a custom score to a node is a few lines:
from langfuse.decorators import observe, langfuse_context
@observe(name="retrieval_round")
def retrieve(query: str, round_idx: int):
chunks = vector_store.search(query, top_k=50)
reranked = reranker.rerank(query, chunks, top_k=5)
langfuse_context.update_current_observation(
metadata={
"round_idx": round_idx,
"chunks_returned": len(reranked),
"rerank_score_top1": reranked[0].score,
"rerank_score_top5_avg": sum(c.score for c in reranked) / 5,
}
)
return reranked
The LangSmith equivalent uses the @traceable decorator with metadata= on the run. The mental model is the same: every span carries the inputs, outputs, latency, and any extra metadata you attach.
Production sampling pattern
The trace tree is where you find failure modes that aggregated metrics hide. The worst case I have personally caught was a query-rewriter chain silently deleting tokens like “ACH” and “SEPA” from payment-related queries because an early normalization step was treating them as low-information noise words. Nobody noticed for days. The generator was good enough to compensate on most queries, end-to-end faithfulness looked acceptable, and the only signal that something was wrong was retrieval precision sliced by intent, which had quietly dropped on the payments slice. Took most of a Friday to find the line in the normalizer.
One pattern that works: pipe RAGAS scores into LangSmith or Langfuse as custom evaluators, then run them on a 1-5% sample of production traces rather than on a static eval set alone. Static evals tell you whether the system passes the cases you thought of. Production sampling tells you what users are actually asking and where it actually fails. A typical setup runs RAGAS asynchronously on sampled traces, writes the scores back onto the trace as metadata, and surfaces them in a dashboard sliced by intent or user segment.
4. Critic scores as a first-class signal
If your agent has a critic step, whether that is a Corrective RAG grader, a Self-RAG reflection token, or a custom LLM judge inside the loop, the critic’s output is one of the most useful signals you have. Most teams treat the critic as internal plumbing and never log its scores. That is a mistake.
The critic is already evaluating the retrieval or the generation on every single production query. If you store its scores, you get a free continuous evaluation signal at a much higher volume than any offline eval set can produce.
Self-RAG reflection tokens as observable scores
Self-RAG emits special tokens during generation that grade the model’s own work. The paper defines four:
[Retrieve]: should we retrieve more?[IsRel]: is the retrieved chunk relevant?[IsSup]: is the generated claim supported by the chunk?[IsUse]: is the final answer useful?
In the paper these are training signals. In production, you can capture them at inference time and log them as per-claim scores on the generation node. A trace where [IsSup] is “no” on three out of five claims is a much sharper warning than a 0.7 faithfulness score on the whole answer.
One caveat. I have found Self-RAG-style reflection tokens to be noisier in practice than the paper suggests. [IsRel] in particular tends to under-flag chunks that are tangentially related, which is exactly where you want it flagging. Calibrate against human labels before treating it as a hard control-flow signal; treat it as a loud diagnostic instead.
CRAG with a grader, worked through
In a Corrective RAG pipeline, a small classifier scores each retrieved chunk as relevant, ambiguous, or irrelevant. The agent uses these labels to decide whether to escalate to web search. If you only log the final decision, all you see is “escalated to web search” or “did not escalate”. If you log the per-chunk scores too, you can answer much more interesting questions.
Consider a week of traces where the agent escalated to web search. If the grader scores look like this:
trace_1: [irrelevant, irrelevant, irrelevant, irrelevant, irrelevant]
trace_2: [ambiguous, ambiguous, ambiguous, irrelevant, irrelevant]
trace_3: [ambiguous, irrelevant, irrelevant, irrelevant, irrelevant]
Trace 1 is a coverage problem: the corpus does not contain anything close to the answer, so the retriever fails cleanly and the escalation is correct. Trace 2 is a chunking problem: the corpus probably has the answer but the chunks are not specific enough for the grader to call them relevant. Different fix in each case. The aggregate “escalation rate” metric cannot tell you which one you have.
Tracking the critic over time
The most useful chart is the rolling weekly average of the critic’s score on retrieval, sliced by intent. A real shape I have seen:
week of 2026-03-15: avg = 0.78
week of 2026-03-22: avg = 0.79
week of 2026-03-29: avg = 0.62 <- something changed
week of 2026-04-05: avg = 0.77
The drop on 2026-03-29 traced back to a batch of around 800 PDFs that had been ingested with a chunker config someone had bumped in a feature branch and merged without flagging in the PR. Recall on those documents was bad. The critic kept rejecting the retrievals, the loop kept retrying, and dashboards kept showing acceptable end-to-end faithfulness because the system was eventually finding something workable. Reverted the chunker config, and the average critic score recovered over the next two days.
I have stopped trusting any single-week trend without checking what shipped that week.
Failure mode clustering
Critic scores let you sort production traces by quality and cluster the bottom quartile. Patterns show up. Maybe the low-score traces share a query intent that the retriever was not built for. Maybe they all hit a particular document type that was not chunked well. The critic is your bug-finder.
A concrete clustering workflow: pull the lowest 5% of traces by critic score from the last week, embed the user queries, run k-means with k=10, then inspect the centroid query from each cluster. Most clusters will be expected hard cases. One or two will be a surprise, and that surprise is usually a real bug.
Wiring critic scores into observability
Attach the critic score as a node-level metric on the same span where the critic runs:
@observe(name="retrieval_critic")
def critic(query, chunks):
per_chunk_scores = grader.score(query, chunks)
decision = "accept" if max(per_chunk_scores) > 0.5 else "retry"
langfuse_context.update_current_observation(
metadata={
"critic_score_max": max(per_chunk_scores),
"critic_score_avg": sum(per_chunk_scores) / len(per_chunk_scores),
"critic_decision": decision,
"critic_per_chunk": per_chunk_scores,
}
)
return decision
Once the critic is observable, it becomes one of the most reliable production-quality signals in the whole system.
5. Retrieval rounds as a behavior signal
Retrieval round count is a metric most teams do not track until something is wrong. It is one of the cleanest signals of agent behavior.
A well-behaved agentic RAG should converge in one or two retrieval rounds on most queries, with the occasional three or four rounds on hard ones. The distribution itself tells you a lot about whether the loop is working.
Four distribution shapes show up often:
Healthy. 70% of queries finish in 1 round, 20% in 2 rounds, 8% in 3 rounds, 2% take 4 or more. The long tail is real hard questions, and the median user gets a fast answer.
Loop-happy. 10% of queries finish in 1 round, 60% take 2 rounds, 25% take 3, and 5% hit the max. Median latency is bad. Usually this means the critic is too strict, triggering retries on retrievals that were actually fine. The fix is calibrating the critic threshold against human-labeled “good enough” retrievals.
Loop-shy. 95% of queries finish in 1 round, almost none retry, and end-to-end faithfulness is mediocre. The critic is too lenient and approving weak retrievals, and the generator turns into confident wrong answers. Tighten the threshold.
Bimodal. Most queries finish in 1 round, then a chunk of queries hit the max retry limit. The max-retry queries are usually a specific kind of question your retriever does not handle well at all. Cluster them and look at the inputs. There is often a single fix (a new metadata field, a different chunker for one document type, a query rewrite template) that collapses the bad tail entirely.
Worked example: a bimodal distribution
A team I was helping had a distribution that looked like:
1 round: 78%
2 rounds: 12%
3 rounds: 4%
4 rounds: 1%
5 rounds (max): 5% <- the suspicious bump
The 5-round bucket was almost entirely queries about a specific category of compliance policy. The corpus had the right document. The retriever could not find it because the document was a scanned PDF that had been ingested with text extraction and no OCR. The chunks were near-garbage. The agent kept retrying, hit the limit every time, and fell back to a generic answer. One re-ingestion pass with a vision parser collapsed the 5% bucket into the 1-round bucket.
That fix would not have been found by any end-to-end metric. The end-to-end score on those queries was a stable mediocre, because the fallback answer was always the same. The distribution histogram was the signal.
Conditional quality by round
The metric you want on the dashboard is the round-count histogram plus the conditional answer quality at each round. “What is the average faithfulness for answers produced after 1 round, after 2, after 3?”
A healthy pattern:
round 1: faithfulness 0.86
round 2: faithfulness 0.91 (the retry usually helps)
round 3: faithfulness 0.93
round 4+: faithfulness 0.88 (diminishing returns)
An unhealthy pattern:
round 1: faithfulness 0.85
round 2: faithfulness 0.84
round 3: faithfulness 0.83
round 4+: faithfulness 0.81 (it's getting worse)
In the unhealthy case the loop is mostly burning tokens. Reducing the max retry limit from 5 to 2 would not hurt quality and would cut tail latency significantly.
Stopping reason taxonomy
Track why the agent stopped, not just when:
critic_satisfied: the critic approved the retrieval. This should be the common case.max_retries_hit: the loop ran out of attempts. Should be rare in a healthy system.tool_failure: a tool returned an unrecoverable error.out_of_scope: the agent decided the query was not answerable from any available source.user_clarification_needed: the agent decided it needed more from the user.
A weekly chart of stopping reason proportions is one of the most useful production health views. A sudden spike in tool_failure means a downstream dependency is degrading. A creeping rise in max_retries_hit means quality is drifting and the loop is compensating.
6. Latency and cost as evaluation metrics
Latency and cost are evaluation metrics, even though most RAG eval frameworks ignore them. A 99% faithful answer that takes 12 seconds is worse than a 92% faithful answer that takes 2 seconds for most use cases. A pipeline that hits the right answer at 8 cents per query when your unit economics allow 1 cent is also a failed pipeline, just on a different axis.
Why end-to-end latency hides cascading delay
End-to-end latency on an agentic RAG run is the sum of all node latencies plus the orchestration overhead. The single number can hide a lot.
Consider two traces with the same 4-second end-to-end latency:
Trace A (acceptable):
retrieval_round_1: 0.4s
critic_round_1: 0.5s
generation: 3.0s <- generation is the bulk
Trace B (problematic):
retrieval_round_1: 1.2s
critic_round_1: 0.5s
retrieval_round_2: 1.2s
critic_round_2: 0.5s
generation: 0.6s <- 4x retrieval, generation is fast
Trace A is a normal slow generation. Probably acceptable. Trace B is the loop spinning.
The end-to-end metric cannot tell them apart. Per-node latency can.
Cost per query
Cost is a metric you can compute from the trace. Roughly:
retrieval_cost = (embedding_tokens × embed_rate) + (vector_search_units × search_rate)
rerank_cost = reranked_pairs × rerank_rate
critic_cost = critic_input_tokens × critic_input_rate
+ critic_output_tokens × critic_output_rate
generation_cost = generation_input_tokens × input_rate
+ generation_output_tokens × output_rate
per_query_cost = sum of all the above across all rounds
A worked example for a 2-round trace with reranker, critic, and a Sonnet-class generator might land at around 2.5 cents. At 100K queries per day, that is $2,500 per day, or about $75K per month. A loop-happy distribution that doubles the average round count doubles a large part of that cost. Cost per query is a metric, the cost distribution is a chart, and the 95th-percentile cost is the right thing to budget against.
Budgeting in the orchestration layer
Once you treat cost and latency as eval metrics, the orchestrator should respect them. Two simple budgets work well:
- Max rounds. A hard cap so the loop cannot run forever. The default should be 3 or 4 for most agents; anything higher needs a specific reason. I generally dislike aggressive retry loops, although one of the better-performing legal RAG systems I have looked at routinely went up to six rounds on multi-document compliance questions and the answers were noticeably better for it. So this is a default, not a rule.
- Max budget per query. Track tokens and time spent across the trace. If the agent has spent 90% of its budget by round 2, the orchestrator should accept whatever the critic has now rather than start round 3.
These budgets are not eval metrics on their own; they are constraints. But the chart of “what fraction of queries hit the budget ceiling” is a useful eval signal. A spike in budget-hits means quality is degrading somewhere upstream, and the budget is catching the symptom.
Worked example: a legitimately expensive query
A user asks “summarize all customer complaints from Q1 about the new pricing model and rank them by frequency.” This is a legitimately hard query. It needs broad retrieval, multiple rounds, and a long generation. It might cost 12 cents and take 9 seconds. That is fine, as long as it is rare. The eval question is whether your distribution has a small tail of legitimately expensive queries, or a fat middle of unnecessarily expensive queries.
Tag every trace with its cost and latency, slice by intent, and look for intents where the median query is expensive. Those are the ones to optimize.
7. Building an eval set that catches real regressions
None of the above replaces an eval set.
Component metrics tell you what is broken. The eval set tells you whether your fix made things worse somewhere else.
A model upgrade looks like a free win on the queries the team manually tested, and a week later support tickets surface a class of question the new model handles worse. The eval set is what catches that before it ships.
A useful eval set covers the query distribution you actually see in production, not just the queries the team finds interesting. Sample from production traces by intent cluster and pick examples that cover each cluster. Hand-written eval queries reflect what the team thought of, not what users ask, and the gap between those two sets is where bugs hide.
It also needs ground-truth answers, or at minimum, the relevant chunks for each query. Building this is slow and the part of the project that tends to get skipped first.
I used to think you could substitute LLM-judged ground-truth and avoid the labeling work. Then I watched a judge model rate an obviously wrong answer as correct because the answer matched a prompt template the judge had been trained on. Start with 50 hand-labeled cases. Grow the set. Every time a user reports a bad answer, add it to the eval set with the correct answer attached. Within six months you have a set that reflects real failure modes, not synthetic ones.
Eval set entry shape
A single eval entry that supports retriever and generator scoring:
{
"id": "evset_0042",
"query": "what is the refund window for enterprise customers?",
"intent": "policy_lookup",
"tags": ["billing", "enterprise"],
"relevant_chunks": ["doc_refund_policy_v3#chunk_18"],
"ground_truth_answer": "Enterprise plans have a 90-day refund window.",
"must_cite": ["doc_refund_policy_v3"],
"must_not_say": ["30-day", "no refund"],
"expected_rounds": 1,
"source": "user_report_2026-04-12"
}
The must_cite and must_not_say fields are cheap to author and catch a lot. A query about enterprise that does not cite the enterprise policy doc is a failure even if the answer text happens to be right. A query that says “no refund” when the policy is 90 days is a failure even if it cites the right doc.
Adversarial cases
The eval set needs adversarial cases too. Queries the system should refuse, queries that have no answer in the corpus, queries that contain personal data the system should not echo back, queries shaped like a prompt injection. These are usually rare in production traces but high-stakes when they happen. A bad refusal looks worse than a slow answer.
Concrete examples to include:
- A query asking for personal information about another user. Expected: refusal.
- A query asking about a topic genuinely not in the corpus. Expected: “I don’t have information on that” rather than a hallucinated answer.
- A query containing a classic prompt injection (“ignore previous instructions and…”). Expected: the agent ignores the injection and answers the actual question or refuses.
- A query with a typo or misspelling of a key term. Expected: the agent recovers, usually via query rewriting.
- A query in the wrong language. Expected: the agent handles it or refuses cleanly, depending on your scope.
What to score
On the eval set:
- End-to-end correctness against ground-truth answers (LLM judge or human-labeled).
- Retrieval recall against the labeled relevant chunks.
- Faithfulness on the generated answer.
- Refusal accuracy on the adversarial cases.
- Round count distribution against expectations.
- Latency p50 and p95.
- Cost p50 and p95.
Run this eval suite on every meaningful change: new embedding model, new chunker, new prompt, new orchestration logic, model upgrade.
Slicing is the whole point
The metric to watch most carefully is not the headline score but the distribution shift. If your overall correctness went up but your retrieval recall on technical queries dropped, you traded one population of users for another. The eval suite will show you this only if you slice by query type.
A sliced report might look like:
baseline new model delta
overall: 0.78 0.82 +0.04
billing: 0.91 0.93 +0.02
technical: 0.84 0.72 -0.12 <- regression
policy_lookup: 0.76 0.81 +0.05
customer_data: 0.62 0.79 +0.17
The headline number went up by 4 points while the technical slice dropped by 12. Slicing is the difference between finding the regression in CI and finding it through support tickets a week later.
Calibrating the judge over time
Calibrate the LLM judges against human labels every quarter. Models change. The judge that was well-calibrated last summer may have drifted by spring. The cheapest version of this is to label 50 eval-set cases by hand, run the judge on the same 50, and compute the correlation. If it has dropped meaningfully, either re-prompt the judge or switch to a different model.
8. What to alert on, what to chart
The signals from all the metrics above are not equally suited for alerting. Alerts wake people up at 3 AM; charts answer questions during business hours. Putting the wrong signal in the wrong channel is one of the easier ways to make a production team stop trusting the eval system.
What deserves an alert
Things that are sharp, fast-moving, and actionable in a minutes-to-hours window:
- Tool error rate exceeds a threshold for more than 5 minutes.
- p95 latency on any major intent slice exceeds budget for more than 15 minutes.
- Canary query set faithfulness drops below threshold. (Canary set described below.)
- Stopping reason
max_retries_hitexceeds 10% of traffic for more than 30 minutes. - A specific tool returns 4xx or 5xx in a sudden spike.
What belongs only on a chart
Things that are noisy, slow-moving, or only meaningful in context:
- Average critic score over a week. Useful trend, terrible alert.
- Round count distribution. Look at it daily, not on call.
- Per-intent faithfulness slices. Investigate weekly.
- Cost per query trends. Budget review, not pager.
Canary queries
A canary query set is the cheapest piece of production monitoring you can build. It is a small set of 20-50 questions that:
- Cover the most common intents.
- Have stable, known-correct answers in your corpus.
- Should always retrieve the same top chunks.
- Should always generate the same answer pattern.
Run the canary set on a cron every 15 minutes against the production system. Compute end-to-end correctness against the known answers. Alert when the score drops below a threshold for two consecutive runs.
The two-consecutive-runs rule matters more than the threshold. A single failure is usually a flaky LLM call, a transient tool timeout, or a vector store reindex. Two in a row almost always means something real shifted.
Example canary entries for a customer support agent:
- “How do I reset my password?” → must cite password reset article.
- “What is your refund policy?” → must mention 30-day window for standard plans.
- “How do I cancel my subscription?” → must cite cancellation flow.
- “Is there a free trial?” → must answer yes or no based on current policy.
When a canary fails, it is almost always one of three things: a deploy broke something, a corpus update changed retrieval, or a model upgrade regressed on a known case. All three are worth a page.
The canary set is the only metric I would alert on with any confidence.
The evaluation stack end to end
- Score the retriever and generator separately with RAGAS. Faithfulness, answer relevancy, context precision, context recall. End-to-end alone hides which half is breaking.
- Use LangSmith or Langfuse for node-level traces. Per-node latency, errors, and metrics make agent failures visible. End-to-end faithfulness will not catch a query rewriter that drops keywords or a tool that 429s one in twenty calls.
- Treat the critic score as a first-class signal. Log it, chart it, and use it to cluster failure modes. The critic is already evaluating every production query.
- Track retrieval round distribution and stopping reason. A loop-happy or loop-shy agent looks fine on end-to-end metrics but is doing the wrong thing on latency or quality. The histogram catches it.
- Measure latency and cost as eval metrics. A 99% faithful answer that takes 12 seconds is a failed answer for most use cases. Budget the loop in the orchestrator.
- Build the eval set from production traces and grow it from real failures. Cover the actual query distribution, include adversarial cases, slice every score, and re-run on every change.
- Alert on canaries, chart everything else. Noisy alerts kill the eval system. A 20-50 query canary set on a 15-minute cron is the right shape for a pager.
A well-evaluated agentic RAG needs a stack of metrics rather than a single score, because each metric catches a different failure mode the others miss.
I am sure parts of this article will look wrong to me in six months. The tooling is moving fast, the failure modes shift as the models change, and the best practices are not yet settled. The stack above is what I would build today.
\
This content originally appeared on HackerNoon and was authored by Tahir Nawaz
Tahir Nawaz | Sciencx (2026-06-01T14:26:26+00:00) What Production-Grade RAG Evaluation Should Look Like. Retrieved from https://www.scien.cx/2026/06/01/what-production-grade-rag-evaluation-should-look-like/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.