How to Measure the Quality of a RAG System

The quality of a RAG system cannot be represented honestly by a single “percentage of correct answers.” At minimum, three stages must be measured separately: whether the system found the required information, whether its cited sources actually support its claims, and whether the final answer is correct, complete, and appropriately cautious. In production, access control, latency, cost, and successful task completion also matter.

This separation is not merely convenient for analysts. If the correct document was never retrieved, changing the model or system prompt will not solve the problem. If the required passage was present in the context but the answer distorted it, the generation stage needs attention. A fluent answer with the wrong citation is yet another failure, even if its conclusion happens to be correct.

What exactly is being evaluated in RAG

RAG, or retrieval-augmented generation, connects a language model to an external knowledge source. Before producing an answer, the system retrieves documents or passages related to the query, places selected context in the model request, and generates a response. This architecture makes it possible to use updateable external memory and retain provenance—two of the motivations behind the original RAG approach.

In practice, the quality chain contains more than two technical modules:

1. document ingestion, chunking, and metadata retention;

2. query processing, search, and reranking;

3. context assembly inside the model window;

4. answer generation;

5. claim-to-citation mapping;

6. access-rule, safety, and user-outcome checks.

The RAGAS paper likewise separates a retriever’s ability to find focused context, a model’s ability to use that context faithfully, and the quality of the generated answer itself. This prevents an attractive average score from hiding the actual source of failure.

Start with a representative golden dataset

Without reference cases, RAG quality is judged largely by intuition. A useful starting point is an expert-reviewed dataset in which every case contains at least:

• a user question phrased as it would occur in real work;

• an acceptable answer or the key facts that must appear;

• the required documents and, ideally, exact passages;

• sources that must not be used, such as an obsolete policy version;

• a flag showing whether the knowledge base can answer the question at all;

• the user role and access level;

• the risk category and likely consequence of an incorrect answer.

The dataset should contain more than common happy-path questions. Add rare terminology, abbreviations, typos, paraphrases, multiple intents, multi-source questions, conflicting documents, and cases with no valid answer. For a multilingual system, test the same intent in Latvian, English, and every other language actually used.

Security cases belong in their own slice: requests for restricted documents, attempts to obtain personal data, malicious instructions embedded in documents, and situations that must be escalated to a person. OpenAI’s evaluation guidance recommends combining typical, edge, and adversarial cases, using expert labels, and expanding the dataset with failures discovered in production.

A golden dataset is not a one-off exam. It is the product’s quality specification. When the system begins serving a new source, user group, or workflow, the dataset must grow to reflect the new risk.

Measuring document retrieval

The core retrieval question is simple: did the model receive the evidence required for a correct answer? Measure this at both document and passage level. The correct file paired with the wrong section may still be useless context.

Precision@k: how clean are the top results?

Precision@k is the proportion of the first k retrieved passages that are relevant to the question. If only two of the first five passages help answer it, Precision@5 is 0.4.

Low precision means the model receives noise. Noise increases cost, consumes the context window, and creates opportunities for a similar but incorrect passage to dominate the answer. The Ragas description of context precision also accounts for ordering: useful passages should rank above irrelevant ones.

Recall@k: did retrieval miss something essential?

Recall@k measures how many of all reference passages required for the answer appear among the first k results. If the answer requires a main policy and an exceptions table but the system finds only the policy, precision may look respectable while recall remains inadequate. Ragas context recall similarly checks how many claims in a reference answer can be supported by retrieved context.

MRR and nDCG: is the best evidence ranked high enough?

MRR, or mean reciprocal rank, is useful when a query normally has one principal source. It rewards a system when the first relevant result appears at the top and sharply discounts it when that result appears lower down.

nDCG works better when passages have different degrees of relevance. A current procedure may be fully relevant, an explanatory slide deck only partly relevant, and an obsolete procedure actively misleading. nDCG incorporates both graded relevance and the position of each result.

No single retrieval metric predicts whether the system will answer successfully. Slice the results by query type, source system, language, and user role. A healthy average Recall@k can hide the fact that finance policies are consistently retrieved while Latvian HR questions frequently fail.

Source and citation accuracy

A retrieved source is not necessarily a correctly used source. Citation quality needs at least four distinct checks.

Claim support. Does the cited passage support the adjacent claim? Keyword overlap is not enough. The passage must contain the relevant condition, value, exception, or conclusion.

Citation completeness. What proportion of verifiable claims in the answer has source support? One citation at the end of a paragraph should not imply that it supports five facts when it covers only one.

Authority and freshness. Is the source the approved current procedure rather than an old attachment, presentation, or duplicate? Preserve the document owner, version, effective date, and status. If sources conflict, the system needs an explicit precedence rule or must acknowledge uncertainty.

Access correctness. A factually correct answer drawn from a document the user was not allowed to access is a serious failure. Security tests must therefore inspect not only the displayed citation but every retrieval result and intermediate context. Document permissions must survive the entire pipeline.

A practical citation check operates at claim level. Split the answer into verifiable claims, locate the cited passage for each, and label it “fully supported,” “partially supported,” “unsupported,” or “source unavailable.” In high-risk domains, a domain expert should review a sample rather than leaving every decision to another model.

Answer reliability is not the same as groundedness

An answer can be entirely grounded in context yet still be wrong because the context is obsolete. It can also be factually correct but ungrounded because the model relied on prior knowledge instead of approved company sources. The final answer therefore needs a multi-criterion scorecard:

• correctness — do the conclusions match the reference answer and domain rules?

• groundedness — can every factual claim be inferred from the allowed context?

• completeness — is a decision-critical condition or exception missing?

• relevance — does the answer address the user’s question directly without distracting material?

• instruction adherence — does it follow the required format, tone, action boundaries, and escalation rules?

• calibrated abstention — does it say that evidence is insufficient when necessary, without refusing when the evidence is adequate?

Microsoft’s RAG evaluator documentation makes a similar distinction between process evaluation—retrieval and document retrieval—and system evaluation, which covers groundedness, relevance, and response completeness. The model is useful even when the Microsoft tool itself is not part of the stack.

Answerable and unanswerable questions should be reported separately. For answerable cases, measure correctness and completeness. For unanswerable cases, measure how often the system safely abstains. A confidently invented answer about a nonexistent policy is usually more dangerous than a correct refusal.

An automated judge is not ground truth

Some checks can be deterministic: document-ID matches, version status, permission filters, latency, cost, and exact field values. More nuanced language qualities can be assessed by an LLM judge, but only with a specific rubric and examples of each score level.

LLM judges have biases of their own. They may prefer longer responses, a particular phrasing, or one answer position. Build a human-labelled control sample first and measure agreement between the automated judge and domain experts. OpenAI’s guidance recommends pairwise comparisons or clear pass/fail decisions, detailed rubrics, and validation of model graders against human labels.

A sensible division of labour is to automate high-volume regression checks, send boundary cases and high-impact failures to people, and turn disagreements between the two into new evaluation cases.

From offline evaluation to production monitoring

Before launch, run an offline evaluation on a fixed dataset and record a baseline for every stage. Repeat it after changing chunking, the embedding model, search filters, reranking, prompts, the language model, or permission logic. Compare not only the overall average but also the weakest slices and critical cases.

In production, a reference answer is unavailable for every request, so use several signals together:

• the rate of no-answer responses and abstentions;

• repeated, rephrased questions in the same session;

• user corrections and human escalations;

• citation opening and citation-problem reports;

• completed tasks, not merely delivered answers;

• retrieval, model, and end-to-end latency;

• cost per successfully completed task;

• unauthorized retrieval attempts and security incidents.

A thumbs-up is not a truth label. Users may like a fast, confident, incorrect answer. They may dislike an accurate warning because it blocks the result they wanted. Product signals must therefore be linked back to expert-audited samples.

The NIST Generative AI Profile treats evaluation as part of trustworthiness management across the AI lifecycle. For RAG, continuous monitoring is essential because not only models but also documents, versions, permissions, and user questions change.

A failure matrix for targeted fixes

This matrix prevents expensive but ineffective changes. Replacing the language model will not restore a missing document, and a new vector database will not help when the model ignores an exception already present in context.

A practical example

Imagine an internal assistant receives this question: “May a sales manager approve their own €850 travel expense?” A correct answer may require the current expense policy, an approval matrix, and the rule prohibiting self-approval.

The retrieval test checks whether all required passages appear among the top results and whether an obsolete policy is ranked lower. The citation test checks that every conclusion maps to the correct clause. The answer test verifies the conclusion, names the required approver, and ensures that no exception is invented. The security test repeats the query as a user who cannot access the approval matrix; the system must respect that permission instead of exposing a “correct” confidential answer.

One question therefore produces several independent tests. This is what reveals whether an improvement in one layer has damaged another.

Setting a release threshold

There is no universal RAG quality threshold. The cost of error differs between an internal document finder and an assistant that recommends action in a finance or HR process. Set thresholds by risk category rather than using one average for the entire system.

Before release, establish at least:

• separate minimum thresholds for retrieval recall, citation support, and answer correctness;

• zero tolerance for known access-control violations;

• mandatory correct abstention on critical unanswerable cases;

• no regressions in high-risk evaluation slices;

• acceptable latency and cost under expected load;

• named owners for incidents, source freshness, and dataset maintenance.

Thousands of cases are not required on day one. A few dozen carefully selected questions for each important workflow are more informative than a large synthetic set with no expert review. Expand the suite with real incidents, user paraphrases, and new document versions.

Conclusion

A reliable RAG system is not simply one that writes confidently. It finds the necessary evidence, excludes unauthorized or obsolete sources, connects claims to the right passages, and abstains when an answer is not supported.

That is why the quality programme must answer three questions separately: what the system retrieved, what the sources actually prove, and what the answer claims. With a representative golden dataset, regression testing, and production monitoring, RAG moves from an impressive demo to a controlled business process. This matters especially when an AI assistant is part of business process automation and its output affects a real decision.