RAG evaluation asks two separate questions: did the system retrieve the right evidence, and did its answer faithfully use that evidence? A fluent response with several links can still fail both tests.
This guide builds on RAG with Local Embeddings. It focuses on observable checks rather than assuming that semantic similarity or an LLM judge establishes truth.
Build a small reference set
Start with questions people actually ask of the collection. For each, record the expected answer or rubric, the supporting document revision and passage, and whether the system should answer at all.
Include exact identifiers, paraphrases, questions requiring multiple passages, ambiguous requests, outdated documents, unanswerable questions, and documents the test user must not access. Keep a held-out set so tuning does not simply memorize the examples.
Evaluate retrieval before generation
| Metric | Question |
|---|---|
| Recall at k | How much of the labeled relevant evidence appears in the first k results? |
| Precision at k | How many of those k results are relevant? |
| First relevant rank | How far must the system look before finding useful evidence? |
| Permission leakage | Did any unauthorized material reach retrieval output? |
| Freshness | Does the result refer to the current document revision? |
Document your relevance labels and scoring unit: whole documents and individual chunks are not interchangeable. For multi-part questions, finding one relevant chunk may not mean the evidence is sufficient.
Compare keyword, vector, and hybrid search on the same questions. Inspect failures before changing chunk size or introducing a reranker. A larger k can raise recall while filling the prompt with distracting text.
Evaluate answer claims and citations
Break an answer into checkable claims. For each claim, ask whether the cited passage supports it, whether the claim is accurate, and whether an important qualification was omitted.
A citation that resolves is merely a valid link. It is not necessarily evidence for the adjacent sentence. Record citation coverage separately from citation correctness: an answer may cite every sentence with the wrong source.
Require document or segment identifiers that map to authorized source records. Check that the citation opens the intended revision or clearly indicates that the source has since changed. Do not trust model-invented URLs as authoritative references.
Test refusal and adversarial content
When evidence is absent, the system should say what is missing instead of filling the gap from model memory. Score unanswerable questions explicitly so the system is not rewarded only for producing answers.
Add a test document containing instructions to ignore the user or reveal unrelated files. The answerer should treat it as source content, not as authority. Also test revoked sharing and deleted documents across indexes and caches.
Change one variable and retain evidence
Record the corpus snapshot, extraction version, embedding model, retrieval settings, prompt, model, and evaluated user scope. Compare quality, latency, and cost before and after a change.
LLM judges can help triage results, but validate their scoring against human examples. They can favor verbosity, overlook unsupported claims, or be influenced by malicious text. Keep a small manually reviewed release gate.