Evaluation Metrics and Evaluators
Purpose and Scope
Evaluation in LlamaIndex is the practice of measuring whether a retrieval-augmented generation system, agent, or other LLM application is doing the right thing. The official evaluation guide separates the problem into response evaluation, which judges generated answers against queries, contexts, labels, or guidelines, and retrieval evaluation, which judges whether the system found useful sources. This page focuses on the metric and evaluator categories exposed by the API reference entries for answer relevancy, context relevancy, correctness, faithfulness, dataset generation, and retrieval metrics. Sources: docs/api_reference/api_reference/evaluation/metrics.md, docs/api_reference/api_reference/evaluation/answer_relevancy.md, docs/api_reference/api_reference/evaluation/context_relevancy.md, docs/api_reference/api_reference/evaluation/correctness.md, docs/api_reference/api_reference/evaluation/dataset_generation.md, docs/api_reference/api_reference/evaluation/faithfullness.md
Use these evaluators when a demo has become a system you need to tune. A RAG pipeline can fail because the retriever missed the right nodes, because synthesis ignored the retrieved evidence, because the answer is off-topic, or because the answer differs from a labeled reference. LlamaIndex intentionally exposes separate evaluator classes and retrieval metric helpers so teams can isolate those failure modes instead of reducing quality to a single opaque score. The API reference pages are concise, but their member lists identify the main entry points you should wire into test runs, notebooks, or batch evaluation jobs.
The evaluation terminology is easiest to apply if you define the unit under test before choosing a metric. A retriever returns ranked source nodes, so retrieval metrics such as hit rate and mean reciprocal rank inspect the ranked list. A query engine returns an answer and often the retrieved context, so response evaluators inspect the query, answer, context, and sometimes a reference answer. Dataset generation closes the loop by producing reusable query and response data, which lets teams repeat the same checks after changing chunking, embeddings, prompts, rerankers, or model providers.
Relevant Source Files
- docs/api_reference/api_reference/evaluation/metrics.md — API reference entry for retrieval metric primitives:
MRR,HitRate,RetrievalMetricResult, andresolve_metrics. - docs/api_reference/api_reference/evaluation/answer_relevancy.md — API reference entry for
AnswerRelevancyEvaluator, the response evaluator category that checks whether an answer addresses the query. - docs/api_reference/api_reference/evaluation/context_relevancy.md — API reference entry for
ContextRelevancyEvaluator, the evaluator category that checks whether retrieved context is relevant to the query. - docs/api_reference/api_reference/evaluation/correctness.md — API reference entry for
CorrectnessEvaluator, the label-aware evaluator category for comparing generated answers with reference answers. - docs/api_reference/api_reference/evaluation/dataset_generation.md — API reference entry for
DatasetGeneratorandQueryResponseDataset, which support reusable evaluation sets. - docs/api_reference/api_reference/evaluation/faithfullness.md — API reference entry for
FaithfulnessEvaluator, the response evaluator category that checks whether answers are grounded in retrieved contexts.
Evaluation Categories
Answer relevancy evaluates whether the produced answer is actually responsive to the user query. This is different from correctness: an answer may be factually true but fail to answer the specific question, or it may answer only a neighboring question that the retrieved context made convenient. The API reference exposes this category through AnswerRelevancyEvaluator, making it a named public surface rather than an informal prompt you write from scratch. In practice, run it after query-engine or chat-engine calls when you want to know whether final responses satisfy user intent. Sources: docs/api_reference/api_reference/evaluation/answer_relevancy.md
Context relevancy evaluates the retrieval side of a RAG system from the perspective of the query. If retrieved chunks are unrelated, even a strong synthesis model can produce vague, hallucinated, or low-confidence answers. The API reference exposes ContextRelevancyEvaluator for this purpose, complementing rank-based retrieval metrics with an LLM-judged assessment of whether the retrieved text itself is useful. Use it when tuning loaders, node parsers, metadata filters, top-k settings, hybrid search, or rerankers, because those changes primarily affect what context enters the response synthesis phase. Sources: docs/api_reference/api_reference/evaluation/context_relevancy.md
Faithfulness evaluates whether the answer stays supported by the retrieved contexts. In RAG applications this is the core hallucination check: the answer might be relevant to the query and fluent, but still introduce claims that are absent from the retrieved evidence. The API reference exposes FaithfulnessEvaluator, and the repository path preserves the historical spelling faithfullness.md. Treat faithfulness as a post-synthesis grounding test. It is especially useful when comparing prompt templates, response synthesis modes, or more capable LLMs that may fill gaps confidently instead of admitting uncertainty. Sources: docs/api_reference/api_reference/evaluation/faithfullness.md
Correctness evaluates a generated answer against a reference answer and therefore belongs to the label-aware side of evaluation. The official guide distinguishes correctness from evaluators that do not require ground-truth labels: correctness needs an expected answer or grading target, while answer relevancy and faithfulness can often operate from the query, response, and context. The API reference exposes CorrectnessEvaluator as the public entry point. Use it for benchmark-style tests, regression suites, or curated golden datasets where the application has known expected behavior. Sources: docs/api_reference/api_reference/evaluation/correctness.md
Semantic similarity and pairwise comparison fill two neighboring needs in the broader evaluation toolbox. Semantic similarity asks whether a prediction and reference convey the same meaning even if they use different wording, so it is useful when exact-match grading would be too brittle. Pairwise comparison asks which of two outputs is better under a criterion, so it is useful for A/B testing prompts, models, retrievers, or synthesis strategies. Even when their dedicated entries are not among this page's targeted source files, they fit the same decision process: choose them when relative or meaning-level judgment is more useful than a single pass/fail score.
Retrieval Metrics Reference
The retrieval metric API reference lists HitRate, MRR, RetrievalMetricResult, and resolve_metrics under llama_index.core.evaluation. HitRate is the direct question of whether at least one expected or relevant item appeared in the retrieved set. MRR, mean reciprocal rank, rewards systems that place the first relevant item earlier in the ranking. RetrievalMetricResult is the result container named by the API reference, and resolve_metrics is the helper named for turning metric declarations into concrete metric objects or selections. Sources: docs/api_reference/api_reference/evaluation/metrics.md
| Public member | Category | Use when |
|---|---|---|
HitRate | Retrieval metric | You need to know whether retrieval found any relevant source in the candidate set. |
MRR | Retrieval metric | You care about the rank position of the first relevant result, not just presence. |
RetrievalMetricResult | Retrieval result type | You need a structured result from retrieval metric evaluation. |
resolve_metrics | Metric helper | You want to normalize or resolve configured metric names into metric implementations. |
Retrieval metrics are most useful before judging generated answers, because they tell you whether the answerer had a fair chance. If hit rate is poor, work on data loading, parsing, metadata, embeddings, vector store configuration, or reranking before spending time on answer prompts. If hit rate is acceptable but MRR is poor, the relevant evidence exists but is buried too deep, which can still hurt synthesis when context windows are limited. These metrics are also good for continuous integration because they produce compact, comparable values across repeated runs against the same evaluation dataset.
Dataset Generation and Repeatable Evaluation
The dataset generation API reference exposes DatasetGenerator and QueryResponseDataset. These names indicate two complementary responsibilities: producing evaluation data and storing query-response examples in a reusable dataset form. The official guide explains that LlamaIndex can generate questions from your data, giving teams a way to bootstrap evaluation when they do not yet have hand-labeled test sets. Generated data should still be inspected for quality, but it provides a practical starting point for measuring whether changes to indexing or prompting improve application behavior. Sources: docs/api_reference/api_reference/evaluation/dataset_generation.md
A repeatable evaluation flow usually starts by selecting representative documents, generating or curating questions, recording references where needed, and then running the same evaluators after each system change. Retrieval metrics can run against expected relevant documents or nodes. Correctness and semantic similarity can run when reference answers exist. Faithfulness, answer relevancy, and context relevancy can run even when the main evidence is the query, produced answer, and retrieved context. QueryResponseDataset is the API reference member that signals this reusable dataset role in the evaluation package.
System-to-Code Mapping
| Reader problem | LlamaIndex concept | API reference member | Source file |
|---|---|---|---|
| Did retrieval find the right evidence? | Retrieval evaluation | HitRate, MRR, RetrievalMetricResult, resolve_metrics | docs/api_reference/api_reference/evaluation/metrics.md |
| Did the answer address the question? | Answer relevancy | AnswerRelevancyEvaluator | docs/api_reference/api_reference/evaluation/answer_relevancy.md |
| Was the retrieved context relevant? | Context relevancy | ContextRelevancyEvaluator | docs/api_reference/api_reference/evaluation/context_relevancy.md |
| Did the answer match a reference? | Correctness | CorrectnessEvaluator | docs/api_reference/api_reference/evaluation/correctness.md |
| Was the answer grounded in context? | Faithfulness | FaithfulnessEvaluator | docs/api_reference/api_reference/evaluation/faithfullness.md |
| How do we create reusable eval data? | Dataset generation | DatasetGenerator, QueryResponseDataset | docs/api_reference/api_reference/evaluation/dataset_generation.md |
This mapping is a practical triage guide. If users complain that answers cite irrelevant material, start with context relevancy and retrieval metrics. If answers sound plausible but contain unsupported details, run faithfulness. If the answer is grounded but not useful, run answer relevancy. If the application has a canonical expected answer, run correctness and consider semantic similarity for less brittle grading. If you are comparing two candidate systems rather than scoring one in isolation, add pairwise comparison to decide which output better satisfies your rubric.
Implementation and Workflow Guidance
A useful evaluation workflow separates data quality, retrieval quality, and response quality into stages. First, build or load a QueryResponseDataset so that repeated experiments operate on the same inputs. Second, run retriever-only checks with HitRate and MRR to see whether the index and retrieval configuration surface relevant evidence. Third, run response evaluators such as ContextRelevancyEvaluator, FaithfulnessEvaluator, AnswerRelevancyEvaluator, and CorrectnessEvaluator against full query-engine or chat-engine outputs. This staged approach prevents a bad score from being misattributed to the wrong subsystem.
When interpreting results, avoid treating any single evaluator as a complete product-quality signal. Faithfulness can be high for an answer that is too narrow. Answer relevancy can be high for an answer that is unsupported. Correctness can be limited by the quality and coverage of reference answers. Hit rate can be high while relevant evidence appears too low in the ranking to be used. Combining metrics gives a more diagnostic picture and makes evaluation results actionable for engineers tuning chunk sizes, prompts, model choices, retrievers, and synthesis strategies.
Next Steps
Start by choosing one response evaluator and one retrieval metric that correspond to your current failure mode. For early RAG experiments, pair HitRate or MRR with FaithfulnessEvaluator so you can see both evidence discovery and grounding. For production regression tests, add CorrectnessEvaluator where reference answers exist and keep evaluation inputs in a reusable dataset. Then read the broader evaluation overview for batch running patterns, the datasets page for durable test data, and the observability pages if you need traces explaining why an evaluator produced a surprising score.