Evaluating Applications
Purpose and Scope
Evaluation in LlamaIndex is the practice of measuring whether an application is improving, regressing, or merely changing its behavior. The evaluation guide frames this as essential for context-augmented systems such as RAG applications and agents, because generated answers are not simple numeric predictions and often need to be judged against context, a query, a reference answer, or explicit guidelines. This page orients builders to the evaluation workflow documented by the framework module guide, then maps the reader-facing concepts to the concrete APIs, datasets, and example families referenced by the documentation.
Sources: docs/src/content/docs/framework/module_guides/evaluating/index.md
The evaluation area is organized as a collapsed framework module-guide section labelled Evaluating, which makes it a learning path rather than a single API page. The entry guide separates the problem into response evaluation and retrieval evaluation. Response evaluation asks whether a generated answer is grounded, relevant, correct, semantically similar to a reference, or compliant with a guideline. Retrieval evaluation asks whether the retrieved sources are relevant to the query before the answer is synthesized. Treating those as separate phases lets teams debug a RAG system more precisely.
Sources: docs/src/content/docs/framework/module_guides/evaluating/_meta.yml, docs/src/content/docs/framework/module_guides/evaluating/index.md
Relevant Source Files
- docs/src/content/docs/framework/module_guides/evaluating/_meta.yml — Defines the Evaluating navigation label and collapsed module-guide section behavior.
- docs/src/content/docs/framework/module_guides/evaluating/index.md — Introduces response evaluation, retrieval evaluation, question generation, integrations, usage-pattern links, and module links.
- docs/src/content/docs/framework/module_guides/evaluating/modules.md — Lists the example notebooks for response evaluation, retrieval evaluation, integrations, batch evaluation, multimodal RAG evaluation, and question generation.
- docs/src/content/docs/framework/module_guides/evaluating/usage_pattern_retrieval.md — Shows the retrieval evaluator API flow, metric selection, synthetic dataset generation, and batch dataset evaluation.
- docs/src/content/docs/framework/module_guides/evaluating/contributing_llamadatasets.md — Explains how to contribute a LabelledRagDataset and related llama-datasets.
- docs/src/content/docs/framework/module_guides/evaluating/evaluating_evaluators_with_llamadatasets.md — Documents LabelledEvaluatorDataset, evaluator benchmarking, pairwise evaluator datasets, and the EvaluatorBenchmarkerPack usage flow.
System-to-Code Mapping
The evaluation documentation maps naturally to three implementation concerns in an application. First, response evaluators score the answer-producing layer, including whether the answer follows the retrieved context and query. Second, retrieval evaluators score the retriever independently, using ground-truth document identifiers or generated question-context pairs. Third, dataset utilities and benchmark packs make repeated experiments possible. The docs name correctness, semantic similarity, faithfulness, context relevancy, answer relevancy, and guideline adherence as response-evaluation categories, and they name ranking-style metrics such as mean reciprocal rank, hit rate, and precision for retrieval evaluation.
Sources: docs/src/content/docs/framework/module_guides/evaluating/index.md, docs/src/content/docs/framework/module_guides/evaluating/usage_pattern_retrieval.md
For a RAG system, this separation is more than a documentation convenience. If retrieval scores are poor, changing the answer prompt may hide the issue but will not fix the missing context. If retrieval scores are strong but faithfulness or answer relevancy scores are weak, the query engine or answer synthesis behavior deserves attention. LlamaIndex also supports question generation from data, which helps teams create an evaluation set from their own corpus instead of relying only on handwritten queries. That makes the evaluation loop practical early in development.
Sources: docs/src/content/docs/framework/module_guides/evaluating/index.md
Response Evaluation Workflow
Response evaluation covers the generated result after the application has retrieved context and produced an answer. The guide emphasizes that many LlamaIndex evaluators are LLM-based and can use a stronger or designated judge model to make qualitative judgments. Some categories require labels, such as correctness against a reference answer and semantic similarity to a reference answer. Others can operate with combinations of query, response, retrieved context, and guidelines, which is useful when a team has a corpus but not a fully labelled benchmark.
Sources: docs/src/content/docs/framework/module_guides/evaluating/index.md
The modules index points readers to concrete notebooks for Faithfulness, Relevancy, Answer and Context Relevancy, Guideline Eval, Correctness Eval, Semantic Eval, Question Generation, Batch Eval, Multi-Modal RAG eval, and several integrations. A useful workflow is to start with a single evaluator that matches the immediate risk, then expand. For example, a legal or policy assistant might prioritize faithfulness and guideline adherence, while a support-search system may first prioritize answer relevancy and context relevancy before introducing labelled correctness checks.
Sources: docs/src/content/docs/framework/module_guides/evaluating/modules.md, docs/src/content/docs/framework/module_guides/evaluating/index.md
Retrieval Evaluation Workflow
Retrieval evaluation is documented as an independent path using a retriever, a query, and expected source identifiers. The retrieval usage pattern shows RetrieverEvaluator.from_metric_names configured with metric names such as mrr and hit_rate, then invoked with a query and expected node identifiers. This is the smallest loop for validating whether the retriever can find the right evidence before any answer-generation step is involved. It is especially useful when changing chunking, embeddings, vector-store configuration, metadata filters, or top-k settings.
Sources: docs/src/content/docs/framework/module_guides/evaluating/usage_pattern_retrieval.md
from llama_index.core.evaluation import RetrieverEvaluator
retriever = ...
retriever_evaluator = RetrieverEvaluator.from_metric_names(
["mrr", "hit_rate"], retriever=retriever
)
retriever_evaluator.evaluate(
query="query", expected_ids=["node_id1", "node_id2"]
)The same usage page also describes synthetic retrieval dataset generation with generate_question_context_pairs. Given nodes, an LLM, and a requested number of questions per chunk, the helper returns an EmbeddingQAFinetuneDataset containing queries, relevant_docs, and corpus. That object can then be passed to aevaluate_dataset for batch evaluation. The documentation explicitly notes that the dataset method should run faster than calling evaluate separately on each query, so it is the better fit for recurring experiments or continuous comparison between retriever versions.
Sources: docs/src/content/docs/framework/module_guides/evaluating/usage_pattern_retrieval.md
from llama_index.core.evaluation import generate_question_context_pairs
qa_dataset = generate_question_context_pairs(
nodes, llm=llm, num_questions_per_chunk=2
)
eval_results = await retriever_evaluator.aevaluate_dataset(qa_dataset)Datasets, Benchmarks, and Contributions
LlamaDatasets provide reusable evaluation assets for benchmarking systems and tasks. The contribution guide explains that a robust RAG evaluation suite benefits from diversified datasets and describes contributing a LabelledRagDataset. The documented contribution process has two high-level steps: create the labelled dataset, save it as JSON alongside source text files for the llama-datasets repository, and submit required dataset metadata to the llama-hub repository. The guide also points to a submission template notebook intended to make new dataset creation or conversion smoother.
Sources: docs/src/content/docs/framework/module_guides/evaluating/contributing_llamadatasets.md
Evaluator benchmarking is a related but distinct workflow: instead of evaluating an application answer, it evaluates the evaluator itself. The LabelledEvaluatorDataset documentation describes examples with attributes such as query, answer, ground truth answer, reference score, and reference feedback. The flow is to run predictions over the dataset with a provided LLM evaluator, then compute metrics by comparing those predictions to the references. The page also names LabelledPairwiseEvaluatorDataset for judging pairwise comparison evaluators that choose the better of two responses.
Sources: docs/src/content/docs/framework/module_guides/evaluating/evaluating_evaluators_with_llamadatasets.md
from llama_index.core.llama_dataset import download_llama_dataset
from llama_index.core.llama_pack import download_llama_pack
from llama_index.core.evaluation import CorrectnessEvaluator
from llama_index.llms.gemini import Gemini
evaluator_dataset, _ = download_llama_dataset(
"MiniMtBenchSingleGradingDataset", "./mini_mt_bench_data"
)
gemini_pro_llm = Gemini(model="models/gemini-pro", temperature=0)
evaluator = CorrectnessEvaluator(llm=gemini_pro_llm)
EvaluatorBenchmarkerPack = download_llama_pack(
"EvaluatorBenchmarkerPack", "./pack"
)Integrations and Example Modules
The evaluation guide does not treat LlamaIndex evaluators as the only possible measurement stack. It explicitly lists community evaluation integrations including UpTrain, Tonic Validate, DeepEval, Ragas, RAGChecker, and Cleanlab. The module index reinforces that by linking example notebooks for Deepeval Integration, Uptrain Integration, RAGChecker Integration, and Cleanlab. Use these integrations when your team already depends on an external evaluation platform, wants visualization, or needs to compare LlamaIndex results with metrics and reports produced outside the core framework.
Sources: docs/src/content/docs/framework/module_guides/evaluating/index.md, docs/src/content/docs/framework/module_guides/evaluating/modules.md
As a practical path, begin with the examples that match your current system boundary. For answer quality, use the response-evaluation notebooks and choose faithfulness, relevancy, correctness, semantic similarity, or guideline checks based on the failure mode you most need to catch. For retriever quality, follow the retrieval usage pattern before tuning response prompts. For repeatable benchmarks, introduce LlamaDatasets and benchmark packs. Once the core loop is stable, compare external integrations if your project needs dashboards, cross-framework reports, or organizational evaluation standards.
Sources: docs/src/content/docs/framework/module_guides/evaluating/modules.md, docs/src/content/docs/framework/module_guides/evaluating/usage_pattern_retrieval.md, docs/src/content/docs/framework/module_guides/evaluating/evaluating_evaluators_with_llamadatasets.md
Next Steps
Read the evaluation metrics page next when you need evaluator-specific behavior and metric categories in more detail. Read the datasets page when you want repeatable benchmarks, shared test sets, or contribution workflows. If your results show retrieval failures, move to the retrievers, vector-store indexing, and ingestion-pipeline pages to inspect how nodes, embeddings, and stores are created. If retrieval is strong but answers are weak, continue with response synthesis, query engines, prompts, callbacks, and observability so the answer-generation path can be measured and improved systematically.