Datasets

Purpose and Scope

Datasets make evaluation repeatable. In a LlamaIndex application, an experiment usually begins with source data, reader integrations, chunking, indexing, retrieval, and response synthesis. Without a stable dataset, every change to prompts, embedding models, retrievers, or index configuration can be hard to compare because the questions, references, and loaded source material may shift between runs. This page explains the dataset-facing parts of the public documentation surface: the DatasetsReader reader entry point, related reader integrations that can populate evaluation corpora, and the evaluation-oriented dataset APIs described by the official docs, including DatasetGenerator, QueryResponseDataset, and the broader Llama Dataset package family.

The repository evidence for this page comes from generated API reference pages. Those pages expose public members using documentation directives rather than long examples, so this page focuses on how the named API surfaces fit into a repeatable evaluation workflow. The most directly relevant reader is DatasetsReader, documented under the readers API reference. Nearby reader pages show the same documentation pattern for concrete connectors such as Azure Blob Storage, GitHub repositories, Agent Search, and Airbyte-based sources. Together, these references show that datasets sit at the boundary between data loading and evaluation: readers provide reproducible inputs, while evaluation dataset objects provide reproducible questions and expected responses. Sources: docs/api_reference/api_reference/readers/datasets.md, docs/api_reference/api_reference/readers/github.md

Relevant Source Files

• docs/api_reference/api_reference/readers/datasets.md — API reference page for llama_index.readers.datasets and the DatasetsReader member.
• docs/api_reference/api_reference/readers/azstorage_blob.md — API reference page for the Azure Blob Storage reader, a representative external storage connector for building datasets from object storage.
• docs/api_reference/api_reference/readers/github.md — API reference page for GitHub readers, including repository contents, issues, and collaborators as possible source material.
• docs/api_reference/api_reference/readers/agent_search.md — API reference page for AgentSearchReader, showing another reader-backed way to bring external data into LlamaIndex.
• docs/api_reference/api_reference/readers/airbyte_cdk.md — API reference page for the generic Airbyte CDK reader integration.
• docs/api_reference/api_reference/readers/airbyte_gong.md — API reference page for the Airbyte Gong reader integration, illustrating source-specific Airbyte connectors.

Core Primitives

A dataset workflow has three distinct primitives. A reader loads raw or structured source data into the LlamaIndex data model. DatasetsReader is the dataset-specific reader named in the public readers reference, while connector readers such as GithubRepositoryReader, GitHubRepositoryIssuesReader, AzStorageBlobReader, AgentSearchReader, AirbyteCDKReader, and AirbyteGongReader represent source-specific ways to gather the material that may become an evaluation corpus. A dataset generator then creates questions or query-response examples from source documents. A dataset object stores those examples so that evaluators can run the same cases against multiple versions of a RAG pipeline. Sources: docs/api_reference/api_reference/readers/datasets.md, docs/api_reference/api_reference/readers/azstorage_blob.md, docs/api_reference/api_reference/readers/agent_search.md, docs/api_reference/api_reference/readers/airbyte_cdk.md, docs/api_reference/api_reference/readers/airbyte_gong.md

The official API reference evidence names DatasetGenerator and QueryResponseDataset under llama_index.core.evaluation, and it also exposes a llama_index.core.llama_dataset reference family. In practical terms, treat generation and storage as separate concerns. Generation is the step that derives evaluation cases from documents or nodes, usually with an LLM. A query-response dataset is the artifact you keep, version, and reuse. Llama Dataset support is the broader dataset layer that lets examples become stable assets rather than one-off notebook variables. That separation matters because the most useful evaluation results compare application behavior against an unchanged set of examples.

System-to-Code Mapping

The readers API pages map directly to ingestion choices. docs/api_reference/api_reference/readers/datasets.md documents llama_index.readers.datasets with DatasetsReader, so it is the reference entry point when the input is already packaged as a dataset. docs/api_reference/api_reference/readers/github.md documents GithubRepositoryReader, GitHubRepositoryIssuesReader, and GitHubRepositoryCollaboratorsReader, which are useful when evaluation data should reflect repository content, issue discussions, or project metadata. docs/api_reference/api_reference/readers/azstorage_blob.md documents AzStorageBlobReader, which points to blob storage as a source for repeatable corpora. Sources: docs/api_reference/api_reference/readers/datasets.md, docs/api_reference/api_reference/readers/github.md, docs/api_reference/api_reference/readers/azstorage_blob.md

Airbyte and specialized search readers broaden the same pattern. The AirbyteCDKReader reference represents the generic Airbyte CDK path, while AirbyteGongReader represents a concrete Airbyte-backed business-system connector. AgentSearchReader is documented as another reader integration under the same generated reference structure. The important design signal is consistency: dataset work does not require a separate ingestion universe. You use the same reader family that powers indexing and RAG applications, then freeze the loaded material or generated query-response pairs into a dataset that can be replayed during evaluation. Sources: docs/api_reference/api_reference/readers/airbyte_cdk.md, docs/api_reference/api_reference/readers/airbyte_gong.md, docs/api_reference/api_reference/readers/agent_search.md

Repeatable Evaluation Flow

A reliable dataset workflow starts by choosing the source of truth. If your evaluation should measure answers over product docs, load those docs from the same location your production ingestion uses. If it should measure repository-assistant behavior, use the GitHub reader family so that examples are grounded in repository files or issues. If it should measure enterprise search or SaaS data, use the appropriate connector reader, such as Azure Blob Storage or an Airbyte integration. This keeps test cases aligned with real user tasks and prevents evaluation from drifting into artificial examples that do not exercise the deployed system. Sources: docs/api_reference/api_reference/readers/github.md, docs/api_reference/api_reference/readers/azstorage_blob.md, docs/api_reference/api_reference/readers/airbyte_gong.md

After loading source material, generate or curate examples. The official evaluation reference identifies DatasetGenerator as the generation entry point and QueryResponseDataset as the object shape for query-response examples. A typical flow is: load documents with a reader, generate candidate questions from those documents, inspect or filter the generated examples, persist the accepted examples, and then run evaluators against the same dataset whenever the application changes. The review step is important because generated questions can overfit to local phrasing, ask about irrelevant details, or fail to represent user intent. A good dataset is not merely large; it is stable, explainable, and tied to the behavior you want to protect.

Once the dataset exists, use it as a regression harness. Run the same query set against different retrievers, index configurations, LLMs, prompts, and response synthesizers. Compare answer quality, context relevance, faithfulness, and retrieval behavior on the fixed examples rather than changing both the system and the benchmark at once. This is where Llama Dataset support becomes a workflow primitive rather than a storage detail: it lets the evaluation asset live alongside code, notebooks, or experiment infrastructure. When a change improves one metric but hurts another, the fixed dataset gives the team a shared object for discussion instead of anecdotal prompt testing.

API Reference Snapshot

Area	Public names shown by the docs	Use in dataset workflows
Dataset reader	DatasetsReader	Load an existing dataset through the readers API surface.
GitHub readers	GithubRepositoryReader, GitHubRepositoryIssuesReader, GitHubRepositoryCollaboratorsReader	Build evaluation corpora from repository content, issues, or collaborator metadata.
Object storage reader	AzStorageBlobReader	Load source files from Azure Blob Storage for repeatable document-based evaluations.
Search reader	AgentSearchReader	Bring externally searched content into LlamaIndex as reader-loaded material.
Airbyte readers	AirbyteCDKReader, AirbyteGongReader	Use Airbyte-backed connectors for business-system or custom-source evaluation corpora.
Evaluation datasets	DatasetGenerator, QueryResponseDataset	Generate and store query-response evaluation examples.
Llama Dataset	llama_index.core.llama_dataset	Work with LlamaIndex dataset assets as a reusable evaluation layer.

This reference snapshot should be read as a map, not as a replacement for the generated API pages. The reader pages define importable documentation targets for source loading, while the evaluation and Llama Dataset docs define the dataset objects used after loading. If you are building a new benchmark, start with the reader that matches your source system, then move to generation and curation. If you already have a benchmark, start with DatasetsReader or the Llama Dataset APIs and focus on running consistent evaluation jobs across application versions. Sources: docs/api_reference/api_reference/readers/datasets.md, docs/api_reference/api_reference/readers/github.md, docs/api_reference/api_reference/readers/azstorage_blob.md

Practical Guidance and Next Steps

Prefer small, intentional datasets before broad automated generation. A compact set of high-signal examples can catch regressions in retrieval, grounding, and answer style more clearly than thousands of unreviewed questions. Use connector readers to keep source material reproducible, record how examples were generated, and version the resulting dataset with the application or experiment configuration that uses it. When changing an index, model, or prompt, run the same dataset before and after the change and inspect both aggregate metrics and individual failures. Next, read the evaluation pages for evaluator behavior, the readers and data-loading page for source connectors, and the ingestion pipeline pages for preparing documents before dataset generation.