Ingestion Pipelines

Purpose and Scope

Ingestion pipelines are the repeatable data-preparation layer between readers and indexes. A reader brings raw content into LlamaIndex as documents; an ingestion pipeline turns those documents into nodes that can be indexed, retrieved, evaluated, and observed consistently. The first-party ingestion guide frames this around IngestionPipeline, which applies an ordered list of transformations to input data. Typical transformations include splitting documents into chunks, extracting metadata such as titles, and computing embeddings before the resulting nodes are returned or inserted into a vector store.

The important design idea is that ingestion is not just a one-time preprocessing script. It is a reusable contract for how source data becomes indexable retrieval units. The official guide also calls out caching at the node-and-transformation level, so repeated runs can avoid recomputing work when the same input and transformation combination appears again. That makes pipelines useful for local notebooks, production refresh jobs, and distributed workflows where ingestion cost is dominated by parsing, metadata extraction, and embedding calls. Sources: docs/api_reference/api_reference/ingestion/index.md, docs/api_reference/api_reference/ingestion/ray.md

Relevant Source Files

• docs/api_reference/api_reference/ingestion/index.md - API reference entry for llama_index.core.ingestion.pipeline, specifically documenting IngestionPipeline and DocstoreStrategy.
• docs/api_reference/api_reference/ingestion/ray.md - API reference entry for llama_index.ingestion.ray, documenting RayIngestionPipeline for distributed ingestion support.
• docs/examples/cookbooks/oreilly_course_cookbooks/README.md - Course outline that places metadata extraction and ingestion pipelines in the practical RAG learning path.

Core Primitives

The central primitive is IngestionPipeline. It owns the ordered transformation list and exposes the run-time operation that accepts documents and produces nodes. In the common pattern, SentenceSplitter or another node parser runs first so later transformations operate on manageable node chunks. Metadata extractors, such as a title extractor, can then enrich nodes with fields useful for retrieval filters, ranking, or response display. Finally, an embedding model transformation can attach vectors that downstream vector indexes and vector stores use for similarity search.

DocstoreStrategy is part of the core ingestion API reference alongside IngestionPipeline, which signals that ingestion also participates in document-store coordination rather than only in-memory transformation. At a high level, a document store strategy determines how the pipeline relates newly produced nodes to stored document state, refresh behavior, and duplicate handling. Treat this as the policy layer for keeping indexed data aligned with source data when ingestion is rerun. The API reference entry is intentionally compact, so application code should pair it with the module guide and the concrete pipeline examples. Sources: docs/api_reference/api_reference/ingestion/index.md

Basic Execution Flow

A minimal ingestion flow starts with source documents, configures transformations, and runs the pipeline. In a real application, the documents usually come from SimpleDirectoryReader or a connector package, but the guide uses Document.example() to show the mechanics without adding a data-loading dependency. The output is a list of nodes. Those nodes can be passed to an index directly, inspected in tests, stored for later processing, or compared against expected metadata during evaluation and debugging.

from llama_index.core import Document
from llama_index.core.extractors import TitleExtractor
from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding
 
pipeline = IngestionPipeline(
    transformations=[
        SentenceSplitter(chunk_size=25, chunk_overlap=0),
        TitleExtractor(),
        OpenAIEmbedding(),
    ]
)
 
nodes = pipeline.run(documents=[Document.example()])

The execution order matters because each transformation receives the output shape created by earlier transformations. Splitting before metadata extraction means titles and embeddings are attached to retrieval-sized units rather than to large source documents. Embedding after text and metadata transformations means the embedded representation corresponds to the final text content that retrieval will search over. This sequence is why ingestion pipelines are the natural place to centralize chunk size, overlap, metadata enrichment, and model-dependent preprocessing decisions instead of scattering them across indexing notebooks.

Connecting Pipelines to Vector Stores

The ingestion guide also supports a production-oriented path where the pipeline inserts resulting nodes into a vector database as part of the run. In that setup, the vector store is passed to IngestionPipeline, and the pipeline becomes both the transformation runner and the writer for vector-ready nodes. This is useful when the operational boundary is a remote vector database such as Qdrant: ingestion can be scheduled or distributed independently, and the query application can later construct an index from the existing vector store.

from llama_index.core import Document
from llama_index.core.extractors import TitleExtractor
from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.vector_stores.qdrant import QdrantVectorStore
 
import qdrant_client
 
client = qdrant_client.QdrantClient(location=":memory:")
vector_store = QdrantVectorStore(client=client, collection_name="test_store")
 
pipeline = IngestionPipeline(
    transformations=[
        SentenceSplitter(chunk_size=25, chunk_overlap=0),
        TitleExtractor(),
        OpenAIEmbedding(),
    ],
    vector_store=vector_store,
    )
 
pipeline.run(documents=[Document.example()])

When using this pattern, think of the pipeline as the authoritative definition of what enters retrieval. If the chunking configuration changes, the vector database may contain nodes with a different granularity than earlier runs. If the embedding model changes, vector similarity semantics also change. Caching can speed repeated runs, but it should be used with awareness of transformation identity and persistence. For production refresh workflows, keep transformation configuration, embedding model choice, vector store collection names, and document-store strategy under versioned application configuration.

Distributed Ingestion with Ray

RayIngestionPipeline is documented in the ingestion Ray API reference under llama_index.ingestion.ray. The naming and placement indicate a distributed counterpart to the core ingestion pipeline for workloads that need parallel execution across Ray workers. Use this page when a single-process ingestion loop becomes too slow because of large document collections, expensive metadata extraction, or embedding throughput limits. The public reference entry is the stable discovery point for the Ray-specific class, while the core ingestion page remains the shared conceptual contract for transformations and node production. Sources: docs/api_reference/api_reference/ingestion/ray.md

Distributed ingestion should preserve the same application-level assumptions as local ingestion: documents are transformed into nodes, transformations should be deterministic enough to cache and rerun, and the output must match the index or vector store expected by retrieval. The operational difference is where the work happens. Ray can help fan out CPU-heavy parsing or high-latency model calls, but it also makes configuration discipline more important because workers need access to the same readers, transformation dependencies, credentials, and persistence targets.

API Reference Snapshot

The repository API reference pages are generated from import targets rather than long-form tutorials. That means the reference is best used to confirm names and module locations, while the module guide should be used for flow, examples, and mental model. In application code, prefer importing from documented package surfaces and keeping pipeline construction explicit. A clear pipeline declaration is easier to test than an implicit ingestion chain hidden inside index construction, and it provides one place to review chunking, metadata extraction, embedding, vector-store writing, and document-store policy.

Learning Path and Testing Signals

The O'Reilly course cookbook positions “Metadata Extraction and Ingestion Pipeline” as Module 4, after introductory RAG concepts, LlamaIndex components, and evaluation. That order is a useful signal for teams adopting LlamaIndex: learn how retrieval quality is measured before treating ingestion settings as fixed. Chunk size, overlap, metadata extractors, and embedding models all affect evaluator results, so ingestion work should be tied to retrieval and response-quality experiments rather than only to data loading convenience. Sources: docs/examples/cookbooks/oreilly_course_cookbooks/README.md

For a practical next step, build a small local pipeline, inspect the resulting nodes, and then add a vector store only after the node shape is correct. Once the local flow is stable, decide whether DocstoreStrategy behavior and persistent caching are needed for refresh jobs. If ingestion runtime becomes the bottleneck, evaluate RayIngestionPipeline with the same transformation list and storage targets. Related pages to read next are Readers and Data Loading, Node Parsers, Metadata Extraction, Vector Store Indexing, Persistence, and Evaluating Applications.