Documents and Nodes

Purpose and Scope

Documents and nodes are the data schema that most LlamaIndex workflows pass between loading, parsing, indexing, retrieval, and response synthesis. A Document is the source-level container: it can represent a PDF, an API result, database output, or manually supplied text. A Node is the chunk-level unit derived from a source document, often text but also capable of representing image or other modality chunks as multimodal support evolves. This page explains how those objects are created, customized, and connected so that downstream indexes and query systems can preserve source context.

Sources: docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/index.md

The reader problem is usually not just “how do I make a text object?” In a RAG application, the object shape determines what is embedded, what metadata appears to the LLM, how retrieved sources are attributed, and how updates are matched back to existing indexed content. LlamaIndex treats nodes as first-class citizens, so you can either let a parser split documents for you or construct nodes directly when you need explicit IDs, relationships, or specialized metadata. Understanding this boundary prevents surprises later when building vector indexes, ingestion pipelines, or document refresh flows.

Sources: docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/index.md, docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/usage_documents.md, docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/usage_nodes.md

Relevant Source Files

• docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/_meta.yml - Declares the documentation navigation label, Documents And Nodes, and indicates that this guide group is collapsed in the docs tree.
• docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/index.md - Defines the core concepts for Document and Node, introduces metadata and relationships, and shows starter usage for VectorStoreIndex.from_documents and SentenceSplitter.
• docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/usage_documents.md - Provides the detailed document guide, including data loader output, manual construction, Document.example(), metadata, doc_id, filename_as_id, and metadata visibility controls.
• docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/usage_metadata_extractor.md - Shows how metadata extractor transformations such as TitleExtractor and QuestionsAnsweredExtractor can be chained with text splitting in an IngestionPipeline or passed into VectorStoreIndex.from_documents.
• docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/usage_nodes.md - Provides the detailed node guide, including SentenceSplitter, manual TextNode creation, NodeRelationship, RelatedNodeInfo, and node_id customization.
• docs/src/content/docs/framework/community/faq/documents_and_nodes.md - Captures FAQ-level behavior: default node chunk_size, adding document metadata, and using doc_id for updating or deleting documents in an index.

Core Schema Concepts

A Document is designed for the loading boundary. Data loaders, including SimpleDirectoryReader and LlamaHub-style connectors, return Document objects through load_data, while application code can also construct them manually. The default document carries text plus attributes such as metadata and relationships. Metadata is a dictionary of annotations; relationships describe links to other documents or nodes. This makes the document a portable envelope that can enter indexing directly or be parsed into smaller node objects before storage and retrieval.

Sources: docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/index.md, docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/usage_documents.md

A Node is the chunk-level representation used after parsing. The official guide describes a node as a chunk of a source document, with metadata and relationship information of its own. By default, every node derived from a document inherits that document’s metadata, which is important because retrieval usually returns nodes rather than whole original files. If a document has a file_name, category, URL, or other source annotation, that information can travel into each retrieved node and become available for source display, filtering, embedding text, or response synthesis.

Sources: docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/index.md, docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/usage_nodes.md

One subtle but important detail is that Document is described as a subclass of TextNode in the document customization guide. That means many text-node settings also apply to documents, even though developers normally think of documents as the input object and nodes as the parsed output object. In practice, you can begin with the higher-level Document abstraction for loading and metadata management, then move to explicit TextNode construction only when you need control over chunk boundaries, IDs, or relationships beyond what a node parser creates automatically.

Sources: docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/usage_documents.md, docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/usage_nodes.md

Creating and Customizing Documents

The simplest document flow is to load data, receive Document objects, and build an index from them. SimpleDirectoryReader('./data').load_data() is the documented loader example, and manual construction uses Document(text=t) for each text value in a list. The overview page then shows VectorStoreIndex.from_documents(documents) as the starter indexing path. This is the right path when you want LlamaIndex to handle parsing and indexing defaults for ordinary text ingestion rather than managing node objects yourself.

from llama_index.core import Document, SimpleDirectoryReader, VectorStoreIndex
 
documents = SimpleDirectoryReader('./data').load_data()
manual_documents = [Document(text=t) for t in text_list]
 
index = VectorStoreIndex.from_documents(documents)

Sources: docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/index.md, docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/usage_documents.md

Metadata can be supplied in several ways. You can pass a metadata dictionary to the constructor, assign document.metadata after creation, or use the file_metadata hook on SimpleDirectoryReader to derive metadata from each filename. The document guide emphasizes that metadata can be filenames, categories, or similar annotations, and also warns vector database users that some stores require metadata keys to be strings and values to be flat values such as str, float, or int. This constraint matters when the same document objects will be embedded into an external vector database.

from llama_index.core import Document, SimpleDirectoryReader
 
document = Document(
    text='text',
    metadata={'filename': '<doc_file_name>', 'category': '<category>'},
)
 
filename_fn = lambda filename: {'file_name': filename}
documents = SimpleDirectoryReader(
    './data', file_metadata=filename_fn
).load_data()

Sources: docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/usage_documents.md, docs/src/content/docs/framework/community/faq/documents_and_nodes.md

Document identity is separate from metadata. The guide calls out doc_id as the identifier used for efficient refresh and document management in an index. With SimpleDirectoryReader, filename_as_id=True sets each document’s doc_id to the full path, which is useful when files on disk are the system of record. You can also assign document.doc_id directly, and the guide notes that the ID can also be set through node_id or id_ on a Document, similar to a TextNode. Use stable IDs when updates and deletes are part of the workflow.

Sources: docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/usage_documents.md, docs/src/content/docs/framework/community/faq/documents_and_nodes.md

Parsing and Customizing Nodes

The standard node flow is to start with documents, instantiate a parser, and call get_nodes_from_documents. The docs use SentenceSplitter as the simple example, while the metadata extraction page uses TokenTextSplitter in a transformation chain. Once nodes are created, they can be passed directly into VectorStoreIndex(nodes). This is the more explicit path when you want to inspect chunks, add transformations, control chunking settings, or guarantee that the exact node list entering the index is the one your application prepared.

from llama_index.core import VectorStoreIndex
from llama_index.core.node_parser import SentenceSplitter
 
parser = SentenceSplitter()
nodes = parser.get_nodes_from_documents(documents)
 
index = VectorStoreIndex(nodes)

Sources: docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/index.md, docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/usage_nodes.md

Nodes can also be constructed manually with TextNode, NodeRelationship, and RelatedNodeInfo. The usage guide shows two text nodes with explicit id_ values and relationships for NEXT and PREVIOUS, plus a PARENT relationship carrying additional metadata. This is useful when chunk order, hierarchy, or graph-like navigation is part of the retrieval strategy. Instead of relying entirely on the parser’s default relationship behavior, application code can make relationships part of the schema that downstream indexes and tools can inspect.

from llama_index.core.schema import TextNode, NodeRelationship, RelatedNodeInfo
 
node1 = TextNode(text='<text_chunk>', id_='<node_id>')
node2 = TextNode(text='<text_chunk>', id_='<node_id>')
 
node1.relationships[NodeRelationship.NEXT] = RelatedNodeInfo(
    node_id=node2.node_id
)
node2.relationships[NodeRelationship.PREVIOUS] = RelatedNodeInfo(
    node_id=node1.node_id
)
node2.relationships[NodeRelationship.PARENT] = RelatedNodeInfo(
    node_id=node1.node_id, metadata={'key': 'val'}
)

Sources: docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/usage_nodes.md

Each node has a node_id property that is generated automatically unless you specify it. The guide describes this ID as useful for updating nodes in storage and defining relationships, including through IndexNode. The FAQ adds that the default node chunk_size is 1024 and points customization to the node parser configuration guide. Taken together, these details imply a practical rule: configure parsing when you want different chunk boundaries, and configure IDs or relationships when you need stable storage updates or structural navigation between chunks.

Sources: docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/usage_nodes.md, docs/src/content/docs/framework/community/faq/documents_and_nodes.md

Metadata Extraction and Transformation Flow

Metadata does not have to be typed by hand. The metadata extraction usage page describes LLM-powered Metadata Extractor modules, including SummaryExtractor, QuestionsAnsweredExtractor, TitleExtractor, and EntityExtractor. These extractors operate over nodes and can be chained with a node parser, making them part of the ingestion transformation sequence. The documented example combines TokenTextSplitter, TitleExtractor(nodes=5), and QuestionsAnsweredExtractor(questions=3) inside an IngestionPipeline, then runs the pipeline over documents with in_place=True and show_progress=True.

Sources: docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/usage_metadata_extractor.md

The same transformations can also be inserted directly into index construction through VectorStoreIndex.from_documents(documents, transformations=[...]). That gives two equivalent integration points: run the pipeline explicitly when you want the transformed nodes as an intermediate artifact, or pass transformations into the index builder when indexing is the immediate goal. Because document metadata is propagated to source nodes and metadata is injected into text for embeddings and LLM calls by default, extractor output can affect both retrieval quality and final answer context unless you customize metadata visibility.

from llama_index.core import VectorStoreIndex
from llama_index.core.extractors import TitleExtractor, QuestionsAnsweredExtractor
from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.node_parser import TokenTextSplitter
 
text_splitter = TokenTextSplitter(separator=' ', chunk_size=512, chunk_overlap=128)
title_extractor = TitleExtractor(nodes=5)
qa_extractor = QuestionsAnsweredExtractor(questions=3)
 
pipeline = IngestionPipeline(
    transformations=[text_splitter, title_extractor, qa_extractor]
)
nodes = pipeline.run(documents=documents, in_place=True, show_progress=True)
 
index = VectorStoreIndex.from_documents(
    documents, transformations=[text_splitter, title_extractor, qa_extractor]
)

Sources: docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/usage_documents.md, docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/usage_metadata_extractor.md

System-to-Code Mapping

Practical Guidance and Next Steps

For most applications, begin with Document objects, attach stable metadata and IDs, then let a parser create nodes during indexing. Move to manual node construction when your application needs ordered chunks, parent-child relationships, externally assigned node IDs, or metadata attached to specific relationships. If you need richer retrieval signals, add extractor transformations during ingestion so summaries, titles, questions, or entities become part of each node’s metadata. Finally, decide deliberately which metadata should be visible to embeddings and LLM prompts, because the document guide states that metadata is included in both by default.

Sources: docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/usage_documents.md, docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/usage_metadata_extractor.md, docs/src/content/docs/framework/module_guides/loading/documents_and_nodes/usage_nodes.md

Read this page before the indexing and ingestion guides because documents and nodes are the objects those systems consume. If your next task is loading files, continue to readers and data loading. If your next task is controlling chunk boundaries, continue to node parsers. If your next task is repeatable enrichment, continue to ingestion pipelines and metadata extraction. If your next task is incremental refresh, use stable doc_id values and the document management path referenced by the FAQ.

Sources: docs/src/content/docs/framework/community/faq/documents_and_nodes.md