Documents and Loaders

Purpose and Scope

Documents and loaders are the ingestion boundary for LangChain retrieval and data-processing workflows. A loader is responsible for reading information from a source system, and a Document is the normalized object that downstream components can index, split, embed, retrieve, or inspect. This page focuses on the Python langchain-core contracts rather than a specific integration package. The official integration docs describe document loaders as a standard interface for reading from sources such as local files, Slack, Notion, or Google Drive into LangChain’s Document format; the core package provides the base classes that make those integrations interchangeable.

The most important design rule is separation of concerns. Raw bytes or text are represented by Blob, parsed text and metadata are represented by Document, and loader implementations expose lazy iteration so large sources do not have to be loaded into memory all at once. This is why the core interfaces define both convenience methods, such as load(), and streaming-style methods, such as lazy_load() and yield_blobs(). Applications can start with simple eager loading, then move to lazy ingestion when datasets grow without changing the document shape consumed by splitters, vector stores, or retrievers.

Sources: libs/core/langchain_core/documents/base.py, libs/core/langchain_core/document_loaders/base.py, libs/core/langchain_core/document_loaders/blob_loaders.py

Relevant Source Files

• libs/core/langchain_core/documents/base.py defines BaseMedia, Blob, Document, shared metadata behavior, and the note that these classes are for retrieval and data processing rather than chat-message multimodal content.
• libs/core/langchain_core/document_loaders/base.py defines BaseLoader, eager and lazy load methods, async loading, load_and_split(), and the BaseBlobParser contract used to turn raw blobs into documents.
• libs/core/langchain_core/document_loaders/blob_loaders.py defines BlobLoader, the raw-content loader interface, and re-exports Blob and PathLike for compatibility.
• libs/core/langchain_core/document_loaders/langsmith.py defines LangSmithLoader, which loads LangSmith dataset examples as Document objects for few-shot example retrieval and related workflows.

Core Primitives

BaseMedia is the shared base class for content that may be stored, indexed, or searched. It supplies an optional id and a free-form metadata dictionary. The source comments state that the identifier is optional today and ideally unique across a document collection, with UUID formatting recommended but not enforced. The metadata field is intentionally arbitrary, because ingestion pipelines often need to preserve source URLs, file paths, dataset attributes, split labels, timestamps, permissions, or other context that later retrieval and filtering steps can use.

Document is the normalized text-bearing object used by retrieval-augmented generation, vector stores, semantic search, and other data-processing flows. The module documentation explicitly distinguishes documents from LLM chat messages: document classes are not the representation for images, audio, or other multimodal content sent to a model in a conversation. For chat I/O, LangChain uses message content blocks instead. That distinction helps prevent an ingestion pipeline from becoming coupled to a provider-specific chat format; loaders produce documents, and model-facing code later decides how retrieved text should be presented in prompts or messages.

Blob sits one layer earlier in the pipeline. It represents raw data, either in memory or by reference to a path, and carries fields such as data, mimetype, encoding, and path. The class docstring shows examples for creating blobs from in-memory data, attaching MIME type and metadata, creating blobs from file paths, reading as strings, reading as bytes, and opening byte streams. This lets a loader fetch raw content without also owning the parsing strategy, which is especially useful when the same file storage layer can contain many content types.

Sources: libs/core/langchain_core/documents/base.py

Loader Interfaces and Execution Flow

BaseLoader is the public interface for implementations that return Document objects. Its docstring recommends implementing lazy loading with generators to avoid reading every document into memory. The eager load() method is deliberately provided for user convenience and simply materializes list(self.lazy_load()). This gives application code a simple entry point for small ingestion jobs while preserving a scalable implementation path for integrations that read directories, remote APIs, databases, or datasets with many records.

The async methods mirror the same contract. aload() collects documents from alazy_load(), while alazy_load() adapts the synchronous lazy_load() iterator through run_in_executor. That implementation detail matters for developers writing integrations: a synchronous loader can still participate in async application code, but the core abstraction continues to encourage a single lazy iterator as the main implementation point. If a subclass does not implement lazy_load() and has not overridden load(), the base class raises NotImplementedError, making incomplete loader implementations fail explicitly.

load_and_split() is a compatibility convenience that loads documents and then applies a text splitter. The method accepts an optional TextSplitter; if none is provided, it attempts to use RecursiveCharacterTextSplitter from langchain-text-splitters. If the text splitter package is not installed, the method raises an ImportError explaining how to install it or pass a splitter explicitly. The source marks load_and_split() with a danger note saying not to override it and that it should be considered deprecated, so new code should prefer composing loaders and splitters directly.

Sources: libs/core/langchain_core/document_loaders/base.py

from langchain_core.document_loaders import BaseLoader
 
class MyLoader(BaseLoader):
    def lazy_load(self):
        # yield Document(...) objects without reading the whole source at once
        ...
 
docs = MyLoader().load()          # eager convenience
chunks = splitter.split_documents(docs)

Blob Loading and Parsing

BlobLoader is the raw-content counterpart to BaseLoader. Its single abstract method, yield_blobs(), returns an iterator of Blob objects. The module-level comment states the goal directly: decouple content loading from content parsing, and make lazy loading the default. In practice, a blob loader might know how to walk a directory, page through object storage, or fetch attachments from a remote system. It should not also need to know every parser required for every MIME type or file extension it encounters.

BaseBlobParser, defined alongside BaseLoader, completes that separation. A parser turns a Blob into one or more Document objects via a lazy parse method. This composition lets teams reuse a parser across storage systems, or reuse a storage loader with multiple parsers. For example, a filesystem blob loader and a cloud bucket blob loader can both emit Blob objects, while a text parser, PDF parser, or custom domain parser decides how raw bytes become page content and metadata. That is the core ingestion architecture behind many loader integrations.

The compatibility exports in blob_loaders.py also matter for existing code. The file re-exports Blob and PathLike from langchain_core.documents.base and includes them in __all__ with BlobLoader. That means older import paths can continue to work while the canonical definitions remain in the document module. When authoring new code, prefer thinking in terms of the layers: blob loaders fetch raw data, blob parsers convert raw data to documents, and document loaders expose finished documents to the rest of LangChain.

Sources: libs/core/langchain_core/document_loaders/blob_loaders.py, libs/core/langchain_core/document_loaders/base.py

LangSmith Dataset Loader

LangSmithLoader is a concrete BaseLoader for LangSmith datasets. It loads dataset examples as Document objects by placing example inputs into document page content and storing the entire example in metadata. The class docstring calls out a common use case: creating few-shot example retrievers. In that pattern, examples from a LangSmith dataset become searchable documents; a retriever can select relevant examples for a new input, and an agent or chain can include those examples in a prompt.

The constructor exposes dataset selection and filtering options that map to common dataset workflows. Callers can select by dataset_id or dataset_name, restrict to example_ids, retrieve examples as of a dataset version tag or timestamp with as_of, select dataset splits, and apply metadata or structured filter criteria. Pagination-style controls are represented by offset and limit, and inline_s3_urls controls whether S3 URLs are inlined. These arguments make the loader suitable for both small example sets and curated slices of larger evaluation or training datasets.

Content extraction is configurable. content_key chooses which key from the example inputs becomes the document page content, with dot-separated paths interpreted as nested keys. format_content converts the selected value into a string, and the default behavior is JSON stringification. The loader also accepts an existing LangSmith Client or keyword arguments for constructing one, but raises a ValueError if both are provided. That validation keeps ownership of client configuration unambiguous and prevents accidentally mixing credentials, endpoints, or other client settings.

Sources: libs/core/langchain_core/document_loaders/langsmith.py

from langchain_core.document_loaders import LangSmithLoader
 
loader = LangSmithLoader(
    dataset_name="few-shot-examples",
    content_key="question",
    limit=100,
 )
 
docs = list(loader.lazy_load())

API Reference

The table summarizes the implementation contracts, but the operational guidance is straightforward: implement the most lazy interface available, keep raw loading separate from parsing when possible, and preserve useful source information in metadata. A custom loader should return Document objects that are already meaningful to downstream components, not opaque provider payloads. A custom blob loader should return Blob objects with enough MIME type, path, and metadata information for parsers to make good decisions without depending on the original storage client.

Next Steps

For a new ingestion integration, start by deciding whether your source produces already-parsed text or raw content. If it produces records that can directly become text, implement BaseLoader.lazy_load() and yield Document objects. If it produces files, attachments, bytes, or mixed content types, implement BlobLoader.yield_blobs() and pair it with one or more blob parsers. Use load() only as the user-facing eager shortcut, and prefer explicit splitter composition over relying on load_and_split() for new workflows.

After documents are loaded, they usually flow into text splitters, embeddings, vector stores, retrievers, or indexing APIs. Keep metadata stable across those steps so retrieval results can be traced back to their original source, dataset example, file path, or access policy. If you are using LangSmith datasets, LangSmithLoader provides a direct bridge from examples to documents, which is useful for building retrievers over curated examples before adding them to prompts, agents, or evaluation workflows.