Indexing API
Purpose and Scope
The indexing API is the part of LangChain that keeps a retrieval index synchronized with a changing document source. Its job is not only to add documents to a vector store or document index, but also to remember which document-derived keys have already been written, when they were written, and which source group they came from. That extra bookkeeping lets repeated indexing runs skip unchanged documents, update changed ones, and remove stale entries when a cleanup mode asks for deletion. In practice, this is the bridge between document loading, embedding-backed storage, and production maintenance of retrieval data.
Sources: libs/core/langchain_core/indexing/api.py, libs/core/langchain_core/indexing/base.py
The repository places this capability in core indexing modules while the LangChain Classic package README still calls out the indexing API as part of the classic package surface. That split matters for readers migrating code: the abstractions that make indexing portable live in langchain_core.indexing, while langchain-classic remains the package associated with legacy chains, community re-exports, deprecated functionality, and the indexing API in its package description. Treat the core files as the behavioral contract and the classic README as package-orientation evidence for where existing users may encounter the feature.
Sources: libs/core/langchain_core/indexing/api.py, libs/langchain/README.md
Relevant Source Files
libs/core/langchain_core/indexing/api.py— Implements the indexing flow: batching, deterministic document key generation, source ID assignment, sync and async indexing entry points, and hashing helpers used to decide whether a document has changed.libs/core/langchain_core/indexing/base.py— Defines the abstract contracts for indexing support, especiallyRecordManager,DocumentIndex, and response types that describe upsert and delete outcomes.libs/core/langchain_core/indexing/in_memory.py— ProvidesInMemoryDocumentIndex, a beta in-memory implementation of theDocumentIndexcontract with simple upsert, delete, get, and retriever behavior.libs/langchain/README.md— Identifies LangChain Classic as the package containing legacy chains, community re-exports, deprecated functionality, and the indexing API, with install and documentation pointers.
Core Primitives
A RecordManager is the durable ledger for indexing. It records a key for each indexed document, the time the key was written, and an optional group or source ID. The base class documentation explains why this is needed: the vector store only needs to support adding and deleting by ID, while the record manager supplies the cross-vector-store memory needed to avoid redundant indexing and delete outdated documents. The abstraction deliberately separates index storage from bookkeeping, which makes it portable across providers but introduces a distributed consistency boundary between the record database and the vector store.
Sources: libs/core/langchain_core/indexing/base.py
A DocumentIndex is the storage-facing abstraction used when the destination is document-oriented rather than a classic vector store interface. Its public shape is visible through the in-memory implementation: an index can upsert a sequence of Document objects, delete documents by IDs, retrieve documents by IDs, and behave as a retriever by returning relevant documents for a query. The in-memory implementation stores documents in a dictionary, assigns a UUID when a document has no ID, and returns structured success and failure information through response objects.
Sources: libs/core/langchain_core/indexing/base.py, libs/core/langchain_core/indexing/in_memory.py
A document key is the stable identity used by indexing, not necessarily the same thing as a storage provider’s internal ID. The API module hashes document content and metadata into deterministic UUID values under a fixed namespace. It also exposes a key_encoder concept in the SHA-1 warning text: SHA-1 is supported for compatibility but emits a one-time warning because it is not collision-resistant, and callers with a stronger threat model should choose stronger algorithms such as blake2b, sha256, or sha512. This means indexing identity is configurable, but it should be chosen deliberately when documents come from untrusted sources.
Sources: libs/core/langchain_core/indexing/api.py
Execution Flow
A typical indexing run starts with a source of Document objects, usually from a loader or iterable, plus a destination and a record manager. The API module imports BaseLoader, Document, VectorStore, DocumentIndex, and RecordManager, which shows the supported orchestration boundary: documents come from loaders or iterables, are assigned deterministic keys, are written to a storage target, and then have those keys recorded. The batching helpers enforce a positive batch size before yielding chunks, so invalid batch configuration fails early rather than partially indexing an unbounded stream.
Sources: libs/core/langchain_core/indexing/api.py
During a run, the indexer computes a hash-based key for each document and asks the record manager which keys already exist. New or changed documents are upserted into the target, while unchanged documents can be skipped. The record manager is then updated with the keys, optional group IDs, and a timestamp. The base class emphasizes that timestamps must be monotonically increasing and should come from the server, because cleanup decisions depend on comparing write times. If a client clock moves backward, an otherwise valid cleanup run could accidentally treat fresh records as stale.
Sources: libs/core/langchain_core/indexing/base.py
Cleanup is the part of indexing that turns a one-way ingestion job into a synchronization job. The API supports cleanup modes that determine when old records are eligible for deletion. In incremental cleanup, stale entries associated with a source group can be removed as newer versions are discovered. In full-style cleanup, deletion is delayed until the run has a complete view of what still exists. In scoped full cleanup, the source ID assignment is especially important because it bounds deletion to a group instead of treating the whole namespace as replaceable. Use a source_id_key string or callable when documents should be grouped by original source.
Sources: libs/core/langchain_core/indexing/api.py, libs/core/langchain_core/indexing/base.py
API Reference
| Component | Contract | Notes |
|---|---|---|
RecordManager(namespace) | Abstract ledger for indexed document keys | Requires schema creation, server time, update, existence checks, listing, and deletion methods in sync and async forms. |
DocumentIndex | Abstract destination for document upsert/delete/get behavior | Used by InMemoryDocumentIndex and compatible indexes that can store LangChain Document objects by ID. |
InMemoryDocumentIndex.upsert(items, **kwargs) | Adds or replaces documents | Generates a UUID for documents without Document.id and returns UpsertResponse(succeeded=..., failed=...). |
InMemoryDocumentIndex.delete(ids, **kwargs) | Deletes documents by ID | Raises ValueError when IDs are omitted and returns deletion counts and failed IDs. |
_batch(size, iterable) and _abatch(size, iterable) | Sync and async batching helpers | Reject non-positive sizes with ValueError. |
_get_source_id_assigner(source_id_key) | Converts a source ID option into a callable | Supports None, metadata-key strings, and callables over Document. |
_hash_string(..., algorithm=...) | Deterministic UUID hashing helper | Supports SHA-1 compatibility with a warning and stronger alternatives including SHA-256, SHA-512, and BLAKE2b. |
The important public shape is that indexing returns counts and uses response objects rather than assuming every write succeeds. InMemoryDocumentIndex.upsert returns separate succeeded and failed ID lists, while delete reports succeeded IDs, deleted count, failed count, and failed IDs. Production indexes should preserve that style because the higher-level indexing flow needs to distinguish “already skipped,” “successfully added,” “updated,” and “deleted” from partial write failures. When implementing a custom destination, make failure reporting explicit instead of swallowing provider errors into silent skips.
Sources: libs/core/langchain_core/indexing/base.py, libs/core/langchain_core/indexing/in_memory.py
Implementation Details and Constraints
The RecordManager documentation names the two biggest operational constraints. First, timestamp ordering is central to correctness, so the implementation should use a server-side monotonic time source through get_time and aget_time. Second, the record manager and vector store are separate systems. If record updates succeed but vector store writes fail, or the reverse happens, the next run must reconcile that split through existence checks, updates, and cleanup. This is the tradeoff that lets one record-manager abstraction work with many vector stores that only know how to add and delete IDs.
Sources: libs/core/langchain_core/indexing/base.py
Hashing is another design constraint. The API serializes nested metadata deterministically before hashing, which helps make document identity stable across runs when metadata dictionaries have different key orderings. The default SHA-1 path is intentionally noisy: it warns once that SHA-1 is not collision-resistant and points readers to stronger algorithms through the key_encoder parameter. For ordinary trusted ingestion, the default may be adequate for compatibility. For adversarial documents, compliance-sensitive data, or multi-tenant ingestion, choose a stronger algorithm and document that choice with your indexing configuration.
Sources: libs/core/langchain_core/indexing/api.py
Practical Usage Pattern
Before running indexing in an application, create the record-manager schema, choose a namespace, and decide which metadata field identifies the source. A namespace should correspond to one logical index, tenant, or collection, because cleanup operates within the record manager’s namespace. A source ID should correspond to the unit you can safely replace, such as a file path, URL, database row group, or customer-owned corpus partition. If you cannot assign a reliable source ID, avoid scoped cleanup modes that depend on grouping because the indexer will not know which stale records belong together.
Sources: libs/core/langchain_core/indexing/api.py, libs/core/langchain_core/indexing/base.py
A minimal development loop can use InMemoryDocumentIndex to validate document IDs and deletion behavior before connecting to a real vector store. Because it stores documents in memory and ranks results by counting query occurrences in page_content, it is useful as a contract example rather than a semantic search backend. Its behavior demonstrates how Document.id participates in storage, how generated IDs are assigned when missing, and how retriever compatibility can be layered on top of an index. Move to a provider-backed vector store when you need embeddings, persistence, concurrency, or production-scale retrieval.
Sources: libs/core/langchain_core/indexing/in_memory.py
Next Steps
Use this page when designing ingestion jobs for retrieval-augmented generation systems. Start by defining the record-manager namespace and source ID strategy, then select a cleanup mode that matches how complete each run’s document view is. For local contract tests, exercise InMemoryDocumentIndex; for production, pair the indexing API with a vector store that supports add and delete by ID. Read the related pages on documents and loaders, embeddings and vector stores, retrievers, and LangChain Core to connect this synchronization layer to the rest of the retrieval stack.
Sources: libs/core/langchain_core/indexing/api.py, libs/core/langchain_core/indexing/base.py, libs/core/langchain_core/indexing/in_memory.py, libs/langchain/README.md