Text Splitters

Purpose and Scope

Text splitters are the preparation layer between raw source material and retrieval-ready Document chunks. In LangChain, a splitter turns long strings, structured markup, or nested data into smaller pieces that can be embedded, indexed, retrieved, and passed to models without exceeding context limits. The official docs recommend starting with recursive character splitting for many workflows because it tries to preserve larger semantic units before falling back to smaller ones. In the Python package, that recommendation maps directly to RecursiveCharacterTextSplitter, exported from the langchain_text_splitters package and implemented alongside lower-level character splitting helpers.

Sources: libs/text-splitters/langchain_text_splitters/init.py, libs/text-splitters/langchain_text_splitters/character.py

This page focuses on the standalone langchain_text_splitters package and the concrete source paths requested for text, HTML, JSON, and language-parser compatibility behavior. The package is intentionally not just a set of string utilities. Its base splitter implements LangChain's document transformation contract by importing BaseDocumentTransformer and Document from langchain_core.documents, so splitters can participate in retrieval and indexing pipelines as document transformers rather than as isolated preprocessing functions. That matters when preserving metadata, creating chunk documents, or composing loaders, splitters, embeddings, vector stores, and retrievers.

Sources: libs/text-splitters/langchain_text_splitters/base.py

Relevant Source Files

• libs/text-splitters/langchain_text_splitters/__init__.py — Defines the package-level public exports for base, character, HTML, JSON, Markdown, language, token, and tokenizer-oriented splitters.
• libs/text-splitters/langchain_text_splitters/base.py — Defines the abstract TextSplitter, constructor validation, document creation behavior, token helpers, and shared splitting contract.
• libs/text-splitters/langchain_text_splitters/character.py — Implements CharacterTextSplitter, RecursiveCharacterTextSplitter, regex splitting, separator handling, and recursive separator selection.
• libs/text-splitters/langchain_text_splitters/html.py — Implements HTML-oriented splitters and supporting HTML element typing for preserving header and document structure.
• libs/text-splitters/langchain_text_splitters/json.py — Implements RecursiveJsonSplitter for chunking nested JSON while preserving hierarchy.
• libs/langchain/langchain_classic/document_loaders/parsers/language/javascript.py — Provides the classic compatibility import path for JavaScriptSegmenter through a deprecated dynamic lookup into the community package.

Core Primitives

The central primitive is TextSplitter, an abstract class with a required split_text(text: str) -> list[str] method. Its constructor defines the common knobs that downstream splitters share: chunk_size, chunk_overlap, a length_function, keep_separator, add_start_index, and strip_whitespace. The implementation validates that chunk size is positive, overlap is nonnegative, and overlap does not exceed chunk size. Those checks prevent silent indexing mistakes where every chunk is too small, overlapping windows never advance, or metadata later implies offsets that do not make sense.

Sources: libs/text-splitters/langchain_text_splitters/base.py

TextSplitter also bridges text splitting and LangChain documents. Its create_documents method accepts a list of texts plus optional metadata dictionaries, runs split_text for each input, deep-copies metadata for every emitted chunk, and can attach the chunk start index when add_start_index is enabled. This is the behavior that makes splitter output useful for retrieval: a vector store can retain source metadata, a retriever can surface the original document context, and an application can trace an answer back to the position of a chunk rather than only to an unstructured string.

Sources: libs/text-splitters/langchain_text_splitters/base.py

The package initializer exposes the main public vocabulary. It re-exports TextSplitter, TokenTextSplitter, Tokenizer, split_text_on_tokens, CharacterTextSplitter, RecursiveCharacterTextSplitter, HTMLHeaderTextSplitter, HTMLSectionSplitter, HTMLSemanticPreservingSplitter, RecursiveJsonSplitter, and multiple specialized splitters for Markdown, Python, LaTeX, spaCy, NLTK, KoNLPy, sentence transformers, and JSX. The initializer also documents an important inheritance detail: MarkdownHeaderTextSplitter and HTMLHeaderTextSplitter do not derive from TextSplitter, so code that expects the base abstract interface should not assume every exported splitter is a subclass.

Sources: libs/text-splitters/langchain_text_splitters/init.py

Execution Flow in Retrieval and Indexing

A typical retrieval workflow starts with loaded content, applies a splitter, embeds the resulting chunks, stores them in a vector store, and later retrieves relevant chunks for a model call. The split step is where the application chooses the unit of retrieval. Chunks that are too large waste model context and may bury the relevant sentence; chunks that are too small can lose surrounding meaning. LangChain's character splitters encode this tradeoff with chunk_size and chunk_overlap, while the shared base class keeps the resulting chunks compatible with Document-based downstream APIs.

Sources: libs/text-splitters/langchain_text_splitters/base.py, libs/text-splitters/langchain_text_splitters/character.py

For most plain text, RecursiveCharacterTextSplitter is the practical default because it tries separators in order. The implementation defaults to paragraph-like breaks, then line breaks, then spaces, and finally individual characters. It recursively chooses a separator that appears in the text and continues splitting oversized fragments with the next available separator. This preserves natural language structure when possible while still guaranteeing that unusually long sections can be broken down. The default keep_separator value is true, so separator context can remain attached to chunks rather than being discarded.

Sources: libs/text-splitters/langchain_text_splitters/character.py

A minimal Python flow looks like this:

from langchain_text_splitters import RecursiveCharacterTextSplitter
 
splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=100,
    add_start_index=True,
 )
 
documents = splitter.create_documents(
    [raw_text],
    metadatas=[{"source": "handbook.md"}],
 )

This example uses public exports from the package initializer and shared constructor behavior from the base class. The important design point is that callers can choose a splitter based on source format, but the downstream shape remains predictable: a list of chunks or a list of LangChain Document objects. The metadata copy behavior means chunk metadata can be safely modified later without accidentally mutating metadata for sibling chunks, which is especially useful when an indexing job adds collection names, source URLs, language tags, or cleanup bookkeeping after splitting.

Sources: libs/text-splitters/langchain_text_splitters/init.py, libs/text-splitters/langchain_text_splitters/base.py

API Components and Options

CharacterTextSplitter is the direct separator-based implementation. It accepts a separator, an is_separator_regex flag, and the shared TextSplitter keyword options. Its split_text method escapes the separator unless regex mode is enabled, delegates to _split_text_with_regex, detects zero-width regex lookaround separators, and decides whether to reinsert the separator when merging splits. This detail is important for precise chunking: lookaround patterns define split positions rather than characters that should be reintroduced, while literal separators can be restored when the caller did not request that they be kept in the split pieces.

Sources: libs/text-splitters/langchain_text_splitters/character.py

RecursiveCharacterTextSplitter extends the same base contract but works across a separator hierarchy. Its constructor accepts separators, keep_separator, and is_separator_regex; when separators are not supplied it uses a default hierarchy from larger text boundaries down to the empty string. Use this splitter when the application cares about preserving paragraph, sentence, or word-level coherence but still needs an upper bound on chunk size. Use CharacterTextSplitter when the input has a known delimiter, such as records separated by a custom marker or a format where a single separator is semantically meaningful.

Sources: libs/text-splitters/langchain_text_splitters/character.py

RecursiveJsonSplitter handles structured JSON rather than plain text. It accepts max_chunk_size and optional min_chunk_size, computes serialized JSON sizes, can convert lists into dictionaries with index-based keys for better recursive chunking, and preserves nested paths as it builds chunk dictionaries. This is a different contract from TextSplitter: the output can remain structured dictionaries or be converted into JSON-formatted strings and documents by methods on the splitter. Use it when retrieval needs to preserve field hierarchy, such as API payloads, configuration trees, product catalogs, or event records.

Sources: libs/text-splitters/langchain_text_splitters/json.py

HTML splitting is similarly structure-aware. HTMLHeaderTextSplitter splits HTML by configured header tags and creates Document objects whose metadata reflects the encountered hierarchy. The source documents that if no configured headers are found, the content can be returned as a single document, and it supports returning each element separately or aggregating elements into semantically meaningful chunks. The module also defines ElementType with url, xpath, content, and metadata, signaling that HTML splitting can preserve both textual content and location-oriented metadata for later attribution.

Sources: libs/text-splitters/langchain_text_splitters/html.py

System-to-Code Mapping

The compatibility file under langchain_classic is worth understanding if you maintain older loader or parser code. It defines DEPRECATED_LOOKUP for JavaScriptSegmenter, creates an importer with create_importer, implements __getattr__, and exposes JavaScriptSegmenter in __all__. That means legacy imports can still resolve dynamically while centralizing deprecation warnings and optional import handling. New text splitting code should prefer the dedicated langchain_text_splitters exports, while older language parser paths can remain operational during migration.

Sources: libs/langchain/langchain_classic/document_loaders/parsers/language/javascript.py, libs/text-splitters/langchain_text_splitters/init.py

Implementation Details and Practical Guidance

Choose a splitter by the structure you want retrieval to respect. Plain prose often benefits from recursive character splitting because paragraphs and lines carry meaning. HTML pages benefit from header-aware splitting because headings are valuable metadata for search results and answer citations. JSON records benefit from recursive JSON splitting because flattening everything into plain text can discard key paths that explain what a value means. The shared rule is to preserve the smallest amount of context that still lets a retrieved chunk stand on its own for the model and the user.

Sources: libs/text-splitters/langchain_text_splitters/character.py, libs/text-splitters/langchain_text_splitters/html.py, libs/text-splitters/langchain_text_splitters/json.py

Tune chunk_size and chunk_overlap with the target model and retriever in mind. The base class does not hard-code token accounting; it accepts a custom length_function, and the module imports optional tokenizer integrations such as tiktoken and Hugging Face transformer tokenizer types when available. That design lets applications measure chunk size by characters, tokens, or another domain-specific unit. Overlap should be large enough to preserve continuity across chunk boundaries, but not so large that the same information dominates the index and degrades retrieval diversity.

Sources: libs/text-splitters/langchain_text_splitters/base.py

Next Steps

For a first implementation, import RecursiveCharacterTextSplitter, split a representative document set, inspect chunk lengths and metadata, then embed and query a small index before scaling the job. If the input format is structured, switch to the format-specific splitter before changing retriever parameters; preserving HTML headings or JSON paths often improves retrieval more directly than simply changing chunk size. Related pages to read next are documents-and-loaders for getting raw content into Document form, embeddings-and-vector-stores for indexing chunks, retrievers for query-time retrieval, and indexing-api for incremental cleanup and re-indexing workflows.