Tokenizers

Purpose and Scope

Tokenizers are the text entry point for Transformer models. A tokenizer turns user text into model-ready tensors by normalizing text, splitting it, applying a tokenization algorithm, inserting special tokens, and later decoding generated token identifiers back into readable text. In this repository, the tokenizers guide is not only an API introduction; it is the reader-facing bridge between model checkpoints and the lower-level preprocessing decisions that made those checkpoints trainable. The English guide frames the common path around loading, encoding, decoding, batch processing, and backend choice, while the translated fast-tokenizer pages preserve the integration story for users of the Rust-backed Tokenizers library.

Sources: docs/source/en/fast_tokenizers.md, docs/source/ar/fast_tokenizers.md, docs/source/es/fast_tokenizers.md, docs/source/ja/fast_tokenizers.md, docs/source/ko/fast_tokenizers.md, docs/source/pt/fast_tokenizers.md

The most important practical rule is that a pretrained model should receive text processed with the same tokenizer conventions used during training. That includes the model vocabulary, merge rules or segmentation algorithm, normalizer, pre-tokenizer, and the special-token scheme. The docs therefore recommend loading by checkpoint when possible, because this lets the library resolve the correct tokenizer class and configuration automatically. When a reader needs to train a tokenizer for a new corpus or use a tokenizer produced by the standalone Tokenizers library, the fast-tokenizer integration path provides direct construction from an in-memory tokenizer object or from a serialized JSON file.

Relevant Source Files

• docs/source/en/fast_tokenizers.md - Main English documentation page for the current tokenizers guide, including the definition of tokenizer responsibilities, the recommended AutoTokenizer loading path, model-specific tokenizer initialization, and encode/decode API examples.
• docs/source/ar/fast_tokenizers.md - Arabic localized page documenting the PreTrainedTokenizerFast integration with Tokenizers, including BPE training, special tokens, tokenizer_object, and tokenizer_file usage.
• docs/source/es/fast_tokenizers.md - Spanish localized page for the same fast-tokenizer integration flow, showing how a trained Tokenizers object is reused directly or saved as tokenizer.json.
• docs/source/ja/fast_tokenizers.md - Japanese localized page that mirrors the fast-tokenizer workflow and reinforces the JSON serialization path for later reuse.
• docs/source/ko/fast_tokenizers.md - Korean localized page documenting the same API surface, including tokenizer_object construction and tokenizer_file loading.
• docs/source/pt/fast_tokenizers.md - Portuguese localized page showing PreTrainedTokenizerFast construction from Tokenizers artifacts and linking that object back to shared Transformers tokenizer methods.

Core Primitives

The highest-level primitive is AutoTokenizer, which is designed for normal inference and fine-tuning workflows. Its from_pretrained method reads the checkpoint metadata, resolves the correct tokenizer class, and returns a configured tokenizer without requiring the user to know whether the checkpoint expects Gemma, BERT, or another model-specific tokenizer family. This is the safest entry point for existing Hub checkpoints because tokenizer details are part of the checkpoint contract. The English guide explicitly presents AutoTokenizer as the recommended approach and shows it returning input_ids and an attention_mask suitable for PyTorch when return_tensors is requested.

The model-specific tokenizer class is the second primitive. It matters when a reader is creating an empty tokenizer for training, or when they need model-specific constructor arguments such as vocabulary and merge files. The English page uses GemmaTokenizer as the example: an empty tokenizer can be created, a corpus can be provided, and a new tokenizer can be trained from an iterator. This is not the default path for using a pretrained model; it is the customization path for adapting tokenization to a new domain while retaining the model family’s pipeline conventions, such as its expected normalizer, pre-tokenizer, and special-token rules.

The fast-tokenizer primitive is PreTrainedTokenizerFast. The localized fast-tokenizer pages emphasize that it depends on the standalone Tokenizers library and can wrap a tokenizer created there. Those pages demonstrate a small Byte Pair Encoding tokenizer built with Tokenizer, BPE, BpeTrainer, and Whitespace, with special tokens such as [UNK], [CLS], [SEP], [PAD], and [MASK]. After training on files, the tokenizer can be passed into Transformers directly. This path is useful when the tokenizer training pipeline already lives in Tokenizers, but the final object still needs the shared Transformers tokenizer methods.

Loading and Customization Flow

For a pretrained checkpoint, start with AutoTokenizer.from_pretrained and the checkpoint identifier. This loading path ties the tokenizer to the model configuration and prevents a common class of errors where the model receives identifiers produced by a mismatched tokenizer. The docs example loads google/gemma-2-2b and immediately calls the tokenizer on a sentence. The result includes token identifiers and an attention mask, which are the common model inputs expected by Transformer forward passes. This pattern is also the easiest way to keep padding, truncation side, special-token insertion, and maximum-length behavior aligned with checkpoint metadata.

from transformers import AutoTokenizer
 
tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b")
encoded = tokenizer("Sphinx of black quartz, judge my vow.", return_tensors="pt")

When a tokenizer is being trained or adapted, the flow changes. The fast-tokenizer docs first create a Tokenizers object, choose a BPE model with an unknown token, configure a trainer with special tokens, attach whitespace pre-tokenization, and train from files. That Tokenizers object can be used immediately in the same runtime or saved for later reuse. In Transformers, the in-memory path is PreTrainedTokenizerFast(tokenizer_object=tokenizer). The persisted path is tokenizer.save("tokenizer.json") followed by PreTrainedTokenizerFast(tokenizer_file="tokenizer.json"). Both paths produce a Transformers tokenizer object that participates in the common tokenizer API.

from transformers import PreTrainedTokenizerFast
 
fast_tokenizer = PreTrainedTokenizerFast(tokenizer_object=tokenizer)
 
tokenizer.save("tokenizer.json")
fast_tokenizer = PreTrainedTokenizerFast(tokenizer_file="tokenizer.json")

Special tokens are part of customization, not decoration. The examples register unknown, classification, separator, padding, and mask tokens during BPE training, and the English guide describes empty model-specific tokenizers as containing minimal special tokens such as pad, end-of-sequence, or beginning-of-sequence markers. These tokens let models distinguish structural boundaries from ordinary text. If a tokenizer is retrained for finance, code, a low-resource language, or another domain, the vocabulary may change, but the model-facing contract still needs explicit treatment of padding, sequence boundaries, and any task-specific markers used by downstream training or inference.

Encoding, Decoding, and Batching APIs

The central runtime API is the tokenizer call itself. In the English documentation this is described as the method that encodes a single text or a batch of text into model inputs such as identifiers and attention masks, while also controlling padding, truncation, and special-token insertion. It is the preferred interface when preparing data for a model because it returns the structured mapping expected by model calls. The lower-level encode method is similar but returns only the token identifiers. That narrower return value is useful for inspecting segmentation or testing a tokenizer, but it omits auxiliary tensors that many models need.

Decoding is the inverse reader task: take generated or predicted token identifiers and convert them back into text. The page definition explicitly includes decoding output identifiers, which matters for language generation, summarization, translation, and any workflow where model outputs are token IDs rather than labels. Decoding is only reliable when performed by the tokenizer that owns the vocabulary and special-token conventions used for encoding. In practice, the same tokenizer loaded from the checkpoint should handle both directions, because token identifiers are not globally meaningful across different vocabularies, algorithms, or model families.

Batch processing follows the same principle as single-example encoding but adds shape management. A batch may contain sentences of different lengths, so the tokenizer call is responsible for applying padding and truncation consistently with model expectations. The attention mask returned in the docs example shows why this matters: models need to know which positions are real tokens and which positions are padding. For training scripts and inference services, using the tokenizer’s batch-aware call avoids hand-written tensor assembly and keeps preprocessing behavior centralized in the tokenizer configuration rather than scattered through application code.

System-to-Code Mapping

The repository mapping is documentation-oriented for this page: the English file is the canonical modern guide, while the Arabic, Spanish, Japanese, Korean, and Portuguese files show the same fast-tokenizer bridge in localized form. The repeated examples across languages are valuable because they reveal the stable API contract: a Tokenizers tokenizer can be trained independently, special tokens are registered during training, the artifact can be serialized as JSON, and PreTrainedTokenizerFast makes the result usable through Transformers tokenizer methods. That shared contract is what downstream model, trainer, and pipeline code relies on rather than the prose language of the documentation page.

Implementation Details and Edge Cases

Choose the loading path based on ownership of the tokenizer artifact. If the tokenizer belongs to an existing model checkpoint, prefer the automatic checkpoint path. If the tokenizer is newly trained in the Tokenizers library, wrap the object directly during experimentation and save a JSON artifact when the tokenizer must be reused, shared, or versioned. If the tokenizer is model-specific but needs a new vocabulary, instantiate the model-specific tokenizer and train from an iterator. These choices avoid mixing incompatible artifacts, such as a model trained with one segmentation scheme and inference text encoded with another.

A second edge case is vocabulary size and algorithm selection. The fast-tokenizer examples use BPE, while the broader tokenizer documentation surface identifies BPE, WordPiece, and SentencePiece as major algorithms used by Transformers models. The key operational point is that the algorithm and learned vocabulary are not interchangeable after pretraining. A tokenizer may split punctuation, whitespace, rare words, or subwords differently, and those differences change the token IDs seen by the model. When adapting tokenization, validate the full pipeline on representative text before fine-tuning or serving, not just the existence of a vocabulary file.

Next Steps

After this page, read the preprocessing page for padding, truncation, and batching in the broader data pipeline, then read the tokenizer and processor API reference for detailed method options. If your workflow is multimodal, continue to processors because text tokenization may be combined with image, audio, or video preprocessing. If you are training or adapting a domain tokenizer, pair this guide with the fine-tuning and training-script pages so that tokenizer saving, dataset preprocessing, model resizing, evaluation, and upload all happen as one reproducible workflow.