Tokenizer and Processor API

Purpose and Scope

Transformers separates model computation from input preparation. The model classes consume tensors, masks, token ids, pixel values, audio features, and other structured inputs; tokenizer and processor APIs are the public layer that turns user data into those model-ready structures. This page is a reference-oriented map of the shared API patterns across text tokenizers, multimodal processors, image processors, video processors, and audio feature extractors. It is intended for developers choosing the right loading class, implementing preprocessing in an application, or adding model-specific processing behavior while staying compatible with the documented Transformers surface.

Sources: docs/source/en/main_classes/tokenizer.md, docs/source/en/main_classes/processors.md, docs/source/en/main_classes/image_processor.md, docs/source/en/main_classes/video_processor.md, docs/source/en/main_classes/feature_extractor.md

A useful mental model is that each preprocessing class owns a modality boundary. Tokenizers own text-to-token conversion and special token management. Image processors own image loading, resizing, normalization, tensor conversion, and model-specific vision post-processing. Video processors extend the image-processing idea to decoded clips and sampled frames. Feature extractors prepare audio or sequence features, including padding. Multimodal processors group two or more of these objects so a model such as a speech-text or vision-text model can expose one callable input interface instead of making users coordinate each modality manually.

The same repository documentation also preserves older terminology. In current multimodal usage, a processor is a grouping object built around tokenizers, image processors, feature extractors, and related helpers. In older dataset workflows, deprecated processors follow the data processor architecture that returns examples and features for GLUE, SQuAD, and similar tasks. When reading existing code, the distinction matters: a ProcessorMixin-based object is about model input preparation, while a deprecated DataProcessor is about benchmark dataset conversion rather than runtime multimodal preprocessing.

Relevant Source Files

• docs/source/en/main_classes/tokenizer.md — defines the tokenizer documentation surface, including PreTrainedTokenizer, PreTrainedTokenizerFast, PreTrainedTokenizerBase, BatchEncoding, common text methods, and multimodal special token behavior.
• docs/source/en/main_classes/processors.md — defines processor terminology, ProcessorMixin, multimodal processor methods, modality-specific processing keyword dictionaries, and the deprecated dataset processor classes.
• docs/source/en/main_classes/image_processor.md — defines image processor responsibilities, backend architecture, BaseImageProcessor, TorchvisionBackend, PilBackend, AutoImageProcessor.from_pretrained, backend, and device behavior.
• docs/source/en/main_classes/video_processor.md — defines video processor responsibilities, AutoVideoProcessor, video decoding requirements, fast video processing behavior, configuration file names, sampling behavior, and device usage.
• docs/source/en/main_classes/feature_extractor.md — defines the feature extractor surface for audio models, including FeatureExtractionMixin, SequenceFeatureExtractor, BatchFeature, and ImageFeatureExtractionMixin.

Core Primitives

A tokenizer prepares text for a model by splitting strings into token strings, converting those tokens to ids, creating model inputs such as input_ids and attention_mask, and decoding ids back to text. Transformers documents two implementation families: PreTrainedTokenizer, a full Python implementation, and PreTrainedTokenizerFast, backed by the Rust tokenizers library. Both rely on PreTrainedTokenizerBase for common behavior such as encoding, saving, loading, and shared call patterns. The fast implementation is especially important for batched tokenization and for alignment methods that map between original characters or words and token positions.

Sources: docs/source/en/main_classes/tokenizer.md

Tokenizer outputs are represented by BatchEncoding, a dictionary-like container returned by __call__, encode_plus, and batch_encode_plus. For Python tokenizers, it behaves like a standard dictionary containing computed model inputs. For fast tokenizers, it additionally exposes alignment features, such as locating the token that contains a character or retrieving the character span represented by a token. This makes BatchEncoding both a transport format for model tensors and an inspection object for tasks like token classification, span extraction, highlighting, and debugging input truncation.

Tokenizers also manage vocabulary extension and special tokens in a tokenizer-independent way. The docs describe adding new tokens regardless of the underlying algorithm, such as BPE or SentencePiece, and managing special tokens such as mask or beginning-of-sentence tokens so they are available as attributes and are not split during tokenization. Multimodal tokenizers extend that idea by exposing model-specific special tokens as convenient attributes. For example, a tokenizer loaded from a vision-language checkpoint may expose an image_token_id placeholder used when text prompts reference image inputs.

Processors are the composition primitive for models that span more than one modality. A multimodal processor groups processing objects such as a text tokenizer, an image processor for vision, and a feature extractor for audio. The documented ProcessorMixin surface includes callable preprocessing plus helper methods such as prepare_inputs_layout, validate_inputs, get_text_with_replacements, create_mm_token_type_ids, and apply_chat_template. That set of names signals that processors are not only a container: they also validate modality-specific inputs, prepare multimodal layouts, and bridge chat-style prompt formatting with model input construction.

Sources: docs/source/en/main_classes/processors.md

Modality APIs

Image processors prepare input features for vision models and may post-process outputs. The documented responsibilities include optional image loading, resizing, normalization, conversion to PyTorch and NumPy tensors, and model-specific post-processing such as converting logits into segmentation masks. The image processor class hierarchy is backend-based. BaseImageProcessor is retained as an abstract base class for backward compatibility and is not the class users should instantiate directly. Model-facing image processor classes inherit from either the default TorchvisionBackend or the older PilBackend, while exposing the same user API.

Sources: docs/source/en/main_classes/image_processor.md

The backend choice is an important runtime decision. TorchvisionBackend is documented as the default when torchvision is available, GPU-accelerated, and significantly faster than PIL. PilBackend is the portable CPU-only alternative, primarily retained for older models and parity with original implementations. Users can inspect processor.backend, and can pass backend to AutoImageProcessor.from_pretrained. If the argument is omitted, Transformers chooses torchvision when installed and otherwise falls back to PIL, with documented interpolation exceptions for some models when torchvision versions do not support required Lanczos behavior.

Video processors extend image preprocessing for video-capable vision-language models. A video processor prepares model input features, post-processes outputs, performs transformations such as resizing and normalization, and converts data into PyTorch. It also handles video decoding from local paths or URLs when torchcodec is available, then samples frames according to model-specific strategies. This distinction matters because treating every frame as an independent image works functionally but can be inefficient. The documented fast video processors use torchvision to process whole batches of videos rather than iterating over each video or frame.

Sources: docs/source/en/main_classes/video_processor.md

Video preprocessing also has its own persistence and sampling concerns. For new or updated vision-language models that enable distinct video preprocessing, saving and reloading stores video-related arguments in video_preprocessing_config.json. The processor still attempts to load video-related configuration from preprocessing_config.json for compatibility. Sampling behavior is controlled by do_sample_frames and model-specific parameters such as num_frames. Fast video processors are loaded by default through AutoVideoProcessor, can receive a device argument, and can be compiled with torch.compile for additional CUDA speed improvements.

Feature extractors cover the audio side of the preprocessing API. The docs define a feature extractor as responsible for preparing input features for audio models, including preprocessing audio sequences to generate features such as Log-Mel spectrograms and converting those features to NumPy or PyTorch tensors. The documented API exposes FeatureExtractionMixin with from_pretrained and save_pretrained, SequenceFeatureExtractor with pad, and BatchFeature as the feature container. This mirrors the tokenizer pattern: load reusable preprocessing configuration, call it on raw inputs, then pass the resulting batch container to a model.

Sources: docs/source/en/main_classes/feature_extractor.md

Loading and Calling Patterns

For application code, the usual pattern is to load the processor object from the same checkpoint family as the model, call it on raw data, and request framework tensors with return_tensors. Text-only code commonly uses AutoTokenizer; image workflows use AutoImageProcessor; video workflows use AutoVideoProcessor; multimodal model classes may provide a model-specific processor that bundles the underlying tokenizer, image processor, or feature extractor. This keeps saved preprocessing configuration aligned with the model checkpoint rather than requiring callers to manually reproduce resize sizes, normalization constants, vocabulary files, or special-token conventions.

from transformers import AutoImageProcessor, AutoVideoProcessor
 
image_processor = AutoImageProcessor.from_pretrained(
    "facebook/detr-resnet-50",
    backend="torchvision",
)
 
video_processor = AutoVideoProcessor.from_pretrained(
    "llava-hf/llava-onevision-qwen2-0.5b-ov-hf",
    device="cuda",
)

The callable pattern is intentionally similar across objects, but each modality has its own keyword space. Processor __call__ methods accept keyword arguments organized by modality, and the docs name ProcessingKwargs, TextKwargs, ImagesKwargs, VideosKwargs, and AudioKwargs as the TypedDict classes that define available fields. Model-specific processors may subclass those dictionaries to add or override fields. This organization helps keep a multimodal call explicit: text options should not be confused with image options, and video sampling options should not be hidden inside a generic untyped argument bag.

Sources: docs/source/en/main_classes/processors.md

Device placement is part of the processor API for fast visual backends. For image processors using the torchvision backend, device specifies where processing should run; by default, processing follows tensor inputs if they already live on a device or otherwise runs on CPU. Fast video processors expose similar behavior and can process batches on CUDA. This does not make preprocessing a replacement for model placement, but it does reduce avoidable host-device transfer and Python iteration overhead in high-throughput image or video inference workloads.

Compact API Reference

The reference table is intentionally organized by responsibility rather than file. In real code, the main decision is not which markdown page defined a name, but which preprocessing boundary your model requires. Choose a tokenizer when the model consumes text only; choose an image, video, or feature extractor API when the raw input is a single non-text modality; choose a processor when the checkpoint expects coordinated multimodal inputs or a single saved preprocessing package. For multimodal and chat-style models, prefer the checkpoint processor because it can encode placeholder tokens, modality layouts, and chat templates consistently with the model.

Sources: docs/source/en/main_classes/tokenizer.md, docs/source/en/main_classes/processors.md, docs/source/en/main_classes/image_processor.md, docs/source/en/main_classes/video_processor.md, docs/source/en/main_classes/feature_extractor.md

Implementation Guidance and Next Steps

When adding or integrating a model, keep preprocessing configuration as part of the checkpoint contract. Save tokenizers and feature extractors with their provided save_pretrained patterns, and use the appropriate auto class to reload them. For video-enabled vision-language models, prefer the dedicated video configuration file for video-specific arguments while preserving compatibility with older preprocessing configuration. For image models, test backend behavior explicitly if numerical parity or interpolation semantics matter, especially when forcing backend="torchvision" or relying on PIL-compatible behavior from older processors.

For runtime applications, start with the highest-level object that matches the checkpoint: a model-specific processor for multimodal models, AutoTokenizer for text, AutoImageProcessor for images, and AutoVideoProcessor for videos. Inspect returned containers such as BatchEncoding or BatchFeature before passing them to the model, because those containers reveal the actual fields the model will receive. If you need span alignment, prefer fast tokenizers. If throughput is constrained by visual preprocessing, prefer the fast torchvision-backed image or video path and consider explicit device placement.

Related pages: preprocessing, tokenizers, processors, image-video-feature-processors, auto-classes-and-model-loading, chat-templates.