Processors
Purpose and Scope
A processor is the Transformers object that gives a multimodal checkpoint one coherent preprocessing entry point. Instead of asking users to call a tokenizer for text, an image processor for pixels, and a feature extractor for audio in the correct order, a processor groups those components behind a shared interface. The English main-classes page defines this as the object required by multimodal models to encode or decode data that combines modalities such as text, vision, and audio, with Wav2Vec2 and CLIP used as representative examples. Sources: docs/source/en/main_classes/processors.md
The same page also warns that the word processor has a second, legacy meaning in Transformers. Older text benchmark utilities for GLUE, XNLI, and SQuAD are also called processors, but they follow the data-processing architecture around DataProcessor, InputExample, and InputFeatures rather than the multimodal ProcessorMixin path. When reading model documentation or writing application code, treat ProcessorMixin processors as the modern model-facing abstraction, and treat GLUE or SQuAD processors as deprecated dataset-conversion helpers retained for compatibility and reference. Sources: docs/source/en/main_classes/processors.md
Processors matter most when a model consumes more than one kind of input or produces outputs that need coordinated decoding. A text-only model can usually be served by AutoTokenizer, and a vision-only model can often use AutoImageProcessor, but multimodal checkpoints need a contract for merging fields into one model-ready batch. The official documentation describes a single call surface that can accept images, text, videos, and audio, route them to the right subprocessor, and return one dictionary of tensors and metadata for the model.
Relevant Source Files
- docs/source/en/main_classes/processors.md - Canonical English reference page for processors. It defines the two meanings of processors, documents multimodal processors through ProcessorMixin, lists the ProcessorMixin methods surfaced by autodoc, and retains deprecated GLUE, XNLI, and SQuAD processor references.
- docs/source/ja/main_classes/processors.md - Japanese localization of the processor reference. It preserves the distinction between multimodal processors and deprecated benchmark processors for non-English readers.
- docs/source/ko/main_classes/processors.md - Korean localization of the processor reference. It explicitly anchors the multimodal section to ProcessorMixin and the deprecated section to DataProcessor-style utilities.
- docs/source/zh/main_classes/processors.md - Chinese localization of the processor reference. It mirrors the multimodal, GLUE, XNLI, and SQuAD structure and includes example-oriented SQuAD processor material.
- docs/source/en/main_classes/backbones.md - Adjacent main-classes reference page that shows how Transformers documents composable vision infrastructure such as AutoBackbone, BackboneMixin, and BackboneConfigMixin.
- docs/source/en/main_classes/callback.md - Adjacent main-classes reference page that illustrates the same autodoc-based API documentation style for higher-level extension objects such as TrainerCallback, TrainerState, and TrainerControl.
Core Primitives
The central primitive is ProcessorMixin. The English processor page exposes it through autodoc and lists the public behaviors that define the processor contract: call, prepare_inputs_layout, validate_inputs, get_text_with_replacements, create_mm_token_type_ids, and apply_chat_template. Those names show that the abstraction is broader than a pass-through wrapper. It validates the incoming modality layout, prepares multimodal inputs, can replace text placeholders with modality-specific token patterns, can create multimodal token type ids, and can apply chat templates when a processor is used with chat-style multimodal models. Sources: docs/source/en/main_classes/processors.md
Processor inputs are organized by modality-specific keyword groups. The English page documents ProcessingKwargs as the base typed dictionary and then separates TextKwargs, ImagesKwargs, VideosKwargs, and AudioKwargs. This matters because model-specific processors can extend or override the fields they accept without losing the shared shape of the API. As a user, you should pass text options with the text pathway, image options with the image pathway, and so on, instead of assuming that every keyword is meaningful to every subcomponent. Sources: docs/source/en/main_classes/processors.md
The objects grouped by a processor are also important primitives. Tokenizers handle the text modality, image processors handle vision inputs, and feature extractors handle audio inputs. In newer multimodal documentation, video processors are part of the same family of modality processors. A processor is therefore best understood as a composition layer: it does not replace tokenization or pixel preprocessing, but it owns the model-specific coordination between them, including how outputs are merged and how placeholder tokens correspond to non-text inputs.
System-to-Code Mapping
The processors documentation is published as a main-classes reference page rather than a single model tutorial. Its English source starts with user-facing definitions, then hands the detailed API surface to doc-builder autodoc blocks. This is the same pattern used by neighboring reference pages such as backbones and callbacks: a short conceptual explanation names the abstraction, followed by autodoc entries for the concrete classes and methods. That structure keeps the page aligned with the source API while still giving readers enough conceptual framing to choose the right object. Sources: docs/source/en/main_classes/processors.md, docs/source/en/main_classes/backbones.md, docs/source/en/main_classes/callback.md
The localized processor pages confirm that this terminology is not just an English-only explanation. Japanese, Korean, and Chinese versions all preserve the distinction between multimodal processors and deprecated processors, and they repeat the same core components: tokenizers for text, image processors for vision, and feature extractors for audio. For documentation maintainers, that means changes to the processor concept should be made carefully because the term appears across translated pages and in older benchmark-oriented sections. Sources: docs/source/ja/main_classes/processors.md, docs/source/ko/main_classes/processors.md, docs/source/zh/main_classes/processors.md
Backbones and callbacks are not processor implementations, but they help explain how this documentation surface is organized. The backbone page defines a composable model component for computer vision feature extraction and exposes AutoBackbone plus mixins through autodoc. The callback page defines extension objects for Trainer and exposes TrainerCallback, TrainerState, and TrainerControl through autodoc. Processors occupy a similar reference layer: they are composable, public-facing infrastructure that model families use to make higher-level workflows predictable. Sources: docs/source/en/main_classes/backbones.md, docs/source/en/main_classes/callback.md
API Components
Use this compact reference when deciding which processor-related API to look for in model docs or generated reference pages. ProcessorMixin is the modern base class for multimodal processor classes and supplies saving and loading behavior alongside the documented call and layout methods. ProcessingKwargs is the common typed-dictionary base for keyword configuration. TextKwargs, ImagesKwargs, VideosKwargs, and AudioKwargs separate options by modality. DataProcessor, InputExample, and InputFeatures belong to the deprecated benchmark processor path rather than the multimodal model preprocessing path. Sources: docs/source/en/main_classes/processors.md
| Component | Role in the documented contract | When to reach for it |
|---|---|---|
| ProcessorMixin | Base mixin for multimodal processors with saving, loading, call, validation, layout, token replacement, token type id, and chat-template behaviors surfaced in docs | Loading or authoring a multimodal model processor |
| ProcessingKwargs | Shared typed dictionary for processor keyword arguments | Defining common processor option groups |
| TextKwargs | Text-specific keyword group | Passing tokenizer-oriented options through a processor |
| ImagesKwargs | Image-specific keyword group | Passing image processor options such as image preprocessing controls |
| VideosKwargs | Video-specific keyword group | Passing video preprocessing options through a multimodal processor |
| AudioKwargs | Audio-specific keyword group | Passing feature-extractor or audio preprocessing options |
| DataProcessor | Deprecated benchmark processor architecture | Maintaining older GLUE, XNLI, or SQuAD conversion code |
| InputExample / InputFeatures | Legacy intermediate representations for dataset examples and model features | Reading or adapting old benchmark scripts |
Execution Flow
A typical multimodal inference flow starts by loading the checkpoint’s processor, then passing raw inputs to the processor rather than manually invoking each subcomponent. The processor receives fields such as text plus images, videos, or audio, validates that the combination matches the model’s expectations, and delegates each modality to the correct underlying object. The result is a single mapping that can be supplied to the model. For chat-like multimodal models, the processor may also apply a chat template and expand modality placeholders such as image, video, or audio markers into the sequence pattern expected by the model.
This coordination is especially important when non-text inputs are interleaved with natural language. If a prompt contains placeholder tokens for images or videos, the model may require a specific number or arrangement of replacement tokens. The processor layer is where that model-specific relationship belongs, because the image or video processor knows the resulting media layout while the tokenizer owns the textual sequence. Keeping that logic in the processor avoids scattering fragile placeholder replacement code across applications.
Deprecated benchmark processors follow a different flow. They load or parse dataset records, produce InputExample objects, and then convert those examples into InputFeatures suitable for a model. The English and localized pages list GLUE processors for MRPC, MNLI, MNLI mismatched, CoLA, SST2, STSB, QQP, QNLI, RTE, and WNLI, and they also describe XNLI and SQuAD processor families. That path is useful for understanding older examples, but it should not be confused with the processor object loaded for a multimodal checkpoint. Sources: docs/source/en/main_classes/processors.md, docs/source/ja/main_classes/processors.md, docs/source/ko/main_classes/processors.md, docs/source/zh/main_classes/processors.md
Implementation and Authoring Guidance
When authoring or reviewing a new multimodal processor, start from the documented public contract instead of inventing a separate preprocessing API. The processor should group the exact modality components the model needs, expose a single call entry point, and keep modality-specific options in the appropriate kwargs groups. If the model uses placeholder repetition or modality token expansion, implement that behavior at the processor layer so callers can provide natural prompts and raw media without reverse-engineering the model’s internal sequence format.
The processor docs also imply a compatibility responsibility. Because processor classes own saving and loading behavior through ProcessorMixin, their configuration and component names become part of the checkpoint experience. A processor that works locally but cannot be saved, loaded, or reconstructed from a pretrained model directory breaks the main value of the abstraction. Likewise, if a model-specific processor accepts custom kwargs, those options should fit the ProcessingKwargs pattern so generated docs and user code can reason about text, image, video, and audio options consistently. Sources: docs/source/en/main_classes/processors.md
For users, the practical rule is simple: prefer the checkpoint’s processor whenever a model card or model documentation advertises one. Use tokenizer, image processor, video processor, or feature extractor classes directly when working with a single modality or when you deliberately need low-level control. Move down to those lower-level APIs only after confirming which component owns the preprocessing step you want to customize. For next steps, read Tokenizers for text-only preprocessing, Image, Video, and Feature Processors for modality-specific tensor preparation, and Chat Templates when your processor formats conversational multimodal prompts.