Image, Video, and Feature Processors

Purpose and Scope

Image, video, and feature processors are the input preparation layer that sits between raw modality data and model tensors. In Transformers documentation, the broader word processor has a precise meaning: for multimodal models, it is an object that groups several processing components, such as a tokenizer for text, an image processor for vision, and a feature extractor for audio. That grouping matters because models such as CLIP or Wav2Vec2 do not consume only one kind of input; they need coordinated preprocessing and decoding across modalities before the model call is valid.

Sources: docs/source/en/main_classes/processors.md

For vision work, an image processor converts images into pixel values that represent color and size in the tensor format expected by a pretrained vision model. The official image processor docs describe common operations such as resizing, center cropping, rescaling, and normalization, all of which align incoming images with the data distribution used during pretraining. A video processor extends that role to temporal inputs, preparing batches of videos for models by applying frame and batch transformations while preserving the video-specific calling convention. Feature extractors complete the picture for audio or other non-text signals inside multimodal processors.

Sources: docs/source/en/main_classes/processors.md

This page is a conceptual and implementation map rather than a model-specific recipe. It explains how the main processor documentation defines the shared contract, how localized documentation repeats that contract for international readers, how backbone documentation connects processed pixels to feature extraction models, and how training callbacks are deliberately separate from preprocessing. Use it when deciding whether to reach for a standalone image or video processor, a combined multimodal processor, a backbone, or a training customization hook.

Sources: docs/source/en/main_classes/processors.md, docs/source/en/main_classes/backbones.md, docs/source/en/main_classes/callback.md

Relevant Source Files

• docs/source/en/main_classes/processors.md - Defines the English processor reference, including the distinction between multimodal processors and deprecated dataset processors, the ProcessorMixin autodoc surface, and modality-specific processing keyword groups.
• docs/source/ja/main_classes/processors.md - Japanese localized processor documentation that mirrors the multimodal processor definition and the deprecated processor section for translated docs users.
• docs/source/ko/main_classes/processors.md - Korean localized processor documentation that names the same multimodal grouping of tokenizers, image processors, and feature extractors.
• docs/source/zh/main_classes/processors.md - Chinese localized processor documentation that preserves the same architecture and deprecated GLUE or SQuAD processor distinction.
• docs/source/en/main_classes/backbones.md - Documents backbones as feature extraction models for higher-level computer vision tasks and lists AutoBackbone, BackboneMixin, BackboneConfigMixin, TimmBackbone, and TimmBackboneConfig.
• docs/source/en/main_classes/callback.md - Documents Trainer callbacks, which are relevant as a boundary: callbacks customize the training loop, while processors prepare data before model execution.

Core Primitives

The central primitive is ProcessorMixin, the base class named by the processor reference as the place where saving and loading functionality is implemented for multimodal processors. The English reference exposes autodoc entries for methods such as call behavior, input layout preparation, input validation, text replacement utilities, multimodal token type identifiers, and chat template application. The same page also documents ProcessingKwargs plus TextKwargs, ImagesKwargs, VideosKwargs, and AudioKwargs, which define how processor calls organize options by modality. This keeps combined processors extensible without making every model accept an unstructured bag of arguments.

Sources: docs/source/en/main_classes/processors.md

Image processors are the vision-side component inside that architecture. A typical image processor is loaded from a pretrained model configuration and then called on one or more images to produce pixel values, commonly with a framework tensor return option. The important design point is not merely conversion to an array; the processor encodes model-specific assumptions about input size, normalization, rescaling, and cropping. If those assumptions are skipped or reproduced incorrectly, a checkpoint may receive tensors that are numerically valid but semantically mismatched to the way it was trained.

Sources: docs/source/en/main_classes/processors.md

Video processors use the same idea for moving images but need video-aware batching and configuration. The official video processor docs describe a configuration file dedicated to video preprocessing, while noting that older models may still store the configuration in a general preprocessor file. They also highlight a performance distinction: treating each frame as an independent image is functional but less efficient than fast video processors that operate over whole batches of videos. In practice, use AutoVideoProcessor for video checkpoints when available, and reserve frame-by-frame image processing for compatibility paths or small experiments.

Sources: docs/source/en/main_classes/processors.md

System-to-Code Mapping

The processor docs deliberately separate modern multimodal processors from older dataset processors. Modern processors encode or decode grouped modalities for models; deprecated processors follow the DataProcessor architecture that returns InputExample objects which can be converted into InputFeatures for GLUE or SQuAD style tasks. This naming overlap is a common source of confusion for new contributors. When working on images, videos, or audio, read the multimodal processor section first. When maintaining older benchmark conversion code, the DataProcessor, InputExample, and InputFeatures sections are the relevant parts of the same page.

Sources: docs/source/en/main_classes/processors.md

The localized processor pages are not separate implementations, but they are important documentation assets. The Japanese, Korean, and Chinese pages preserve the same first-order meaning: processors can be multimodal input preparation objects, or deprecated objects from older data preprocessing workflows. They also repeat the examples of multimodal models that combine speech and text or text and vision. For OpenWiki readers, this means terminology should stay consistent across pages: a processor is a composition point, while an image processor or feature extractor is one of the components that may be composed.

Sources: docs/source/ja/main_classes/processors.md, docs/source/ko/main_classes/processors.md, docs/source/zh/main_classes/processors.md

Backbones connect this preprocessing layer to vision feature extraction. The backbone reference defines a backbone as a model used for feature extraction for higher-level computer vision tasks, including object detection and image classification. It documents AutoBackbone for initializing a Transformers backbone from pretrained weights, BackboneMixin for returning output features and indices, BackboneConfigMixin for configuring those outputs, and TimmBackbone classes for timm integrations. A typical flow is therefore raw image, image processor, pixel values, backbone, feature maps, and then a task head or downstream system.

Sources: docs/source/en/main_classes/backbones.md

Callbacks define the opposite side of the boundary. The callback reference describes callbacks as objects that customize the PyTorch Trainer loop by inspecting state for logging, progress reporting, early stopping, or integrations. It also states that callbacks are read only apart from the TrainerControl they return, and that deeper training-loop changes should subclass Trainer. That matters for processors because preprocessing choices should be encoded in the dataset transform, data collator, processor, or model input pipeline, not hidden inside a callback that runs after batches are already formed.

Sources: docs/source/en/main_classes/callback.md

Execution Flow

A standard image inference flow begins by selecting a checkpoint, loading the matching processor configuration, loading an image, and calling the processor to create model inputs. The official docs show AutoImageProcessor loading a ViT checkpoint and returning PyTorch tensors from a PIL image. The processor configuration is model-specific, so the safest practice is to load it from the same checkpoint or local directory as the model. That keeps pixel value ranges, image size, channel order, and normalization consistent with the pretrained weights used by the model.

Sources: docs/source/en/main_classes/processors.md

from PIL import Image
from transformers import AutoImageProcessor, AutoModel
image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224")
model = AutoModel.from_pretrained("google/vit-base-patch16-224")
image = Image.open("example.png").convert("RGB")
inputs = image_processor(image, return_tensors="pt")
outputs = model(**inputs)

A multimodal flow adds coordination. For a vision-text model, the combined processor may wrap a tokenizer and an image processor so text and images can be prepared together. The contrastive image-text example evidence shows this pattern with VisionTextDualEncoderProcessor constructed from AutoImageProcessor and AutoTokenizer, then saved beside the model. The general processor documentation explains why this exists: multimodal models need an object that encodes or decodes grouped data across text, vision, and audio. Saving the combined processor with the model makes later inference less error-prone.

Sources: docs/source/en/main_classes/processors.md

Video inference follows the same contract but changes the shape and performance concerns. Use a video processor for video checkpoints so the processor can apply video-specific arguments and efficient batch handling. If a model or compatibility path treats frames independently, verify that the resulting tensor layout still matches the model documentation. The processor reference’s ImagesKwargs and VideosKwargs split is useful here: it signals that images and videos share some transformations but should not be collapsed into the same API surface when model authors need separate behavior.

Sources: docs/source/en/main_classes/processors.md

API Components and Options

The compact reference for this area starts with ProcessorMixin and the modality keyword groups. ProcessorMixin provides the shared save, load, call, validation, layout, multimodal token type, and chat-template surface shown in the English processor page. ProcessingKwargs is the umbrella, while TextKwargs, ImagesKwargs, VideosKwargs, and AudioKwargs organize call-time options by input type. Model-specific processors may subclass those typed dictionaries to add or override fields, which gives model authors a controlled extension point without changing every processor in the library.

Sources: docs/source/en/main_classes/processors.md

For feature extraction systems, pair processors with backbone APIs rather than task-specific heads when you need reusable visual representations. AutoBackbone initializes a backbone from pretrained weights; BackboneMixin exposes functions for output features and indices; BackboneConfigMixin configures those outputs; and TimmBackbone or TimmBackboneConfig cover timm-backed models. The backbone page lists supported families such as BEiT, ConvNext, DINOV2, ResNet, Swin, and ViTDet. Processors prepare correctly shaped and normalized inputs; backbones decide which intermediate features become available to downstream computer vision tasks.

Sources: docs/source/en/main_classes/backbones.md

Practical Guidance and Next Steps

When debugging poor vision or video results, first check that the processor and model come from the same checkpoint or from an intentionally compatible pair. Next, inspect whether you are using an image processor for still images, a video processor for video batches, or a combined processor for multimodal inputs. Then confirm the tensor framework and batch dimensions you request at call time. If the problem appears during training, separate data preparation from loop control: fix the processor, dataset transform, or collator before adding Trainer callbacks for logging or early stopping.

Sources: docs/source/en/main_classes/processors.md, docs/source/en/main_classes/callback.md

Read the Tokenizer and Processor API page next for shared public interfaces, the Processors page for multimodal composition patterns, the Vision Tasks page for downstream uses of processed pixel values, and the Model Reference Catalog for model-family-specific processor behavior. If your goal is feature extraction rather than classification or detection, continue with backbone APIs and image feature extraction examples. If your goal is serving video-language models, prioritize AutoVideoProcessor-compatible checkpoints and verify whether the checkpoint stores modern video preprocessing configuration or an older general preprocessing configuration.

Sources: docs/source/en/main_classes/processors.md, docs/source/en/main_classes/backbones.md