Preprocessing Inputs

Purpose and Scope

Preprocessing in Transformers is the handoff between raw examples and model-ready batches. A dataset item may contain text token identifiers, labels, scores, paired responses, image-derived tensors, audio features, or other fields produced earlier by a tokenizer, image processor, feature extractor, or processor. The data collator is the final assembly point before the model sees those examples. It collects individual samples from a dataset and turns them into a rectangular batch, applying padding when necessary so variable-length values can be represented as tensors instead of uneven Python lists.

Sources: docs/source/en/data_collators.md

The most important design idea is dynamic padding. Instead of padding every example in a dataset to one global maximum length, the collator can pad only to the longest sequence in the current batch. This keeps batches compatible with tensor operations while avoiding avoidable padding tokens in shorter batches. The documented example shows three dataset rows with different input lengths becoming a single tensor where shorter rows receive padding up to the longest row in that batch, while labels are stacked into a separate tensor.

Sources: docs/source/en/data_collators.md

This page focuses on the batching boundary rather than the earlier conversion step. Tokenizers, image processors, feature extractors, and processors prepare values that are meaningful for a model family; the collator decides how those values are grouped for training or evaluation. That distinction matters when debugging. If token identifiers are wrong, inspect the tokenizer or processor. If examples are correct individually but fail when batched, inspect the collator, padding strategy, custom fields, and whether every tensor-like field has a consistent shape after collation.

Sources: docs/source/en/data_collators.md

Relevant Source Files

• docs/source/en/data_collators.md - Documents the data collator concept, dynamic batch padding, the standard tokenizer-backed extension point, and the lower-level mixin for custom batch assembly.

Core Primitives

A data collator is a callable batching component. It receives a list of dataset samples and returns a batch object suitable for the model or training loop. The documented default mental model is simple: each sample is a dictionary, and the output is another dictionary whose values have been padded, stacked, or otherwise transformed into batched tensors. This is why collators are central to preprocessing even when tokenization has already happened; they define the exact shape and field layout of every minibatch.

Sources: docs/source/en/data_collators.md

Use DataCollatorWithPadding when the tokenizer already knows how to pad the main model inputs and you only need a small amount of extra behavior. The documented customization pattern removes fields that the tokenizer padding method does not understand, lets the parent collator build input_ids and attention_mask, and then adds the custom tensor back to the batch. This avoids reimplementing standard tokenizer padding while still allowing task-specific metadata such as scores, regression values, or other per-example annotations.

Sources: docs/source/en/data_collators.md

Use DataCollatorMixin when the batch structure is not a straightforward tokenizer output. The documentation points to preference training as the motivating example: each sample contains a chosen response and a rejected response, and those paired values must be separated, concatenated, masked, and padded in a shape the model can consume. In that kind of workflow, the collator is not merely adding padding; it is defining how multiple logical inputs inside a single training example become one model-facing batch.

Sources: docs/source/en/data_collators.md

System-to-Code Mapping

The source page intentionally presents the collator as a small but powerful extension point. The simple subclass keeps the tokenizer in charge of ordinary fields, which is safer for models that expect tokenizer-created attention masks or special padding behavior. The mixin path gives up that convenience in exchange for full control. A good rule is to start with the tokenizer-backed collator if your samples look like ordinary model inputs plus a few scalar annotations, then move to the mixin only when the batch shape cannot be produced by tokenizer padding alone.

Sources: docs/source/en/data_collators.md

Execution Flow

A typical training flow begins with individual dataset records that have already been preprocessed into model-oriented fields. When the training loop requests a minibatch, it passes a list of those records to the configured collator. The collator inspects the list, removes or reorganizes fields if needed, computes the batch maximum length or task-specific structure, pads shorter sequences, builds attention masks when the custom logic requires them, and returns tensors keyed by the names expected by the model. The Trainer integration is explicit: pass the custom collator through the data_collator argument.

Sources: docs/source/en/data_collators.md

trainer = Trainer(
    ...,
    data_collator=DataCollatorWithScore(tokenizer=tokenizer),
)

The main edge case is unsupported fields. Tokenizer padding methods know about model input fields, but they do not automatically know how to pad or stack arbitrary values. The documented score example handles this by popping the score values before calling the parent implementation, then restoring them as a floating point tensor. This pattern prevents accidental failures from sending unknown keys into tokenizer padding, and it also makes dtype choices explicit. The same principle applies to labels, preference scores, auxiliary masks, or any field whose shape differs from ordinary token sequences.

Sources: docs/source/en/data_collators.md

API Components and Customization

DataCollatorWithPadding is the standard extension point for tokenizer-based batches. Its public customization surface is the callable behavior: subclass it and override __call__ to transform the list of features before and after parent collation. The documented example uses a dataclass with a tokenizer attribute, extracts a per-example score, calls super().__call__(features), and then writes a new tensor into the returned batch. The important contract is that the returned dictionary must match what the downstream model or trainer step expects.

Sources: docs/source/en/data_collators.md

import torch
from dataclasses import dataclass
from transformers import DataCollatorWithPadding, PreTrainedTokenizerBase
 
@dataclass
class DataCollatorWithScore(DataCollatorWithPadding):
    tokenizer: PreTrainedTokenizerBase
 
    def __call__(self, features):
        scores = [f.pop('score') for f in features]
        batch = super().__call__(features)
        batch['score'] = torch.tensor(scores, dtype=torch.float)
        return batch

DataCollatorMixin is the lower-level option for full-control preprocessing. The documented preference-style flow separates chosen and rejected token lists, concatenates them into one list, generates attention masks from the raw token ids, and pads both inputs and masks. This is useful when each dataset item contains multiple model inputs, when a task uses paired alternatives, or when padding must be coordinated across fields that the tokenizer cannot see as a single ordinary example. The tradeoff is responsibility: the custom collator must create every required field correctly.

Sources: docs/source/en/data_collators.md

Practical Next Steps

When adding a new preprocessing path, first inspect one raw dataset item and one tokenized or processor-produced item before batching. Confirm that every field has a clear owner: tokenizers and processors should create model-specific per-example values, while the collator should create batch-level tensor structure. Then choose the smallest collator abstraction that fits. Prefer the tokenizer-backed collator for normal text-like batches with extra scalar fields, and choose the mixin when examples contain paired inputs, multiple alternatives, or custom mask logic. Afterward, run a single batch through the model before launching a full training job.

Sources: docs/source/en/data_collators.md

Related pages to read next: Tokenizers for text conversion, Processors for multimodal preparation, Trainer for where the collator is passed into training, Fine-tuning for the end-to-end dataset-to-model workflow, and Tokenizer and Processor API for the lower-level classes that create the per-example fields consumed by collators.