Trainer

Purpose and Scope

Trainer is the high-level PyTorch training abstraction in Transformers. It gives you a complete training and evaluation loop around a model, datasets, and TrainingArguments, so you can focus on task data, model selection, metrics, and customization points instead of rewriting batching, optimizer setup, distributed execution, checkpointing, and prediction plumbing for every experiment. The English, Korean, Japanese, and Chinese Trainer docs all frame the class as a feature-complete training API, paired with TrainingArguments, and adapted through Seq2SeqTrainer and Seq2SeqTrainingArguments for sequence-to-sequence tasks such as summarization and translation.

Sources: docs/source/en/main_classes/trainer.md, docs/source/ko/main_classes/trainer.md, docs/source/ja/main_classes/trainer.md, docs/source/zh/main_classes/trainer.md

Use Trainer when your model follows the Transformers model contract and you want a standard training lifecycle: create arguments, pass datasets, run train(), evaluate, predict, save checkpoints, and optionally push artifacts to the Hub. The documented warning matters: Trainer is optimized for Transformers models and may be surprising with arbitrary PyTorch modules. A compatible custom model should return tuples or ModelOutput subclasses, compute loss when labels is supplied, return that loss first when returning tuples, and use TrainingArguments.label_names for multiple label arguments while avoiding a label argument literally named label.

Sources: docs/source/en/main_classes/trainer.md, docs/source/ja/main_classes/trainer.md, docs/source/ko/main_classes/trainer.md, docs/source/zh/main_classes/trainer.md

Relevant Source Files

• docs/source/en/main_classes/trainer.md — canonical English API page for Trainer, Seq2SeqTrainer, TrainingArguments, and Seq2SeqTrainingArguments, including the compatibility warning for non-Transformers models.
• docs/source/ja/main_classes/trainer.md — localized Trainer page that includes an expanded list of subclassable training-loop methods, an example compute_loss override, and checkpoint behavior.
• docs/source/ko/main_classes/trainer.md — localized Trainer page that reinforces the same core API contract, distributed and mixed precision support, and seq2seq inheritance model.
• docs/source/zh/main_classes/trainer.md — localized Trainer page that documents the customization method list, a weighted-loss subclass example, and checkpoint and Hub resume options.
• docs/source/en/main_classes/backbones.md — defines backbones, AutoBackbone, and backbone utility classes that are relevant when a Trainer workload uses feature-extraction models for vision tasks.
• docs/source/en/main_classes/callback.md — documents callbacks, TrainerCallback, TrainerState, and TrainerControl, which are the supported read-only hook mechanism around the Trainer loop.

Core API Components

The main public entry points are intentionally small in number. Trainer is the general-purpose PyTorch trainer. TrainingArguments is the configuration object that controls batch sizes, training duration, output directories, distributed strategies, mixed precision, logging, saving, and related runtime behavior. Seq2SeqTrainer inherits from Trainer and specializes evaluation and prediction for generation-oriented sequence-to-sequence tasks. Seq2SeqTrainingArguments inherits from TrainingArguments and carries the task-specific settings used by that trainer variant. The source docs expose these components with doc-builder [[autodoc]] blocks, meaning the published API reference is generated directly from the Python objects.

Sources: docs/source/en/main_classes/trainer.md, docs/source/ko/main_classes/trainer.md, docs/source/zh/main_classes/trainer.md

Execution Flow

A typical run starts by selecting a compatible model and preparing datasets. You then instantiate TrainingArguments before the Trainer, because those arguments define the execution plan the trainer uses for training, evaluation, logging, saving, precision, and distributed behavior. When training begins, the loop creates or receives dataloaders, batches examples, runs the model forward pass, computes loss, backpropagates gradients, steps the optimizer and scheduler, logs progress, evaluates when configured, and saves checkpoints into the configured output directory. The official docs summarize this as a loop that handles batching, shuffling, padding into tensors, forward passes, loss, gradient updates, and weight updates.

Sources: docs/source/en/main_classes/trainer.md, docs/source/ja/main_classes/trainer.md, docs/source/zh/main_classes/trainer.md

The localized Trainer pages are especially useful for understanding where to customize that flow. They list methods for creating the train, evaluation, and test dataloaders; logging; creating the optimizer and scheduler; computing loss; running a training step; running a prediction step; evaluating; and predicting. Those names are the boundary between ordinary configuration and subclassing. If you only need to change when logging, evaluation, or early stopping happens, prefer callbacks. If you need to change what the loop computes, such as a custom forward path or loss, subclass Trainer and override the relevant method.

Sources: docs/source/ja/main_classes/trainer.md, docs/source/zh/main_classes/trainer.md, docs/source/en/main_classes/callback.md

from transformers import Trainer, TrainingArguments
 
args = TrainingArguments(
    output_dir="./outputs",
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    do_train=True,
    do_eval=True,
    save_strategy="epoch",
    eval_strategy="epoch",
)
 
trainer = Trainer(
    model=model,
    args=args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    tokenizer=tokenizer,
    data_collator=data_collator,
    compute_metrics=compute_metrics,
 )
 
trainer.train()
metrics = trainer.evaluate()
predictions = trainer.predict(test_dataset)
trainer.save_model()

Customization Boundaries

There are two customization mechanisms, and they solve different problems. Subclassing changes the loop’s computation; callbacks observe the loop and return TrainerControl decisions. The callback documentation calls callbacks read-only pieces of code, except for the control object they return. They can inspect TrainingArguments, TrainerState, and loop events, and they are suited for progress reporting, TensorBoard or other platform logging, and decisions like early stopping. For changes that require modifying data loading, optimization, loss computation, or prediction behavior, the callback page explicitly points readers back to subclassing Trainer.

Sources: docs/source/en/main_classes/callback.md, docs/source/ja/main_classes/trainer.md, docs/source/zh/main_classes/trainer.md

The documented subclassing surface includes get_train_dataloader, get_eval_dataloader, get_test_dataloader, log, create_optimizer_and_scheduler, create_optimizer, create_scheduler, compute_loss, training_step, prediction_step, evaluate, and predict. The Japanese and Chinese docs include a weighted-loss example that subclasses Trainer, pops labels from the input batch, calls the model forward pass, extracts logits, computes a weighted CrossEntropyLoss, and returns either just the loss or a (loss, outputs) pair depending on return_outputs. That pattern keeps the rest of the trainer lifecycle intact while changing the training objective.

Sources: docs/source/ja/main_classes/trainer.md, docs/source/zh/main_classes/trainer.md

from torch import nn
from transformers import Trainer
 
class CustomTrainer(Trainer):
    def compute_loss(self, model, inputs, return_outputs=False):
        labels = inputs.pop("labels")
        outputs = model(**inputs)
        logits = outputs.get("logits")
        loss_fct = nn.CrossEntropyLoss(weight=torch.tensor([1.0, 2.0, 3.0], device=model.device))
        loss = loss_fct(logits.view(-1, self.model.config.num_labels), labels.view(-1))
        return (loss, outputs) if return_outputs else loss

Evaluation, Prediction, and Checkpoints

Evaluation and prediction are first-class parts of the Trainer API rather than separate scripts you must assemble yourself. evaluate runs the evaluation loop and returns metrics. predict returns predictions for a test set and can include metrics when labels are available. Seq2SeqTrainer highlights evaluate and predict because generation tasks often need task-aware decoding and metric computation. In practice, this means the same trainer object that performed training can be reused to validate intermediate checkpoints, compute final metrics, or produce test-set outputs with the same preprocessing and batching assumptions.

Sources: docs/source/en/main_classes/trainer.md, docs/source/ja/main_classes/trainer.md, docs/source/zh/main_classes/trainer.md

Checkpointing is tied to TrainingArguments.output_dir. The Japanese and Chinese pages state that checkpoints are saved under subfolders named checkpoint-xxx, where xxx represents the training step. Training can resume with Trainer.train(resume_from_checkpoint=True) to pick up the latest checkpoint, or with resume_from_checkpoint=checkpoint_dir to resume from a specific directory. When push_to_hub=True, checkpoints can also be saved to the Model Hub, and the documented hub-strategy values include keeping the latest checkpoint in a last-checkpoint folder or pushing all checkpoints as they appear in the output folder.

Sources: docs/source/ja/main_classes/trainer.md, docs/source/zh/main_classes/trainer.md

Model Compatibility and Backbones

The Trainer contract is broader than text classification, but the model still needs to cooperate with the training loop. For NLP and multimodal tasks, this usually means a Transformers model class whose forward method accepts labels and returns a loss. For vision tasks, a workload may involve a backbone: the backbone docs define a backbone as a feature-extraction model used for higher-level computer vision tasks such as object detection and image classification. Transformers provides AutoBackbone, BackboneMixin, BackboneConfigMixin, TimmBackbone, and TimmBackboneConfig to initialize and configure such models and their output features.

Sources: docs/source/en/main_classes/backbones.md, docs/source/en/main_classes/trainer.md

Backbones matter to Trainer users because many training recipes compose a feature extractor with a task head. Trainer does not remove the need for a well-defined forward contract; it standardizes the outer loop. If the model returns the expected ModelOutput or tuple, computes loss from labels, and exposes label names correctly, the same training infrastructure can apply across text, vision, audio, and multimodal tasks. If the model is only a feature extractor, you usually need a head or wrapper that produces task logits and loss before it can be trained directly with Trainer.

Sources: docs/source/en/main_classes/backbones.md, docs/source/en/main_classes/trainer.md

Callback API Reference

Callbacks are the public hook system for reacting to training events without rewriting the loop. The callback docs identify TrainerCallback as the main class, TrainerState as the object exposing internal training-loop state, and TrainerControl as the object through which callbacks influence actions. The available callback list includes built-ins and integrations such as DefaultFlowCallback, PrinterCallback, ProgressCallback, EarlyStoppingCallback, TensorBoardCallback, WandbCallback, MLflowCallback, CometCallback, AzureMLCallback, CodeCarbonCallback, ClearMLCallback, DagsHubCallback, FlyteCallback, KubeflowCallback, DVCLiveCallback, SwanLabCallback, and others documented through autodoc.

Sources: docs/source/en/main_classes/callback.md

The docs show two registration styles. You can pass callback classes or instances in the callbacks argument when constructing Trainer, or call trainer.add_callback() later with either a callback class or an instance. By default, TrainingArguments.report_to is documented as none which is a useful reminder that external reporting integrations must be configured deliberately rather than assumed. Treat callbacks as lifecycle observers and decision points; if a callback starts needing to mutate batches, replace optimizer creation, or alter loss, move that behavior into aTrainer` subclass instead.

Sources: docs/source/en/main_classes/callback.md

from transformers import TrainerCallback
 
class MyCallback(TrainerCallback):
    def on_train_begin(self, args, state, control, **kwargs):
        print("Starting training")
 
trainer = Trainer(
    model=model,
    args=args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    callbacks=[MyCallback],
 )
 
trainer.add_callback(MyCallback())

Practical Next Steps

Start with the simplest possible Trainer: a compatible model, TrainingArguments, a train dataset, and optionally an evaluation dataset. Add metrics and a data collator only when your task needs them. Once the baseline works, decide whether each customization is configuration, callback, or subclassing. Runtime decisions such as logging, progress display, integrations, and early stopping belong in callbacks. Changes to dataloaders, optimizer creation, loss computation, training steps, or prediction steps belong in subclass overrides. For sequence-to-sequence tasks, prefer Seq2SeqTrainer so evaluation and prediction use the task-aware path documented for that class.

Sources: docs/source/en/main_classes/trainer.md, docs/source/en/main_classes/callback.md, docs/source/ja/main_classes/trainer.md, docs/source/zh/main_classes/trainer.md