Trainer API
Purpose and Scope
The Trainer API is the primary reference surface for training loops in Transformers. It is meant for developers who want the library to own the repetitive mechanics of PyTorch training while still exposing enough extension points for real projects. In this API family, Trainer owns the loop, TrainingArguments describes how the loop runs, data collators assemble batches, callbacks observe and steer events, and optimizer or scheduler helpers create the update policy. The documentation page for Trainer explicitly frames this pair as a complete training API for PyTorch with distributed training, mixed precision, and configurable behavior across common accelerator setups.
Sources: docs/source/en/main_classes/trainer.md
This page is a reference map, not a fine-tuning tutorial. It helps you decide which public object to open in the generated documentation, which extension point matches the change you need, and which supporting utility appears when reading Trainer internals. The most important boundary is between configuration, observation, batch construction, and loop modification. Use TrainingArguments to configure routine behavior, a data collator to convert dataset elements into model-ready batches, a callback to react to training events, and a Trainer subclass only when you need to change what the training loop computes.
Sources: docs/source/en/main_classes/trainer.md, docs/source/en/main_classes/callback.md, docs/source/en/main_classes/data_collator.md
The reference also covers sequence-to-sequence training. Seq2SeqTrainer and Seq2SeqTrainingArguments inherit from the base trainer and argument classes, but are adapted for tasks such as summarization and translation. That distinction matters because sequence-to-sequence workflows often evaluate generated sequences rather than only logits or scalar losses. When your workflow depends on generation-aware evaluation or prediction, open the sequence-to-sequence entries rather than assuming the base trainer reference contains every task-specific behavior.
Sources: docs/source/en/main_classes/trainer.md
Relevant Source Files
- docs/source/en/main_classes/trainer.md — Defines the public Trainer reference page, including
Trainer,Seq2SeqTrainer,TrainingArguments,Seq2SeqTrainingArguments, distributed and mixed-precision framing, and model compatibility warnings. - docs/source/en/main_classes/callback.md — Defines the callback reference, the read-only callback contract, registration examples,
TrainerCallback,TrainerState,TrainerControl, and built-in integration callbacks. - docs/source/en/main_classes/data_collator.md — Defines the data collator reference for default collation, tokenizer-aware padding, token classification, sequence-to-sequence batches, language modeling masks, whole-word masks, permutation language modeling, flattening, and multiple choice.
- docs/source/en/main_classes/optimizer_schedules.md — Defines the optimization reference for
Adafactor,SchedulerType,get_scheduler, individual schedule factories,GreedyLR, and the documented gradient accumulation capability of the optimization module. - docs/source/en/internal/trainer_utils.md — Defines internal Trainer utilities such as
EvalPrediction,IntervalStrategy, determinism helpers, distributed coordination,CallbackHandler,HfArgumentParser, and numerical debug utilities.
Core Training Contract
Trainer is optimized for Transformers models, and the documentation calls out model-output expectations that custom models must satisfy. A compatible model should return tuples or subclasses of ModelOutput, should compute a loss when a labels argument is provided, and should return that loss as the first tuple element when tuple outputs are used. The same warning explains that multiple label arguments are supported through label_names in TrainingArguments, but none of those arguments should be named label. These constraints are practical: the training loop must locate losses, labels, outputs, and predictions consistently across many model families.
Sources: docs/source/en/main_classes/trainer.md
TrainingArguments is the configuration object paired with the loop. The source page does not enumerate every option in handwritten prose because it delegates the full inventory to the documentation builder through an autodoc entry for all members. The conceptual contract is still clear from the page: the trainer and the arguments class are designed to work together, and the arguments class offers the wide range of options needed to customize how a model is trained. When you need exact option names, defaults, or generated parameter documentation, the TrainingArguments autodoc section is the authoritative reference entry.
Sources: docs/source/en/main_classes/trainer.md
The sequence-to-sequence API sits directly on top of this base contract. Seq2SeqTrainer inherits from Trainer, while Seq2SeqTrainingArguments inherits from TrainingArguments. The source page documents only selected Seq2SeqTrainer methods, specifically evaluate and predict, because those are the methods where generation-aware workflows most often need reference detail. Use the base trainer entries for shared loop behavior, then consult the sequence-to-sequence entries for evaluation and prediction behavior in summarization, translation, and similar encoder-decoder tasks.
Sources: docs/source/en/main_classes/trainer.md
Callback API and Extension Boundaries
Callbacks are objects that customize behavior by inspecting training state and taking decisions at defined loop events. The callback documentation gives progress reporting, TensorBoard or other ML platform logging, and early stopping as representative uses. A callback receives the TrainingArguments used to instantiate the trainer, can access internal state through TrainerState, and returns decisions through TrainerControl. This makes callbacks the right API for asking when or whether training should log, evaluate, save, stop, or report progress without taking ownership of the training step itself.
Sources: docs/source/en/main_classes/callback.md
The callback contract is deliberately read-only except for the TrainerControl object. The documentation states that callbacks cannot change anything in the training loop apart from the control object they return. That is the key safety rule for choosing between callbacks and subclassing. If your change is event-oriented, such as emitting metrics, adding a platform reporter, or stopping early, implement a callback. If your change requires a different dataloader, forward pass, loss computation, optimizer construction, or checkpoint algorithm, subclass Trainer and override the method that owns that behavior.
Sources: docs/source/en/main_classes/callback.md
The callback reference also documents registration style. A custom callback can subclass TrainerCallback and implement an event method such as on_train_begin. The callback can be passed in the callbacks list when constructing Trainer, either as a class or an instance, or it can be registered after construction with trainer.add_callback. Built-in callbacks include DefaultFlowCallback, PrinterCallback, ProgressCallback, EarlyStoppingCallback, and integrations for Comet, TensorBoard, Trackio, Weights and Biases, MLflow, AzureML, CodeCarbon, ClearML, DagsHub, Flyte, Kubeflow, DVCLive, and SwanLab.
Sources: docs/source/en/main_classes/callback.md
Data Collators and Batch Assembly
Data collators are the batch assembly layer between dataset rows and model inputs. The source documentation defines a collator as an object that forms a batch from a list of dataset elements, where those elements have the same type as items produced by train_dataset or eval_dataset. That definition is intentionally broad because collators range from simple feature stacking to task-aware transformations. A trainer does not require every dataset to be pre-padded or reshaped ahead of time; the collator can perform work that depends on the examples grouped into the current batch.
Sources: docs/source/en/main_classes/data_collator.md
Padding is the most common reason to choose a specialized collator. DataCollatorWithPadding is the tokenizer-aware entry for dynamic padding, which is usually preferable to padding every example to a global maximum length during preprocessing. Task-specific collators encode additional invariants. DataCollatorForTokenClassification handles token-level labels, DataCollatorForSeq2Seq prepares encoder-decoder batches, DataCollatorForMultipleChoice supports answer-choice layouts, and DataCollatorWithFlattening supports flattened batch layouts. These objects keep task-specific batch logic close to the trainer input boundary rather than scattering it through the dataset pipeline.
Sources: docs/source/en/main_classes/data_collator.md
Some collators also perform random batch-time augmentation. The language modeling collators are the clearest example: DataCollatorForLanguageModeling, DataCollatorForWholeWordMask, and DataCollatorForPermutationLanguageModeling document masking helper methods for NumPy and PyTorch paths. Random masking belongs naturally in a collator because it can vary across epochs while still operating on the formed batch. When choosing a collator, ask whether the transformation depends on batch shape, padding, labels, or randomness; if it does, the collator is usually a better fit than one-time dataset preprocessing.
Sources: docs/source/en/main_classes/data_collator.md
Optimizers, Schedules, and Utilities
The optimization reference groups update-policy helpers used with training workflows. It states that the .optimization module provides an optimizer with fixed weight decay for fine-tuning, several schedule objects that inherit from _LRSchedule, and a gradient accumulation class for accumulating gradients across multiple batches. The named optimizer entry is Adafactor. Scheduler selection starts with SchedulerType and get_scheduler, while individual factory entries expose constant, warmup, cosine, hard-restart, minimum-learning-rate, greedy, linear, polynomial, inverse-square-root, reduce-on-plateau, and warmup-stable-decay schedules.
Sources: docs/source/en/main_classes/optimizer_schedules.md
These schedule helpers are useful even when most users configure training through TrainingArguments. Example scripts, custom loops, and trainer subclasses sometimes need to construct the optimizer or learning-rate schedule directly. The factory names describe the shape of the schedule and the presence of warmup, restarts, minimum learning-rate behavior, plateau reduction, or stable-decay phases. Because the reference is generated through autodoc, use the individual entries for exact parameters, but use this page to decide which helper belongs in the training stack you are assembling.
Sources: docs/source/en/main_classes/optimizer_schedules.md
The internal utilities page is for readers who are studying Trainer code or building advanced extensions. It explicitly says most entries are only useful when studying the Trainer implementation. The page lists EvalPrediction for evaluation outputs, IntervalStrategy for timing choices, enable_full_determinism and set_seed for reproducibility, and torch_distributed_zero_first for distributed coordination. It also lists CallbackHandler under callback internals, HfArgumentParser for argument parsing, and DebugUnderflowOverflow for numerical debugging. Treat these as support APIs rather than the first surface for ordinary training scripts.
Sources: docs/source/en/internal/trainer_utils.md
Compact API Reference
| Area | Documented entry points | Source-level contract |
|---|---|---|
| Training loop | Trainer | Feature-complete PyTorch training API; expects compatible model outputs and loss behavior. |
| Training arguments | TrainingArguments | Configuration companion for customizing how training runs; full member list is generated by autodoc. |
| Seq2Seq loop | Seq2SeqTrainer.evaluate, Seq2SeqTrainer.predict | Sequence-to-sequence evaluation and prediction entries for summarization, translation, and related tasks. |
| Seq2Seq arguments | Seq2SeqTrainingArguments | Argument class inheriting from TrainingArguments for sequence-to-sequence workflows. |
| Callback protocol | TrainerCallback, TrainerState, TrainerControl | Read-only event extension model; callbacks communicate decisions through TrainerControl. |
| Callback registration | callbacks=[MyCallback], trainer.add_callback(MyCallback), trainer.add_callback(MyCallback()) | Custom callback classes or instances can be passed at construction or added later. |
| Collation | default_data_collator, DefaultDataCollator, DataCollatorWithPadding | Convert lists of dataset elements into batches, optionally applying padding. |
| Task collators | DataCollatorForTokenClassification, DataCollatorForSeq2Seq, DataCollatorForLanguageModeling, DataCollatorForWholeWordMask, DataCollatorForPermutationLanguageModeling, DataCollatorWithFlattening, DataCollatorForMultipleChoice | Build task-specific batches and, for language modeling collators, expose documented masking helpers. |
| Optimization | Adafactor, SchedulerType, get_scheduler, GreedyLR | Optimizer, scheduler selector, scheduler enum, and greedy schedule class entries. |
| Schedule factories | get_constant_schedule, get_constant_schedule_with_warmup, get_cosine_schedule_with_warmup, get_cosine_with_hard_restarts_schedule_with_warmup, get_cosine_with_min_lr_schedule_with_warmup, get_cosine_with_min_lr_schedule_with_warmup_lr_rate, get_greedy_schedule, get_linear_schedule_with_warmup, get_polynomial_decay_schedule_with_warmup, get_inverse_sqrt_schedule, get_reduce_on_plateau_schedule, get_wsd_schedule | Public factory entries for learning-rate schedules documented in the optimization reference. |
| Internal utilities | EvalPrediction, IntervalStrategy, enable_full_determinism, set_seed, torch_distributed_zero_first, CallbackHandler, HfArgumentParser, DebugUnderflowOverflow | Support types and helpers used by or around Trainer internals. |
Sources: docs/source/en/main_classes/trainer.md, docs/source/en/main_classes/callback.md, docs/source/en/main_classes/data_collator.md, docs/source/en/main_classes/optimizer_schedules.md, docs/source/en/internal/trainer_utils.md
Practical Selection Guidance
Start with Trainer and TrainingArguments when the default supervised loop matches your objective and your model follows the documented loss and output contract. Add a data collator when examples need dynamic padding, task-specific label shaping, multiple-choice reshaping, sequence-to-sequence preparation, or stochastic masking. Add callbacks for platform reporting, progress display, early stopping, or other decisions that depend on training state. Move to optimizer and scheduler helpers when you need explicit construction rather than argument-level configuration. Reach for internal utilities when you are debugging determinism, distributed ordering, argument parsing, callback dispatch, or numerical underflow and overflow.
Sources: docs/source/en/main_classes/trainer.md, docs/source/en/main_classes/callback.md, docs/source/en/main_classes/data_collator.md, docs/source/en/main_classes/optimizer_schedules.md, docs/source/en/internal/trainer_utils.md
For next steps, read the task-oriented Trainer and fine-tuning guides after using this page to identify the relevant API object. The API reference tells you which classes and functions exist; the guides explain the order in which to prepare datasets, tokenize or process inputs, choose a collator, instantiate arguments, define metrics, and launch training. If the change you need affects what the loop computes, study the Trainer methods and subclass deliberately. If the change only affects when actions occur, implement a callback and keep the training loop itself intact.