Distributed and Parallel Training

Purpose and Scope

Distributed and parallel training in Transformers is best understood as a set of scaling choices around the same fine-tuning loop documented in the training guides. The core workflow remains: load a pretrained checkpoint, tokenize or otherwise preprocess data, assemble batches with a collator or processor, configure TrainingArguments, and run a trainer or framework-native loop. Parallelism changes where model states, batches, activations, optimizer updates, and communication happen; it should not change the semantic task you are training. This page compares the main choices so you can pick the smallest scaling mechanism that solves your bottleneck before moving to more complex sharding or model-parallel systems.

Sources: docs/source/en/training.md, docs/source/en/mixed_precision_training.md

The fine-tuning guide explicitly frames fine-tuning as continuing training from a pretrained model rather than starting from random weights, which is why distributed training is usually about throughput, memory, or model size rather than changing the learning objective. It also notes that TrainingArguments exposes the full set of training run options while the tutorial highlights only common ones and leaves specialized scenarios, including distributed training, to the complete API surface. In practice, that means your first design decision is not which launcher to use, but whether the model and batch fit on one device, whether data throughput is the limit, and whether optimizer or activation memory dominates.

Sources: docs/source/en/training.md

Relevant Source Files

• docs/source/en/training.md - Defines the primary Transformers fine-tuning workflow with tokenization, a data collator, model loading, TrainingArguments, and Trainer; it also identifies distributed training as a specialized scenario beyond the common options shown in the tutorial.
• docs/source/en/mixed_precision_training.md - Explains bf16, fp16, and tf32 training behavior, including the fp32 master-weight pattern and hardware guidance that directly affects distributed memory and throughput planning.
• docs/source/en/tasks/training_vision_backbone.md - Shows a task-specific training flow for vision models using a pretrained backbone, a task head, frozen backbone weights, dataset splitting, augmentation, and an image processor.
• docs/source/de/training.md - German localization of the earlier fine-tuning tutorial, preserving the same conceptual split between dataset preparation and training with Trainer, TensorFlow Keras, or native PyTorch.
• docs/source/es/training.md - Spanish localization of the earlier fine-tuning tutorial, useful for confirming the stable reader-facing terminology around pretrained models, tokenizers, padding, truncation, and fine-tuning.
• docs/source/ar/training.md - Arabic localization of the earlier fine-tuning tutorial, reinforcing that the documented training path is multilingual and centered on pretrained checkpoints plus task-specific datasets.

Core Training Primitives

The primitives that stay constant across distributed strategies are the pretrained model, the processed dataset, the batching function, the training configuration, and the trainer or native framework loop. The English fine-tuning guide loads a dataset, tokenizes a text column, removes columns the model forward method does not accept, splits the dataset for evaluation, and uses DataCollatorForLanguageModeling to dynamically pad each batch. This matters for multi-GPU work because dynamic padding can reduce wasted compute on every worker, while clean model inputs avoid per-rank failures caused by unexpected columns or inconsistent batch dictionaries.

Sources: docs/source/en/training.md

TrainingArguments is the configuration boundary where simple single-device scripts become scalable runs. The guide uses it as the place for training-run options and points readers to the full API for less common scenarios. When you move to distributed execution, keep the documented high-level options such as batch size, evaluation, saving, and dtype choices aligned across workers, and let the launch layer decide process placement. The safest pattern is to validate preprocessing and loss on a tiny subset first, then scale the number of devices, because data or shape bugs are easier to diagnose before collectives and replicated workers are involved.

Sources: docs/source/en/training.md

Vision training follows the same idea even though the inputs are images and annotations rather than token IDs. The backbone guide describes a computer-vision stack of pretrained backbone, optional neck, and task-specific head, then demonstrates DINOv3 ConvNext features with a DETR object-detection head. It freezes the backbone to preserve pretrained features and reduce trainable parameters, then uses AutoImageProcessor and augmentation to produce clean model inputs. In distributed settings, freezing a large backbone is a practical scaling lever: fewer trainable weights means less gradient communication and optimizer state, even when the forward pass still consumes memory.

Sources: docs/source/en/tasks/training_vision_backbone.md

Strategy Comparison

Distributed data parallelism, commonly abbreviated DDP, is the default first step when the model fits on one GPU but training is too slow or the desired global batch is larger than one device can handle. Each worker owns a full copy of the model, receives different batches, computes gradients, and synchronizes updates. This maps naturally to the documented Transformers fine-tuning flow because the model, tokenizer or processor, collator, and TrainingArguments stay conceptually unchanged. The main user-facing adjustment is to reason about per-device batch size versus global batch size, and to ensure preprocessing is deterministic enough that every worker sees valid examples.

Sources: docs/source/en/training.md, docs/source/de/training.md, docs/source/es/training.md, docs/source/ar/training.md

Tensor parallelism addresses a different problem: the model itself, or parts of its computation, are too large or too expensive for one device. Instead of replicating every weight on every GPU, tensor parallel systems split selected tensors and operations across devices. This is more invasive than DDP because the model’s layers, attention projections, feed-forward matrices, or similar large operations must be partitioned and coordinated. From the perspective of the documented fine-tuning loop, tensor parallelism comes after you have already minimized avoidable memory, selected an appropriate dtype, and confirmed that a single-device or DDP replica cannot fit the model at the needed sequence length or resolution.

Sources: docs/source/en/training.md, docs/source/en/mixed_precision_training.md

Expert parallelism is most relevant to mixture-of-experts models, where only a subset of expert modules may be active for a token or example. Its purpose is to distribute experts across devices so the aggregate parameter count can be large without every device holding or executing every expert in the same way. Expert parallelism should be evaluated as a model-architecture scaling choice, not as a generic replacement for DDP. In a Transformers training workflow, the same preprocessing and TrainingArguments discipline still applies, but the model’s routing and load-balancing behavior become part of the performance and stability envelope.

Sources: docs/source/en/training.md

Multi-GPU training is the umbrella term that includes DDP, tensor parallelism, expert parallelism, and combinations with precision or memory optimizations. A practical selection order is to begin with single-device correctness, enable mixed precision when supported, scale with DDP if the model fits, and only then introduce model-parallel techniques when memory or per-step compute remains the blocker. The mixed precision guide is important here because dtype choices affect every strategy: bf16 is recommended on Ampere or newer GPUs because it has the fp32 exponent range, while fp16 is a fallback for older hardware such as V100 or T4.

Sources: docs/source/en/mixed_precision_training.md

Precision, Memory, and Throughput

Mixed precision is often the simplest optimization to apply before or alongside distributed training. The mixed precision guide explains that full precision stores and computes everything in fp32, while mixed precision uses fp16 or bf16 for compute-heavy forward and backward passes while preserving an fp32 copy of weights for optimizer updates. This reduces activation and compute cost while maintaining training stability. In distributed jobs, those savings can determine whether DDP replication is feasible, whether larger per-device batches are possible, or whether tensor parallelism can be delayed until a genuinely model-size-driven constraint appears.

Sources: docs/source/en/mixed_precision_training.md

The same guide warns that loading the model directly in fp16 or bf16 makes autocast a no-op and leaves no fp32 master copy for optimizer updates. That distinction matters in multi-GPU settings because a silent dtype mistake is multiplied across ranks and can look like a distributed instability rather than a local configuration issue. If the model is numerically stable, direct lower-precision training can be a deliberate memory-saving choice; otherwise, use TrainingArguments(..., bf16=True) or TrainingArguments(..., fp16=True) so the mixed precision loop preserves the intended optimizer behavior.

Sources: docs/source/en/mixed_precision_training.md

tf32 is another throughput-oriented option for Ampere and newer GPUs. The mixed precision guide describes it as a compute mode for matrix multiplications that uses a shorter mantissa and can speed up training, especially with bf16 or fp16. Setting TrainingArguments(..., bf16=True, tf32=True) makes the choice explicit regardless of PyTorch version or environment defaults. For distributed training, this is a low-friction tuning knob: it does not change the parallelism topology, but it can improve per-rank compute throughput and reduce the pressure to add more devices prematurely.

Sources: docs/source/en/mixed_precision_training.md

Task-Specific Scaling Considerations

Text fine-tuning and vision training stress different parts of the system. In the text guide, tokenization creates input_ids and attention_mask, and dynamic padding avoids computing over unnecessary padding tokens. That makes sequence length distribution and collator behavior important for distributed efficiency: two jobs with the same number of examples can have very different step times if one has much longer sequences. In a multi-GPU setup, sorting, bucketing, or simply validating realistic batch shapes can be as important as choosing DDP, because every worker’s slowest operations influence synchronized step time.

Sources: docs/source/en/training.md

The vision backbone guide highlights different levers. It combines a pretrained backbone, task head, image processor, dataset split, and augmentation pipeline, and it explicitly freezes the backbone after assigning the pretrained checkpoint. For object detection or segmentation, image size, augmentation, and annotation cleanup influence both memory and correctness. When scaling across GPUs, keep augmentations valid and ensure bounding boxes remain clean before attempting large runs. A distributed job with invalid annotations can fail later and more expensively than a single-device smoke test, so the documented sequence of loading, splitting, augmenting, and processing remains valuable.

Sources: docs/source/en/tasks/training_vision_backbone.md

Recommended Execution Flow

Start by reproducing the smallest documented training path on one device. For language-model fine-tuning, that means loading the dataset, tokenizing the training column, splitting train and evaluation data, creating a data collator, loading the pretrained model with dtype="auto" when appropriate, and running a short Trainer job. For vision, it means confirming that the backbone, head, processor, augmentation, and labels work on a small subset. This step is not optional busywork; it establishes that the model contract and data contract are valid before adding process groups, cross-device synchronization, or model partitioning.

Sources: docs/source/en/training.md, docs/source/en/tasks/training_vision_backbone.md

Next, choose the least complex scaling mechanism that addresses the measured bottleneck. If training is correct but slow and each replica fits, use DDP-style data parallel scaling. If memory is tight, first try documented dtype controls such as bf16, fp16, tf32, careful batch sizing, dynamic padding, and freezing components that do not need updates. If the model still cannot fit or a single layer dominates memory and compute, evaluate tensor parallelism. If the architecture is mixture-of-experts, evaluate expert parallelism as an architecture-aware distribution strategy rather than as a generic accelerator.

Sources: docs/source/en/training.md, docs/source/en/mixed_precision_training.md

Finally, scale gradually and keep the training contract observable. Use small sample limits or small dataset splits while validating launch behavior, then increase data size, sequence length, image resolution, and number of devices. Check that evaluation, checkpoint saving, and Hub upload behavior still matches the intended workflow from the fine-tuning guide. When a run becomes unstable, isolate whether the cause is data preprocessing, dtype, optimizer behavior, batch shape, or the parallelism strategy itself. Related next pages are trainer, fine-tuning, accelerate, deepspeed-fsdp, llm-optimization, and inference-optimization.