OpenWikihuggingface/transformers

Accelerate Integration

Purpose and Scope

Accelerate is the Transformers integration point for running the same training code on increasingly different hardware layouts. The documentation presents it as a unified interface over distributed backends such as FSDP and DeepSpeed, and as the layer that detects devices, process counts, distributed backends, and mixed precision choices before training begins. For a Transformers user, this means the model, optimizer, data loading, device placement, and distributed wrapping should not be hand-coded separately for every workstation, cluster, or notebook. The integration exists to let model code stay focused on the task while Accelerate handles the execution environment.

Sources: docs/source/en/accelerate.md

The current English page is especially focused on how Accelerate works with the Transformers Trainer. Trainer delegates distributed mechanics to Accelerate, so a Trainer-based script can be configured either with an Accelerate configuration file or with TrainingArguments. The localized pages show the complementary lower-level workflow for a native PyTorch training loop: install Accelerate, create an Accelerator object, pass training components through prepare, and replace the usual backward call with the Accelerate backward helper. Read together, these sources define both the managed Trainer path and the custom-loop path.

Sources: docs/source/en/accelerate.md, docs/source/de/accelerate.md, docs/source/es/accelerate.md

Relevant Source Files

  • docs/source/en/accelerate.md - Canonical English documentation for the current Accelerate page, including Trainer delegation, Accelerate config files, FSDP example settings, TrainingArguments integration, and accelerator_config usage.
  • docs/source/ar/accelerate.md - Arabic localized tutorial showing the native PyTorch loop pattern with Accelerator construction, prepare, backward, accelerate config, accelerate launch, and notebook_launcher.
  • docs/source/de/accelerate.md - German localized tutorial with the same distributed-training loop adaptation and a diff showing the small code changes needed for Accelerate.
  • docs/source/es/accelerate.md - Spanish localized tutorial explaining installation, Accelerator detection, prepare, backward, and distributed training from scripts.
  • docs/source/hi/accelerate.md - Hindi localized tutorial preserving the custom training loop sequence and the replacement of manual device placement.
  • docs/source/it/accelerate.md - Italian localized tutorial covering setup, prepare, backward, and the minimal code delta for distributed execution.

Core Primitives

The main primitive is the Accelerator object from the Accelerate library. In a custom training loop, it represents the runtime coordinator for the available distributed setup. After construction, it detects the environment and initializes the pieces needed for training, which is why the translated tutorials remove explicit model placement on a selected device. The prepare method is the second primitive: it receives the train dataloader, evaluation dataloader, model, and optimizer, then returns versions that are compatible with the chosen distributed mode. The backward method is the third primitive and replaces direct loss backpropagation so mixed precision gradient scaling and distributed concerns are handled consistently.

Sources: docs/source/ar/accelerate.md, docs/source/hi/accelerate.md, docs/source/it/accelerate.md

For Trainer users, the important primitives are configuration surfaces rather than direct loop calls. The English page states that Trainer calls the appropriate Accelerate APIs and delegates the distributed mechanics. An Accelerate configuration file can describe the hardware and backend selection, while TrainingArguments can carry backend-specific settings such as FSDP or DeepSpeed configuration. The accelerator_config field is also available for settings that do not have a dedicated top-level TrainingArguments field. This split matters because it lets a training script remain mostly stable while deployment-specific choices move into configuration.

Sources: docs/source/en/accelerate.md

Configuration Paths

Use the Accelerate config-file path when you want the launch environment to own the distributed setup. The documented flow is to run the Accelerate configuration command, answer questions about hardware and training, and produce a default configuration in the user cache. The example shown for FSDP includes a local machine environment, FSDP as the distributed type, sharded state dict behavior, a transformer-layer class to wrap, bfloat16 mixed precision, one machine, and four processes. Then a Trainer-based script is started with the Accelerate launcher, which reads that configuration before training starts.

Sources: docs/source/en/accelerate.md

accelerate config
accelerate launch train.py

Use TrainingArguments when the training script should carry the backend-specific choices directly. The English documentation shows FSDP receiving either a JSON config file or a dictionary through the FSDP configuration argument, and DeepSpeed receiving a JSON config file or dictionary through the DeepSpeed argument. It also calls out that when the Accelerate configuration file already covers those settings, the overlapping TrainingArguments fields are unnecessary. A practical rule is to choose one authority for distributed backend configuration per run so it is clear whether the launch file or the script controls the backend.

Sources: docs/source/en/accelerate.md

from transformers import TrainingArguments
 
args = TrainingArguments(
    ...,
    dataloader_pin_memory=True,
    accelerator_config={"non_blocking": True},
)

Execution Flow for Custom Loops

A native PyTorch loop usually starts with manual device selection, moving the model to that device, moving each batch to the same device, calling the model, computing loss, and running backward. The localized Accelerate tutorials replace that device-specific wiring with a shorter sequence. First, import and instantiate Accelerator. Second, create the model and optimizer normally. Third, pass the dataloaders, model, and optimizer through prepare. Fourth, remove per-batch device transfer and call the Accelerate backward helper on the loss. Optimizer stepping, scheduler stepping, zeroing gradients, and progress updates remain recognizable to PyTorch users.

Sources: docs/source/de/accelerate.md, docs/source/es/accelerate.md, docs/source/hi/accelerate.md

from accelerate import Accelerator
 
accelerator = Accelerator()
train_dataloader, eval_dataloader, model, optimizer = accelerator.prepare(
    train_dataloader, eval_dataloader, model, optimizer
)
 
for epoch in range(num_epochs):
    for batch in train_dataloader:
        outputs = model(**batch)
        loss = outputs.loss
        accelerator.backward(loss)
        optimizer.step()
        lr_scheduler.step()
        optimizer.zero_grad()

The reason this small change is enough is that prepare is the handoff point where Accelerate can adapt objects to the detected distributed setting. The translated pages explicitly emphasize that the user no longer needs to place the model on a device by hand, and the diff removes per-batch transfer from the loop. That is an important edge case for readers migrating existing scripts: if Accelerate is preparing the dataloaders and model, continuing to push batches manually to a hard-coded device can work against the abstraction and make the script less portable across CUDA, TPU-style notebook launching, or multi-process configurations.

Sources: docs/source/ar/accelerate.md, docs/source/de/accelerate.md, docs/source/it/accelerate.md

Selecting Accelerators and Precision

Accelerate chooses and configures the distributed environment, but you still need to decide which hardware should be visible to the run. The accelerator selection documentation in the official docs explains that hardware-specific environment variables can restrict and order devices before the Python program starts. For CUDA, setting the visible devices on the same command line as the run is safer than exporting a long-lived shell variable, because it reduces the chance of accidentally training on the wrong GPUs later. This applies whether a script is launched directly, with torchrun, or through the Accelerate launcher.

CUDA_VISIBLE_DEVICES=0,2 accelerate launch train.py
CUDA_VISIBLE_DEVICES=2,0 accelerate launch train.py
CUDA_VISIBLE_DEVICES= python train.py

Mixed precision is another selection decision that crosses Transformers and Accelerate. The English Accelerate page describes mixed precision as part of environment detection and shows bfloat16 in the FSDP configuration example. The mixed precision guide explains the practical choice: use bfloat16 on newer accelerators such as Ampere or newer GPUs when available, and use float16 on older hardware when bfloat16 is not the right fit. In Trainer workflows this is commonly expressed through TrainingArguments, while Accelerate is responsible for the distributed and gradient-scaling mechanics behind the configured run.

Sources: docs/source/en/accelerate.md

Practical Next Steps

For a new project, start with Trainer plus an Accelerate configuration file if you want the least code and the most reproducible launch setup. Use TrainingArguments for backend settings when the script itself should document the distributed policy. For an existing custom PyTorch loop, follow the localized tutorial pattern: instantiate Accelerator, prepare the dataloaders, model, and optimizer, remove manual device movement, and replace the direct backward call. After the loop works on one device, rerun it with a selected set of accelerators and then scale to FSDP or DeepSpeed only when the model size or throughput target requires it.

Sources: docs/source/en/accelerate.md, docs/source/es/accelerate.md, docs/source/hi/accelerate.md

Related pages: Trainer, Distributed and Parallel Training, DeepSpeed and FSDP, Mixed Precision Training, Environment Variables.

Ask a question...