DeepSpeed and FSDP

Purpose and Scope

This page helps you choose and configure the large-model training integrations that reduce memory pressure when ordinary data parallel training is not enough. In Transformers documentation, the most directly source-backed path here is Fully Sharded Data Parallel, documented as FSDP2 in the English page and as FSDP in localized pages. FSDP shards parameters, gradients, and optimizer states across GPUs, then gathers the full parameters only when computation needs them. DeepSpeed ZeRO solves a similar memory problem with staged partitioning and optional CPU or NVMe offload, and it is typically selected through Trainer training arguments or a DeepSpeed configuration file.

Sources: docs/source/en/fsdp.md, docs/source/ko/fsdp.md, docs/source/zh/fsdp.md

The reader problem is usually concrete: a model, optimizer state, or batch size does not fit in GPU memory, but training should remain integrated with the Transformers Trainer workflow. FSDP and DeepSpeed are not model architectures; they are distributed training strategies that change how model state is stored and synchronized across ranks. The docs emphasize that the savings come with communication costs, so the right choice is a tradeoff between peak memory, throughput, checkpoint portability, and operational complexity rather than a universal upgrade over DDP.

Sources: docs/source/en/fsdp.md, docs/source/ko/fsdp.md, docs/source/zh/fsdp.md

Relevant Source Files

• docs/source/en/fsdp.md — English FSDP2 guide defining the sharding model, TrainingArguments.fsdp_config, reshard_after_forward, auto_wrap_policy, CPU offload, and transformer-layer wrapping behavior.
• docs/source/ko/fsdp.md — Korean FSDP guide with Accelerate setup, sharding strategy names, CPU offload, wrapping policies, checkpointing guidance, and TPU notes.
• docs/source/zh/fsdp.md — Chinese FSDP guide with the same operational flow plus concrete accelerate config, checkpoint restore, full-state-dict save, TPU, and example YAML material.
• docs/source/_config.py — Documentation build configuration that injects the default first notebook install cell with transformers datasets evaluate accelerate, which matters because FSDP docs depend on Accelerate.
• docs/source/ar/_config.py — Arabic documentation build configuration mirroring the install cell behavior for localized docs.

Core Concepts

FSDP2 is described as a sharding strategy where each GPU owns only a shard of parameters, gradients, and optimizer state at rest. Before a forward computation, ranks all-gather the parameter shards so each GPU can compute with the complete parameters, and after the computation the system frees or reshards state to recover memory. Gradients are then reduce-scattered so each rank keeps only its shard for the optimizer step. This is the central distinction from DDP-style replication: FSDP spends more communication to avoid keeping the full training state resident on every GPU.

Sources: docs/source/en/fsdp.md

The most important FSDP2 tuning knob in the English guide is reshard_after_forward, configured through TrainingArguments.fsdp_config. Setting it to true favors lower peak memory by resharing parameters after the forward pass. Setting it to false favors throughput by keeping gathered parameters between forward and backward, avoiding another all-gather at the cost of higher memory. This makes FSDP configuration an explicit memory-versus-speed decision. Start with the setting that lets the model fit, then relax it only if memory headroom allows and communication becomes the bottleneck.

Sources: docs/source/en/fsdp.md

DeepSpeed ZeRO uses a related but differently packaged model of memory reduction. The official DeepSpeed docs describe ZeRO stages that progressively shard optimizer states, gradients, and parameters, with later stages saving more memory but requiring more communication. ZeRO-2 is often the less expensive choice when sharding gradients and optimizer states is enough; ZeRO-3 is appropriate when parameters must also be partitioned to fit. In Transformers, Trainer integrates DeepSpeed configuration handling, while lower-level use can rely on HfDeepSpeedConfig to keep a DeepSpeed configuration available during model loading.

FSDP Configuration Flow

The localized FSDP guides show the practical setup path through Accelerate. Install Accelerate, run accelerate config, choose FSDP as the distributed type, and answer the prompts that generate a training-environment configuration file. Transformers can then launch Trainer-based training using the environment described by Accelerate. The documentation configuration files also inject notebook setup cells that install transformers, datasets, evaluate, and accelerate, reinforcing that distributed examples are expected to have Accelerate available before FSDP features are exercised.

Sources: docs/source/ko/fsdp.md, docs/source/zh/fsdp.md, docs/source/_config.py, docs/source/ar/_config.py

pip install accelerate
accelerate config

FSDP sharding strategy names in the localized docs describe how much state is partitioned. FULL_SHARD partitions parameters, gradients, and optimizer states. SHARD_GRAD_OP partitions gradients and optimizer states, but not the full parameter state in the same way. NO_SHARD behaves like no sharding and is equivalent to DDP for this purpose. HYBRID_SHARD and HYBRID_SHARD_ZERO2 are hybrid variants where sharding is scoped within workers while complete copies also exist at another grouping level. These options are selected with the fsdp_sharding_strategy setting produced by Accelerate.

Sources: docs/source/ko/fsdp.md, docs/source/zh/fsdp.md

Wrapping, Loading, and Checkpointing

Wrapping policy determines whether FSDP actually produces useful memory savings. The English guide warns that without wrapping, the entire model becomes one FSDP unit and the memory benefit is lost. TRANSFORMER_BASED_WRAP is the default-oriented strategy for Transformer models because it wraps transformer layers as separate units; each unit gathers its own parameters for the forward pass, then releases previous units’ parameters. The localized docs also mention size-based wrapping, where FSDP applies to modules above a parameter-count threshold. Use transformer-based wrapping first for common Transformer architectures, then size-based wrapping when layer classes are unusual.

Sources: docs/source/en/fsdp.md, docs/source/ko/fsdp.md, docs/source/zh/fsdp.md

Checkpointing is another place where memory strategy becomes an operational constraint. The localized guides recommend saving intermediate checkpoints as SHARDED_STATE_DICT, because saving a full state dict on rank 0 can take long enough to cause distributed timeout problems, especially with CPU offload. To resume, load the sharded checkpoint through Accelerate state loading. At the end of training, switch the FSDP plugin to FULL_STATE_DICT before trainer.save_model(...) so the output artifact is not tied only to FSDP-style loading.

Sources: docs/source/ko/fsdp.md, docs/source/zh/fsdp.md

# Resume from a sharded FSDP checkpoint.
accelerator.load_state("ckpt")
 
# Save a final portable model after FSDP training.
if trainer.is_fsdp_enabled:
    trainer.accelerator.state.fsdp_plugin.set_state_dict_type("FULL_STATE_DICT")
 
trainer.save_model(script_args.output_dir)

DeepSpeed, FSDP, and Related Parallelism Choices

Use FSDP when you want PyTorch-native sharding integrated through Accelerate and Trainer configuration, especially when the model or optimizer states do not fit on a single GPU. Use DeepSpeed ZeRO when your training stack already standardizes on DeepSpeed JSON configuration, ZeRO stages, or offload behavior, or when ZeRO-2 versus ZeRO-3 gives a clearer operational ladder for the memory target. Both approaches are designed to make larger models trainable; neither removes the need to tune batch size, activation checkpointing, precision, checkpoint format, and communication settings.

Sources: docs/source/en/fsdp.md, docs/source/ko/fsdp.md, docs/source/zh/fsdp.md

Ulysses-style sequence parallelism belongs to the same family of large-model scaling decisions, but it targets a different axis: sequence dimension partitioning rather than only model-state sharding. In practice, treat it as complementary design space to evaluate when long context length, attention memory, or sequence length dominates your training limit. The source-backed FSDP material on this page explains how Transformers documents state sharding and Trainer/Accelerate configuration; use the specific Ulysses or sequence-parallel guide for implementation-specific launch flags, compatibility constraints, and attention-backend requirements.

Execution Checklist

A safe workflow is to first prove the task with ordinary single-node or DDP training, then enable FSDP only when memory requires it. Next, generate an Accelerate configuration, choose an FSDP sharding strategy, set transformer-based wrapping with the correct layer class, and decide whether CPU offload is worth the transfer overhead. During training, save sharded intermediate checkpoints and test resume before running a long job. Before publishing or reusing the model outside the FSDP job, switch to a full state dict and call save_model so downstream users can load the checkpoint normally.

Sources: docs/source/en/fsdp.md, docs/source/ko/fsdp.md, docs/source/zh/fsdp.md

Next, read the Trainer and distributed-training pages to see how TrainingArguments, Accelerate launch configuration, mixed precision, and multi-GPU execution fit around these settings. If you are using DeepSpeed, compare the ZeRO stage and offload plan against the FSDP sharding table above instead of enabling both blindly. If the bottleneck is not model-state memory but long sequence length, follow the repository’s sequence-parallel or Ulysses-specific documentation when available, because the required constraints are different from FSDP wrapping and state-dict management.