Parameter-Efficient Fine-tuning

Purpose and Scope

Parameter-efficient fine-tuning, usually shortened to PEFT, is the Transformers workflow for adapting a pretrained model while training only a small number of extra parameters. The documentation defines those extra trainable parameters as adapters placed on top of a frozen base model. Because the optimizer tracks gradients and states for the adapter parameters rather than the full model, the workflow can substantially reduce memory use during fine-tuning. The adapter artifact is also much smaller than a full checkpoint, so it is easier to save, share, move between machines, and load from the Hub.

Sources: docs/source/en/peft.md, docs/source/ar/peft.md, docs/source/de/peft.md, docs/source/ja/peft.md

Transformers exposes PEFT as a first-class model capability rather than as a separate wrapping step for the main documented path. The English guide says the integration is available through PeftAdapterMixin on PreTrainedModel classes, allowing users to load, add, train, switch, and delete adapters directly from a Transformers model instance. The supported scope is deliberately bounded: non-prompt-learning methods such as LoRA, IA3, and AdaLoRA are supported by this integration, while prompt tuning, prompt learning, and prefix tuning should be handled through the PEFT library directly.

Sources: docs/source/en/peft.md, docs/source/en/main_classes/peft.md

Relevant Source Files

• docs/source/en/peft.md — Main English task guide for parameter-efficient fine-tuning, covering installation, adding adapters, choosing target modules, training with Trainer, checkpoint contents, and direct adapter management on Transformers models.
• docs/source/en/main_classes/peft.md — API reference page for integrations.PeftAdapterMixin, listing the adapter-management methods exposed on compatible Transformers models.
• docs/source/ar/peft.md — Arabic PEFT guide that corroborates the adapter-loading workflow, Hub or local adapter requirements, supported methods, and quantized loading examples.
• docs/source/de/peft.md — German PEFT guide with the same older adapter-loading structure, setup commands, supported PEFT method list, and AutoModelFor usage pattern.
• docs/source/ja/peft.md — Japanese PEFT guide that documents loading adapters, bitsandbytes-backed 8-bit or 4-bit loading, and adding adapters to an existing adapter model.
• docs/source/en/model_doc/d_fine.md — Model documentation page for D-FINE, useful here only as adjacent documentation context showing that model reference pages coexist with PEFT task and API pages.

Core Primitives

The core primitive is an adapter: a compact set of trainable parameters associated with a pretrained model. A base model such as a causal language model is loaded normally, then a PEFT configuration describes what adapter to insert and how it should behave. The current English guide uses LoraConfig and TaskType.CAUSAL_LM as the example configuration for a model loaded with AutoModelForCausalLM. Important configuration choices include rank, alpha, dropout, inference mode, optional modules to save, and explicit target modules when defaults are not appropriate for an architecture.

Sources: docs/source/en/peft.md

PeftAdapterMixin is the Transformers-facing API surface for those primitives. The API reference lists load_adapter, add_adapter, set_adapter, disable_adapters, enable_adapters, enable_peft_hotswap, active_adapters, get_adapter_state_dict, and delete_adapter. In practice, this means an application can keep a normal PreTrainedModel object and still manage adapter lifecycle operations from that object. This is especially useful for projects that already rely on AutoModel classes, Trainer, save_pretrained, or Hub loading patterns and want adapter fine-tuning without changing the surrounding model orchestration.

Sources: docs/source/en/main_classes/peft.md, docs/source/en/peft.md

Execution Flow

A typical PEFT flow starts by installing the PEFT package, with the current guide requiring a recent PEFT version for the direct Transformers integration. The user loads a pretrained model, creates a configuration such as LoraConfig, and attaches it with add_adapter under an optional adapter name. For common architectures such as Llama, Gemma, and Qwen, the guide explains that PEFT already has default target modules, so users often do not need to name layers manually. When adapting unusual modules or a model without defaults, target_modules can be supplied as a list or regex pattern.

Sources: docs/source/en/peft.md

Training then proceeds through the normal Trainer abstraction. Once the adapter is attached, the model can be passed to Trainer with TrainingArguments and a training dataset. The guide emphasizes that Trainer updates only parameters with requires_grad enabled, which means the frozen base model remains unchanged while adapter parameters are optimized. During checkpointing, the saved artifacts contain adapter weights and adapter configuration rather than a full duplicate of the base model. This is the main operational advantage: repeated experiments can produce many small adapter checkpoints for one shared pretrained model.

Sources: docs/source/en/peft.md

from peft import LoraConfig, TaskType
from transformers import AutoModelForCausalLM
 
model = AutoModelForCausalLM.from_pretrained("google/gemma-2-2b")
config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    inference_mode=False,
    r=8,
    lora_alpha=32,
    lora_dropout=0.1,
    modules_to_save=["lm_head"],
)
model.add_adapter(config, adapter_name="my_adapter")

Loading, Quantization, and Adapter Artifacts

The localized PEFT guides preserve an additional loading-oriented workflow: a Hub repository or local directory can represent a PEFT adapter if it contains adapter_config.json and the adapter weights. In that flow, a PEFT adapter can be loaded directly through an AutoModelFor class by passing the adapter model identifier to from_pretrained, or it can be loaded onto a separately loaded base model by calling load_adapter. The guides also note that either AutoModelFor classes or concrete base model classes such as OPTForCausalLM or LlamaForCausalLM can be used for adapter loading.

Sources: docs/source/ar/peft.md, docs/source/de/peft.md, docs/source/ja/peft.md

Quantized loading is a practical companion to adapter fine-tuning and inference when the base model is large. The Arabic and Japanese guides describe using bitsandbytes-backed 8-bit or 4-bit precision through BitsAndBytesConfig passed as quantization_config to from_pretrained. Combined with automatic device placement, this can reduce the memory needed to load the base model while the adapter remains small. This is distinct from the PEFT method itself: PEFT reduces trainable parameter and optimizer-state cost, while quantization reduces the storage or runtime representation cost of the loaded model weights.

Sources: docs/source/ar/peft.md, docs/source/ja/peft.md

API Components

The reference page is intentionally concise because the detailed signatures are generated through the documentation autodoc directive. For readers implementing a task, the guide page provides the sequence and examples, while the main class page names the stable API surface to look up when building adapter management into a larger application. The important design point is that these methods sit on the model object exposed by Transformers, not on a separately documented custom training loop. That keeps PEFT compatible with normal model loading, generation, Trainer usage, checkpointing, and Hub-centered workflows.

Sources: docs/source/en/main_classes/peft.md, docs/source/en/peft.md

Constraints and Next Steps

Use the integrated Transformers path when you are working with non-prompt-learning adapters such as LoRA, IA3, or AdaLoRA and you want to keep using PreTrainedModel and Trainer directly. Use the PEFT library documentation directly when the method is prompt based, because the reference states that prefix tuning and related prompt-learning approaches cannot be injected into a torch module through this mixin. If you need to fine-tune selected full layers alongside an adapter, configure modules_to_save so those modules are updated together with the adapter instead of remaining frozen.

Sources: docs/source/en/peft.md, docs/source/en/main_classes/peft.md

For a complete workflow, read this page together with the Fine-tuning and Trainer pages. The PEFT page explains how the trainable parameter set is reduced; the Fine-tuning workflow explains dataset tokenization, data collators, evaluation, and Hub upload; and the Trainer API explains the training loop and checkpoint behavior. If memory is the limiting factor, also read the Quantization Overview and Quantization Methods pages, because PEFT and quantization address different parts of the memory budget and are often combined for large language model experiments.