Fine-tuning

Purpose and Scope

Fine-tuning in Transformers means continuing training from a pretrained checkpoint on a smaller dataset that is specific to your task or domain. The English training guide describes it as the same training process as pretraining, except the model does not start from random weights, so it typically requires much less compute, data, and time. This page focuses on the standard workflow exposed in the docs: prepare and tokenize data, choose a collator, load a pretrained model, configure TrainingArguments, run Trainer, evaluate, and optionally push the result to the Hub.

Sources: docs/source/en/training.md

The repository also preserves earlier localized training guides that frame the same concept for a broader set of frameworks: Trainer, TensorFlow with Keras, and native PyTorch. Those pages use a Yelp Reviews classification example to teach dataset preparation, tokenization with padding and truncation, and optional dataset subsetting for faster experimentation. Read them as historical and localized tutorial surfaces; the current English guide is the most direct source for the modern large-language-model fine-tuning path, while the translated guides confirm that preprocessing before training is a stable teaching pattern across the docs.

Sources: docs/source/ar/training.md, docs/source/de/training.md, docs/source/es/training.md

Relevant Source Files

• docs/source/en/training.md - Current English fine-tuning guide covering login, tokenization, data collation, checkpoint loading, TrainingArguments, and Trainer-based language-model fine-tuning.
• docs/source/en/mixed_precision_training.md - Optimization guide for enabling bf16, fp16, and tf32 through TrainingArguments, including hardware guidance and precision caveats.
• docs/source/en/tasks/training_vision_backbone.md - Task-specific fine-tuning guide showing how a pretrained vision backbone can be combined with a task head, frozen, augmented, and trained for object detection.
• docs/source/ar/training.md - Arabic localized training guide that demonstrates preparing Yelp Reviews, using AutoTokenizer, mapping a tokenization function, and selecting smaller train/eval subsets.
• docs/source/de/training.md - German localized training guide with the same high-level choices among Trainer, TensorFlow/Keras, and native PyTorch workflows.
• docs/source/es/training.md - Spanish localized training guide showing dataset loading, tokenization with padding and truncation, and smaller dataset subsets for quicker fine-tuning.

Standard Workflow

A typical Transformers fine-tuning run starts with authentication if the trained artifact will be uploaded. The current guide instructs users to call huggingface_hub.login() before training so the finished model can be pushed to the Hub. That step is not part of the numerical training loop, but it is part of the developer workflow: it connects local experiments to reproducible sharing, model versioning, and later inference by model identifier. If you are only experimenting locally, you can skip upload-specific settings, but you should still choose a stable output directory and preserve tokenizer files with the model.

Sources: docs/source/en/training.md

Data preparation is the first model-facing step. In the current language-model example, the dataset is loaded with datasets.load_dataset, and the text column named horoscope is passed to an AutoTokenizer loaded from the same checkpoint family as the model. The tokenizer creates input_ids and attention_mask, which are the inputs expected by the model forward pass in that tutorial. The guide explicitly removes original dataset columns after tokenization, because columns such as raw text are useful for humans but are not accepted by the model call used during training.

Sources: docs/source/en/training.md

from datasets import load_dataset
from transformers import AutoTokenizer, DataCollatorForLanguageModeling
 
model_name = "Qwen/Qwen3-0.6B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
dataset = load_dataset("karthiksagarn/astro_horoscope", split="train")
 
def tokenize(batch):
    return tokenizer(batch["horoscope"], truncation=True, max_length=512)
 
dataset = dataset.map(tokenize, batched=True, remove_columns=dataset.column_names)
dataset = dataset.train_test_split(test_size=0.1)
data_collator = DataCollatorForLanguageModeling(tokenizer, mlm=False)

The tokenizer settings define the shape and cost of training. truncation=True with max_length=512 bounds the sequence length so batches fit within memory. The data collator then assembles tokenized examples into batches. For causal language modeling, DataCollatorForLanguageModeling(tokenizer, mlm=False) dynamically pads each batch to the longest sequence in that batch instead of padding every example in the full dataset to a fixed length. That distinction matters because unnecessary padding consumes memory and compute without adding training signal.

Sources: docs/source/en/training.md

Model Loading, TrainingArguments, and Trainer

After preprocessing, load a compatible pretrained checkpoint. The current guide uses AutoModelForCausalLM.from_pretrained(model_name, dtype="auto"), which keeps weights in their saved dtype instead of forcing them into torch.float32. This can avoid doubling memory usage when the checkpoint is stored in a lower precision such as bfloat16. Pair the model class with the task: causal language modeling uses AutoModelForCausalLM, while classification, question answering, vision detection, and other tasks require task-specific model heads or configurations.

Sources: docs/source/en/training.md

TrainingArguments is the central configuration object for a Trainer run. The docs describe it as the place where common training options are customized while less common or scenario-specific options keep reasonable defaults. In practice, this is where you set the output directory, evaluation strategy, batch sizes, learning rate, number of epochs or steps, logging and save behavior, Hub upload behavior, and precision flags. Trainer then combines the model, arguments, datasets, tokenizer or processor-related artifacts, and data collator into a training loop that can also evaluate and save outputs.

Sources: docs/source/en/training.md

A compact causal language modeling setup looks like this in shape, even though the exact arguments should be adjusted for your hardware and dataset. Use the same checkpoint name for the tokenizer and model, pass the split datasets created by train_test_split, and keep the collator aligned with the objective. For masked language modeling, the collator configuration differs; for causal language modeling, mlm=False tells the collator not to randomly mask tokens because the model learns next-token prediction from the original sequence.

Sources: docs/source/en/training.md

from transformers import AutoModelForCausalLM, TrainingArguments, Trainer
 
model = AutoModelForCausalLM.from_pretrained(model_name, dtype="auto")
 
args = TrainingArguments(
    output_dir="qwen-astro-horoscope",
    per_device_train_batch_size=2,
    per_device_eval_batch_size=2,
    eval_strategy="steps",
    logging_steps=50,
    save_steps=500,
    push_to_hub=True,
 )
 
trainer = Trainer(
    model=model,
    args=args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    data_collator=data_collator,
 )
 
trainer.train()
trainer.evaluate()
trainer.push_to_hub()

Evaluation, Upload, and Iteration

Evaluation is easiest to reason about when you create a validation split before training. The current guide uses train_test_split(test_size=0.1) so the trainer has examples that were not used for gradient updates. The older localized guides show the same concern from another angle: they create smaller shuffled train and evaluation subsets from tokenized Yelp Reviews to reduce turnaround time. That pattern is useful for debugging preprocessing, checking that labels and tensors line up, and confirming that the training loop runs before committing to a full dataset.

Sources: docs/source/en/training.md, docs/source/ar/training.md, docs/source/de/training.md, docs/source/es/training.md

Uploading to the Hub should happen after the run produces a useful checkpoint and associated tokenizer or processor files. The fine-tuning guide places login at the beginning because push_to_hub workflows need credentials, but the conceptual artifact is created at the end: a model repository containing weights, configuration, tokenizer or processor assets, and training metadata. When sharing a fine-tuned model, keep the model card clear about the base checkpoint, dataset, task, evaluation setup, and any important limitations introduced by truncation, dataset filtering, or domain-specific data.

Sources: docs/source/en/training.md

Precision and Memory Choices

Mixed precision is an optimization choice, not a different fine-tuning objective. The mixed precision guide explains that full precision training stores and computes in fp32, while mixed precision uses fp16 or bf16 for compute-heavy forward and backward passes and keeps an fp32 copy of weights for optimizer updates. In Transformers, the training-facing switch is direct: set TrainingArguments(..., bf16=True) or TrainingArguments(..., fp16=True). The guide recommends bf16 on Ampere or newer GPUs such as A100 and H100, and fp16 on older hardware such as V100 or T4.

Sources: docs/source/en/mixed_precision_training.md

The important caveat is how you load the model. The mixed precision guide warns that the model should be loaded in fp32 for mixed precision, otherwise autocast can become a no-op and there is no fp32 master copy for the optimizer update. This is different from the current fine-tuning guide's memory-saving dtype="auto" loading recommendation, which is useful when you intentionally want to load weights in their saved dtype. Choose one strategy deliberately: mixed precision for stable optimizer updates with autocast, or direct lower-precision training only when the model is numerically stable in that mode.

Sources: docs/source/en/training.md, docs/source/en/mixed_precision_training.md

from transformers import TrainingArguments
 
args = TrainingArguments(output_dir="run-bf16", bf16=True, tf32=True)
args = TrainingArguments(output_dir="run-fp16", fp16=True)

Beyond Text: Vision Backbone Fine-tuning

The same fine-tuning pattern applies beyond language, but the preprocessing and model assembly change with the modality. The vision backbone guide describes a computer vision workflow that starts with a pretrained backbone for feature extraction, adds a neck for feature enhancement, and attaches a task-specific head such as DETR for object detection. Its example combines a DINOv3 ConvNext backbone with a DETR object-detection configuration, sets num_labels=1 for license plate boxes, assigns the pretrained backbone, freezes it, and loads a DetrImageProcessor.

Sources: docs/source/en/tasks/training_vision_backbone.md

That guide highlights two modality-specific differences. First, images need an image processor and augmentation pipeline rather than text tokenization; the example installs albumentations, rescales images, flips them, applies affine transforms, and rebuilds object annotations to keep bounding boxes valid. Second, a pretrained backbone may be frozen to preserve feature extraction while training the task head, which reduces trainable parameters and can stabilize smaller datasets. The general recipe remains familiar: load data, split train and validation sets, preprocess into model-ready batches, configure the model, train, evaluate, and save or share the result.

Sources: docs/source/en/tasks/training_vision_backbone.md

Compact Reference

Next Steps

Start with the current docs/source/en/training.md workflow if you are fine-tuning a language model with Trainer. Move to the task-specific guides when your inputs are images, audio, video, or structured multimodal data, because preprocessing and collators are often task dependent. If memory pressure appears before the first successful run, read the mixed precision guide before changing model size or sequence length. After you have a clean baseline, consider parameter-efficient fine-tuning, distributed training, or task-specific example scripts to scale the same workflow safely.