Auto Classes and Model Loading

Purpose and Scope

Auto classes are the main way Transformers lets application code stay independent from a specific architecture name. Instead of importing BertModel, LlamaForCausalLM, or another model-specific class up front, you choose the public auto class that matches the resource or task you need, then call from_pretrained() with a checkpoint name or local path. The documentation frames this as automatic retrieval of the relevant configuration, model, tokenizer, vocabulary, or processor based on the supplied pretrained artifact.

Sources: docs/source/en/model_doc/auto.md, docs/source/en/models.md

This page explains the user-facing loading contract: what an architecture is, what a checkpoint is, how configurations participate in loading, and why there are several AutoModel variants. It is written for developers who want code that can switch checkpoints or model families without being rewritten, while still selecting the correct head for a task. If you need the lower-level details of individual model outputs or training loops, treat this page as the orientation layer before moving into task or API-specific documentation.

Relevant Source Files

• docs/source/en/model_doc/auto.md - Defines the Auto Classes documentation surface, including AutoConfig, AutoTokenizer, AutoFeatureExtractor, AutoImageProcessor, AutoVideoProcessor, AutoProcessor, generic AutoModel, task-specific AutoModelFor... classes, and the documented registration pattern for custom classes.
• docs/source/en/models.md - Explains model loading with PreTrainedModel.from_pretrained, the difference between architectures and checkpoints, the role of configuration and modeling files, safetensors loading behavior, and when to choose AutoModel versus a model-specific class.

Core Concepts

A model architecture is the skeleton: the layers, operations, dimensions, activation choices, and other structural decisions that define how computation happens. A checkpoint is a set of trained weights for a particular architecture, such as google-bert/bert-base-uncased for BERT. The loading guide notes that the word model is often used loosely for either the architecture or the checkpoint, so accurate reading depends on context. Auto classes bridge that ambiguity by inspecting the pretrained resource and constructing the correct library class for the requested use case.

Sources: docs/source/en/models.md

Configuration is the key link between a checkpoint and a concrete Python class. The loading guide describes each model as having a configuration file with attributes such as hidden layer count, vocabulary size, and activation function, and a modeling file that uses those attributes to build the layers. Loading pretrained weights with from_pretrained() combines the selected class, the configuration, and the weights from the Hub or a local directory. When safetensors weights are available, the documented loading path prefers that safer and faster format over traditional pickle-based PyTorch serialization.

Sources: docs/source/en/models.md

System-to-Code Mapping

The most important design choice is whether you need a bare model or a model with a head. A bare model, such as AutoModel, is useful when you want hidden states or are building your own downstream head. A task-specific auto class, such as AutoModelForCausalLM or AutoModelForSequenceClassification, attaches the appropriate head for generation, classification, question answering, token classification, and other supported tasks. The Auto Classes page explicitly organizes these classes by generic use, pretraining, natural language processing, computer vision, and additional task families.

Sources: docs/source/en/model_doc/auto.md, docs/source/en/models.md

Loading Flow

A typical loading flow starts with a checkpoint identifier and a task decision. For language generation, you might select AutoModelForCausalLM; for a reusable encoder, choose AutoModel; for text preprocessing, pair the model with AutoTokenizer; for multimodal or non-text inputs, use the matching processor class. The documented examples show from_pretrained() as the common entry point, with options such as device_map="auto" passed through when loading large models or letting the library place weights across available devices.

Sources: docs/source/en/model_doc/auto.md, docs/source/en/models.md

from transformers import AutoModelForCausalLM, AutoTokenizer
 
checkpoint = "meta-llama/Llama-2-7b-hf"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto")

The same pattern works when the exact architecture changes, as long as the checkpoint supports the selected task. That is the practical value of auto classes: your code states the intent, such as causal language modeling or sequence classification, and the library resolves the concrete class from the checkpoint configuration. If you instead import a model-specific class, you gain explicitness for one architecture but lose the ability to swap across supported architectures with the same small loading block.

Sources: docs/source/en/models.md

Extending Auto Classes

Auto classes are also extensible for custom architectures. The Auto Classes documentation shows the registration flow: define a custom configuration class, define the custom model, register the configuration key with AutoConfig, and register the model class against that configuration with AutoModel. The documented constraint is important: if the configuration subclasses PreTrainedConfig, its model_type should match the registration key; if the model subclasses PreTrainedModel, its config_class should match the registered configuration class.

Sources: docs/source/en/model_doc/auto.md

from transformers import AutoConfig, AutoModel
 
AutoConfig.register("new-model", NewModelConfig)
AutoModel.register(NewModelConfig, NewModel)

Registration is useful when you want project code to behave like the built-in model families without hard-coding imports everywhere. It also makes the custom model participate in the same high-level loading story: downstream code can keep using AutoConfig and AutoModel instead of learning a private constructor. The documentation presents this as an extension of the normal auto class workflow, not a separate plugin system, so the same configuration and checkpoint terminology still applies.

Practical Guidance

Use AutoTokenizer or an appropriate processor from the same checkpoint whenever possible, because model weights and preprocessing assets are designed to agree. A tokenizer determines token IDs, special tokens, and attention masks for text models; image, video, feature, and multimodal processors prepare non-text inputs into the tensor format expected by the model. Loading a model without its matching preprocessing component is a common way to produce shape, vocabulary, or input-name mismatches even when the model itself loaded correctly.

Sources: docs/source/en/model_doc/auto.md

Prefer task-specific auto classes when you intend to run a standard task end to end. For example, AutoModelForCausalLM is the correct choice for next-token language modeling, while AutoModelForSequenceClassification is the correct choice for sequence-level labels. Reach for AutoModel when you need hidden states or a backbone-like representation rather than logits from a predefined task head. When memory is a concern, keep loading options such as device_map="auto" in the same from_pretrained() call that retrieves weights and configuration.

Sources: docs/source/en/model_doc/auto.md, docs/source/en/models.md

Next Steps

After you understand auto classes, read the tokenizer and processor API pages to learn how inputs are prepared before inference, then read the generation or task-specific guides for the model heads you actually use. If you are adding a new architecture, combine this page with the contributor and custom model guidance so your configuration, model class, and auto class registration remain consistent with the documented PreTrainedConfig and PreTrainedModel conventions.