OpenWikihuggingface/transformers

Quantization Overview

Purpose and Scope

Quantization in Transformers is the set of techniques used to reduce model memory requirements and often improve practical inference throughput by storing weights, and sometimes activations, in lower precision. The overview documentation frames the main motivation clearly: full precision floating point weights are expensive for today’s large models, while half precision and integer representations can make larger checkpoints usable on available hardware. This page explains the decision workflow rather than every backend detail, so readers can choose between on-the-fly loading, calibrated compression, fine-tuning-oriented methods, and hardware-specific acceleration without treating all quantization methods as interchangeable.

Sources: docs/source/en/quantization/overview.md, docs/source/en/main_classes/quantization.md

The central tradeoff is resource efficiency versus model quality. Lower precision reduces storage, bandwidth, and memory pressure, but it introduces approximation error that may affect downstream accuracy. Transformers exposes many quantization methods because the best choice depends on the target bit width, whether calibration data is available, whether the model must be serialized back to the Hub, and whether the runtime is CPU, CUDA, ROCm, Apple Metal, Intel GPU, or a compiled PyTorch path. The quantization overview’s compatibility table is the first place to compare these operational constraints before choosing a backend.

Sources: docs/source/en/quantization/overview.md, docs/source/en/optimization_overview.md

Relevant Source Files

  • docs/source/en/quantization/overview.md - Defines the quantization guide entry point and the compatibility matrix for methods, hardware targets, bit widths, PEFT support, serialization, and Transformers support.
  • docs/source/en/kernel_doc/overview.md - Explains optimized kernel loading, runtime platform detection, caching, fallback behavior, and determinism considerations that can affect quantized inference performance.
  • docs/source/en/main_classes/quantization.md - Lists the public quantization configuration classes and the HfQuantizer extension point documented in the API reference.
  • docs/source/en/optimization_overview.md - Places quantization alongside compilation, attention backends, kernels, caching, parallelism, and continuous batching as an inference optimization technique.
  • docs/source/en/quantization/aqlm.md - Documents AQLM behavior, installation, model loading, codebook-based compression, LoRA compatibility, and kernel choices.
  • docs/source/en/quantization/auto_round.md - Documents AutoRound’s algorithm, installation, supported bit widths, hardware compatibility, offline quantization flow, command line usage, and Python API usage.

Core Concepts

A quantized model replaces some high precision numeric representation with a lower precision representation. In practical Transformers workflows, this usually means loading a checkpoint so linear layers or other supported modules use lower precision weights such as eight bit, four bit, or even lower bit encodings. Some methods operate immediately during loading, while others first run an offline quantization or calibration step and then save a quantized artifact. The repository documentation distinguishes these cases because they change the user experience: on-the-fly methods optimize convenience, while offline or calibrated methods can optimize accuracy, compression, or deployment reproducibility.

Sources: docs/source/en/quantization/overview.md, docs/source/en/quantization/auto_round.md

Method selection starts with the deployment objective. If the goal is to fit a model that otherwise exceeds memory, use the overview table to filter by supported bits and hardware. If the goal is speed, remember that smaller tensors reduce data movement, but pure speedups also depend on kernels, attention backends, compilation, and device support. If the goal is fine-tuning, check whether the method supports PEFT workflows and whether it can be serialized by Transformers. If the goal is production portability, prefer methods whose quantized outputs can be saved and reloaded through the documented Transformers APIs.

Sources: docs/source/en/quantization/overview.md, docs/source/en/optimization_overview.md

Loading and Configuration Pattern

The common loading pattern is still the normal Transformers model-loading flow: choose an Auto model class, call the pretrained loading method, and pass the appropriate quantization configuration or load a checkpoint that already carries quantized weights. The API reference documents quantization as a first-class main class area and lists configuration types such as QuantoConfig, AqlmConfig, VptqConfig, AwqConfig, EetqConfig, GPTQConfig, BitsAndBytesConfig, HqqConfig, TorchAoConfig, AutoRoundConfig, SinqConfig, and others. It also documents HfQuantizer as the extension point for techniques that are not already integrated.

Sources: docs/source/en/main_classes/quantization.md

from transformers import AutoModelForCausalLM
 
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B",
    device_map="auto",
    dtype="auto",
)

AQLM and AutoRound illustrate why the overview does not reduce quantization to a single flag. AQLM groups multiple weights and represents them as sums of vector codes, supports LoRA fine-tuning through PEFT, and is compatible with torch.compile for faster inference and training. It also has explicit installation requirements, including Python 3.10 or newer and optional GPU and CPU kernel support. AutoRound, by contrast, focuses on strong low-bit accuracy through tuning of rounding values and clipping thresholds, supports CPU, XPU, and CUDA, and currently documents offline generation of quantized models.

Sources: docs/source/en/quantization/aqlm.md, docs/source/en/quantization/auto_round.md

Method Selection Guide

Use the quantization overview matrix as a compatibility filter before considering benchmark claims. First, identify the hardware that will run the model, because the table separates CPU, CUDA GPU, ROCm GPU, Apple Silicon, and Intel GPU support. Next, decide the bit budget: some methods target four or eight bit use cases, while AQLM and other aggressive approaches can target one or two bits. Then check operational requirements such as on-the-fly quantization, torch.compile compatibility, PEFT fine-tuning, and whether the result is serializable through Transformers. This order prevents choosing an accurate method that cannot run in the target environment.

Sources: docs/source/en/quantization/overview.md

For quick experimentation, on-the-fly methods reduce setup because they do not require a separate calibration or tuning pass. For higher compression or stronger low-bit accuracy, methods like AQLM or AutoRound may require additional installation, quantization commands, or saved artifacts. The AutoRound documentation shows both command line and Python API workflows, including bit width, group size, symmetric quantization, optional mixed-bit layer configuration, and export format choices. Those details matter for teams that need repeatable builds, because the quantization recipe becomes part of the model artifact rather than just a runtime option.

Sources: docs/source/en/quantization/auto_round.md

auto-round --model facebook/opt-125m --bits 4 --group_size 128 --output_dir ./tmp_autoround

Performance, Kernels, and Runtime Behavior

Quantization is one optimization layer in a broader inference stack. The optimization overview explicitly distinguishes memory reduction from pure speed optimization and lists quantization alongside compilation, attention backends, kernels, caching, parallelism, and continuous batching. In practice, a quantized model may become faster because there is less data to move, but the result still depends on whether the runtime has efficient kernels for the quantized operations. This is why the quantization compatibility matrix includes torch.compile and hardware columns instead of only reporting the number of bits.

Sources: docs/source/en/optimization_overview.md, docs/source/en/quantization/overview.md

Kernel support is especially important for production expectations. The kernel overview explains that PyTorch operations are general purpose, while specialized kernels target specific platforms and operations such as attention, normalization, and fused computation. When enabled, Transformers can identify layers with available optimized kernels, download and cache binaries from the Hub only when needed, and fall back to standard PyTorch when no kernel is available. That fallback is useful for portability, but it also means a quantized model can load correctly without receiving the fastest possible execution path on every operation.

Sources: docs/source/en/kernel_doc/overview.md

API Components Reference

The quantization API surface is intentionally configuration oriented. Readers should look for a configuration class matching the backend or artifact they plan to use, then pass that configuration through the model-loading path documented for that method. The main classes page is the compact reference for supported names, while method pages explain backend-specific installation and workflow requirements. AQLM users should review codebook settings and supported inference kernels. AutoRound users should review bit width, group size, symmetry, mixed-bit layer configuration, recipe selection, and output format before generating artifacts for later loading.

Sources: docs/source/en/main_classes/quantization.md, docs/source/en/quantization/aqlm.md, docs/source/en/quantization/auto_round.md

  • Public extension point: HfQuantizer.
  • Common documented config families: BitsAndBytesConfig, GPTQConfig, AwqConfig, AqlmConfig, AutoRoundConfig, QuantoConfig, TorchAoConfig, HqqConfig, CompressedTensorsConfig, and SinqConfig.
  • Common selection dimensions: on-the-fly support, hardware support, bit width, torch.compile support, PEFT fine-tuning support, serializability, and Transformers integration status.
  • Common runtime companions: optimized kernels, attention backends, caching, compilation, and device mapping.

Practical Next Steps

Start by deciding whether the model must be merely loaded smaller, tuned into a quantized artifact, or served under strict latency constraints. Then use the overview table to eliminate methods that do not support the target hardware, bit width, serialization requirement, or fine-tuning workflow. After that, read the method-specific guide for installation and examples, because dependencies and calibration workflows vary significantly. Finally, test task quality and runtime behavior on representative inputs; the documentation notes that optimized kernels can affect determinism and that fallback paths may change performance even when model loading succeeds.

Sources: docs/source/en/quantization/overview.md, docs/source/en/kernel_doc/overview.md, docs/source/en/quantization/aqlm.md, docs/source/en/quantization/auto_round.md

Ask a question...