Quantization API
Purpose and Scope
The Quantization API page is the reference entry point for configuration objects and integration hooks used when loading or preparing lower-precision Transformers models. Quantization is the practice of representing model weights, and sometimes activations, with lower-precision data types so a model can use less memory and, on suitable hardware, run faster. In this documentation surface, the API reference is intentionally paired with a broader conceptual overview: the reference names the configuration classes and extension point, while the overview helps users choose a method based on hardware, bit width, serialization, fine-tuning support, and whether calibration is required.
Sources: docs/source/en/main_classes/quantization.md, docs/source/en/quantization/overview.md
This page is for readers who already know they want a quantized model and need to identify the public configuration class or integration category to use. In Transformers, quantization is not presented as one monolithic switch. The documentation lists separate configuration types for algorithms and backends such as bitsandbytes, AWQ, GPTQ, HQQ, Quanto, torchao, compressed-tensors, and several specialized formats. That separation matters because each backend has different hardware assumptions, supported bit widths, serialization behavior, and workflow requirements. The API layer gives each method a typed place to express those requirements.
Relevant Source Files
docs/source/en/main_classes/quantization.md— The main API reference page for quantization. It introduces quantization, points readers to the guide, and declares the autodoc sections for public configuration classes such asBitsAndBytesConfig,GPTQConfig,AwqConfig,TorchAoConfig, andHfQuantizer.docs/source/en/quantization/overview.md— The guide-level overview that explains why quantization reduces memory requirements and compares supported methods across hardware, bit widths, on-the-fly behavior, PEFT fine-tuning, serialization, and Transformers support.
API Components
The public reference page exposes quantization through named configuration classes rather than through one universal options dictionary. The documented entries include QuantoConfig, AqlmConfig, VptqConfig, AwqConfig, EetqConfig, GPTQConfig, BitsAndBytesConfig, HiggsConfig, HqqConfig, MetalConfig, Mxfp4Config, FbgemmFp8Config, CompressedTensorsConfig, TorchAoConfig, BitNetQuantConfig, SpQRConfig, FineGrainedFP8Config, QuarkConfig, FourOverSixConfig, FPQuantConfig, AutoRoundConfig, and SinqConfig. The same page also documents quantizers.base.HfQuantizer, which is the extension point named for quantization techniques that are not already supported directly by Transformers.
Sources: docs/source/en/main_classes/quantization.md
The configuration classes are best understood as method-specific contracts passed into model-loading or quantization workflows. For example, the official reference describes QuantoConfig as a wrapper for models loaded with Quanto, with fields for target weight dtype, optional activation dtype, and modules that should not be converted. It describes AqlmConfig in terms of additive quantization grouping, codebooks, bits per codebook, and optional linear weights to leave unquantized. Those examples show the general shape of the API: each class captures the decisions that are meaningful for one backend while keeping model-loading code from needing backend-specific ad hoc arguments everywhere.
Several entries correspond to commonly selected production or fine-tuning paths. BitsAndBytesConfig is the familiar route for 8-bit and 4-bit bitsandbytes loading, including QLoRA-oriented workflows. GPTQConfig and AwqConfig represent 4-bit methods that are usually associated with calibration or pre-quantized checkpoints. HqqConfig, SinqConfig, and EetqConfig are examples of methods called out in the docs ecosystem for on-the-fly quantization. CompressedTensorsConfig, FbgemmFp8Config, FineGrainedFP8Config, and Mxfp4Config map to more specialized formats or low-precision regimes. The reference list is therefore a catalog of integration points, not a recommendation that all methods behave alike.
Compact Reference
| Public API entry | Role in the quantization surface | Notes for use |
|---|---|---|
BitsAndBytesConfig | bitsandbytes 8-bit and 4-bit quantization configuration | Common for easy loading and PEFT/QLoRA-oriented workflows on supported accelerators |
GPTQConfig | GPTQ quantization configuration | Typically used with GPTQ quantized checkpoints or workflows that rely on calibration |
AwqConfig | AWQ quantization configuration | Represents the AWQ algorithm, commonly selected for 4-bit inference-oriented accuracy |
QuantoConfig | Quanto loading configuration | Official docs expose weight dtype, activation dtype, and modules excluded from conversion |
AqlmConfig | AQLM additive quantization configuration | Official docs expose input/output grouping, codebooks, bits per codebook, and exclusions for linear weights |
HqqConfig | HQQ quantization configuration | Used for Half-Quadratic Quantization style workflows |
TorchAoConfig | torchao quantization configuration | Intended for torchao-backed quantization paths, including workflows that may combine with PyTorch compilation features |
CompressedTensorsConfig | compressed-tensors format configuration | Represents loading or serializing supported compressed tensor formats |
HfQuantizer | Base extension point for unsupported or custom quantization techniques | Use this when implementing an integration not already represented by a built-in config class |
System-to-Code Mapping
The documentation structure separates the API contract from selection guidance. docs/source/en/main_classes/quantization.md is the reference surface: it starts by defining quantization as a way to reduce memory and computational costs with lower-precision weights and activations, then delegates each public type to doc-builder autodoc blocks. That means the page is intended to remain synchronized with the Python API declarations behind those names. When a reader wants the exact constructor fields, validation behavior, and generated reference text, this is the page in the docs tree that publishes those entries.
Sources: docs/source/en/main_classes/quantization.md
docs/source/en/quantization/overview.md provides the selection context that a pure API list cannot provide. It explains that full precision is often fp32, that fp16 and bf16 are common lower-precision baselines, and that quantization can go further to integer representations such as int8 or int4. It also states that Transformers supports many methods with tradeoffs, including methods that need calibration for higher accuracy or extreme compression and methods that work out of the box with on-the-fly quantization. The overview table then compares methods by on-the-fly support, CPU, CUDA, ROCm, Apple Metal, Intel GPU, torch compile compatibility, bit widths, PEFT fine-tuning, serialization, and Transformers support.
Sources: docs/source/en/quantization/overview.md
This split is important when designing application code. The configuration class is the object you use in code, but the overview is where you decide whether the class is appropriate for the deployment target. A server running on CUDA GPUs, a laptop on Apple Silicon, an Intel GPU environment, and a CPU-only offline conversion job may need different methods even if all of them are described as quantization. Likewise, a method suitable for loading a pre-quantized checkpoint may not be the right choice for on-the-fly conversion, and a method that serializes cleanly in Transformers may be easier to share or redeploy.
Selection and Execution Flow
A practical quantization workflow starts by defining the constraint: model too large for memory, inference latency too high, fine-tuning budget too small, or deployment hardware too limited. The overview page frames this as a method-selection problem because quantization trades efficiency against accuracy. Lower precision can shrink storage and memory use, but it may introduce quantization noise. Before choosing an API class, decide whether the model must be quantized on the fly at load time, whether you can run a calibration step, whether the target hardware supports the backend, and whether you need to save and reload the quantized artifact through Transformers.
After the constraint is clear, select the configuration class that corresponds to the method. For a straightforward 4-bit or 8-bit bitsandbytes workflow, use BitsAndBytesConfig. For a GPTQ checkpoint, use GPTQConfig; for AWQ, use AwqConfig; for torchao, use TorchAoConfig; and for Quanto, use QuantoConfig. The public API reference exists so these method-specific choices can be expressed explicitly. If a project introduces a quantization approach not represented by the documented classes, the reference page points to HfQuantizer as the base extension point for adding support.
The final step is to validate the operational consequences of the choice. Check whether the method is expected to run on the target CPU, CUDA GPU, ROCm GPU, Apple Metal device, or Intel GPU. Check the supported bit widths and whether the method is compatible with PEFT fine-tuning if adapters or QLoRA-style training are planned. Check whether the result is serializable with Transformers if the artifact must be pushed, cached, or reused. The overview table is specifically designed to answer those questions before you commit the API object to code.
Implementation Details and Constraints
The API list makes clear that quantization support in Transformers is integration-heavy. Some classes describe algorithms, some describe backend libraries, and some describe file formats or numerical formats. That is why a generic field such as “number of bits” is not enough to model the system. AQLM needs codebook-related settings; Quanto exposes supported target dtypes and conversion exclusions; bitsandbytes centers on 4-bit and 8-bit loading behavior; compressed-tensors refers to stored compressed formats; FP8-related entries capture hardware and format-specific expectations. The named config classes keep these differences explicit.
The overview also warns against assuming every method has the same preparation cost. Methods that require calibration may produce better accuracy for aggressive compression, but they need an additional representative dataset or calibration phase. On-the-fly methods are easier to apply at load time, but the best choice still depends on target accuracy, hardware kernels, and serialization needs. For library users, that means the quantization API is not just a performance option; it is part of the model compatibility contract. A quantized checkpoint, its config object, and the runtime backend have to match.
When documenting or extending a quantization integration, keep the public reference and overview aligned. A new method should have a configuration class or quantizer entry that can be autodocumented in the main API page, and it should be represented in the selection guidance if users need to compare it with other methods. The table-driven overview is especially useful for avoiding ambiguous claims such as “supports GPUs” or “supports 4-bit”; the docs encode the hardware families, bit ranges, and feature flags that users actually need when planning deployments.
Testing and Next Steps
For application developers, the next step is to move from this API catalog to the method-specific guide that matches the selected class. Start with the overview table, identify the viable methods for the hardware and bit width, then open the corresponding quantization guide for concrete loading examples and caveats. Use the API reference when you need exact constructor names or when reviewing code that passes a quantization configuration into model loading. If you are building a new integration, review HfQuantizer as the documented extension point before adding method-specific behavior.
For maintainers, the main signal to preserve is consistency between the reference page and the guide page. A public class listed in docs/source/en/main_classes/quantization.md should have enough generated API detail to be usable, and a method promoted in docs/source/en/quantization/overview.md should have accurate compatibility information. When adding or changing a backend, update the class reference, selection matrix, and method guide together so users can answer three questions without reading implementation code: what object do I instantiate, what hardware and bits does it support, and can I serialize or fine-tune the result?
Sources: docs/source/en/main_classes/quantization.md, docs/source/en/quantization/overview.md