Quantization Methods

Purpose and Scope

Transformers exposes several quantization integrations so readers can trade memory footprint, speed, accuracy, calibration effort, and hardware support without changing the basic model-loading workflow. This page compares the method-specific guides for bitsandbytes, GPTQ, AWQ, HQQ, torchao, and Optimum Quanto. The shared pattern is to keep using the familiar auto model APIs, then pass a method-specific quantization configuration or load a checkpoint that already declares its quantization metadata. The differences matter: some methods quantize on the fly, some require calibration, some focus on pre-quantized checkpoints, and some are optimized for particular accelerator kernels.

Sources: docs/source/en/quantization/bitsandbytes.md, docs/source/en/quantization/gptq.md, docs/source/en/quantization/awq.md, docs/source/en/quantization/hqq.md, docs/source/en/quantization/torchao.md, docs/source/en/quantization/quanto.md

Use this page when you already understand why quantization is useful and need to choose an implementation path. If the goal is a quick memory reduction for a large language model, bitsandbytes and Quanto provide simple configuration-driven loading. If the goal is a highly accurate 4-bit checkpoint with known calibration cost, GPTQ and AWQ are more relevant. If the goal is no-calibration quantization across modalities with flexible bit widths, HQQ is the most directly documented path. If the goal is PyTorch-native composition with compilation, sparsity, optimizer quantization, or training features, torchao is positioned as the broader architecture-optimization option.

Relevant Source Files

docs/source/en/quantization/bitsandbytes.md - Documents the bitsandbytes integration, installation command, supported hardware, 8-bit and 4-bit features, and use through BitsAndBytesConfig with from_pretrained.
docs/source/en/quantization/gptq.md - Documents GPT-QModel usage in Transformers, GPTQConfig fields such as bits, dataset, and tokenizer, calibration expectations, device_map usage, and saving or pushing quantized models.
docs/source/en/quantization/awq.md - Documents loading AWQ checkpoints, detecting AWQ metadata in config.json, dtype and device_map behavior, FlashAttention2 usage, and fused module constraints.
docs/source/en/quantization/hqq.md - Documents HQQ installation, no-calibration quantization, HqqConfig for global or layer-specific settings, backend selection, PEFT compatibility, and torch.compile compatibility.
docs/source/en/quantization/torchao.md - Documents torchao as a PyTorch architecture optimization library, its supported quantization techniques, module-level configuration, hardware compatibility, and installation options.
docs/source/en/quantization/quanto.md - Documents Optimum Quanto installation, QuantoConfig weight quantization, modality and device compatibility, torch.compile usage, and the integration boundary for activation quantization or QAT.

Method Selection Model

A practical way to choose a method is to first identify whether quantization happens before loading, during loading, or as part of a broader optimization stack. bitsandbytes, HQQ, Quanto, and torchao are documented as configuration-driven ways to transform model layers during loading or preparation. GPTQ is documented as post-training quantization that uses a dataset and tokenizer to calibrate weights, while AWQ is documented mainly from the perspective of loading already-quantized checkpoints whose configuration records the quantization method. This distinction affects both developer workflow and operational risk, because calibration can take substantial time and pre-quantized checkpoints may already encode important implementation choices.

Sources: docs/source/en/quantization/bitsandbytes.md, docs/source/en/quantization/gptq.md, docs/source/en/quantization/awq.md

bitsandbytes is the most direct route when the team wants a widely adopted integration for large models and can satisfy its accelerator requirements. The guide describes a lightweight Python wrapper around hardware accelerator functions, with quantized linear layers, optimized optimizers, and matrix multiplication operations. It highlights two headline features: LLM.int8 for 8-bit inference and QLoRA for 4-bit trainable adaptation. The documented loading pattern works for models in any modality when they support Accelerate and contain standard linear layers, which makes it a good first experiment before adopting a more specialized quantization workflow.

Sources: docs/source/en/quantization/bitsandbytes.md

GPTQ and AWQ should be understood as accuracy-oriented 4-bit approaches with more up-front structure. GPT-QModel implements GPTQ as a post-training technique that quantizes each row of a weight matrix to minimize error, stores int4 weights, and restores them to fp16 on the fly during inference through fused kernels. AWQ preserves a small fraction of weights considered important for language model performance, and Transformers supports loading checkpoints produced by AWQ libraries such as llm-awq and autoawq. Both methods are attractive when a suitable quantized checkpoint already exists on the Hub or when the cost of quantization is acceptable.

Sources: docs/source/en/quantization/gptq.md, docs/source/en/quantization/awq.md

HQQ, torchao, and Quanto cover a different set of tradeoffs around flexibility and PyTorch-native composition. HQQ is documented as fast on-the-fly quantization for 8, 4, 3, 2, and 1-bit settings without calibration data, and it can apply one configuration to all linear layers or a dynamic mapping to selected layer names. torchao is broader than weight-only quantization: its guide lists quantization-aware training, float8 training, sparsity, optimizer quantization, KV cache quantization, custom kernels, and FSDP2 composition. Quanto sits between simplicity and portability by offering linear weight quantization to float8, int8, int4, or int2 across modalities and devices.

Sources: docs/source/en/quantization/hqq.md, docs/source/en/quantization/torchao.md, docs/source/en/quantization/quanto.md

System-to-Code Mapping

These source files are documentation pages, but they map directly to how users exercise the Transformers API. The common entry point is loading a model with a quantization configuration through the same pretrained model factory used elsewhere in the library. The method pages name configuration classes such as BitsAndBytesConfig, GPTQConfig, HqqConfig, AwqConfig, and QuantoConfig, then show how those configurations are passed to model loading. That organization is intentional: the docs teach users to keep task code stable while changing the quantization backend and configuration values.

Method	Main documented pattern	Calibration	Notable constraints or features
bitsandbytes	Pass BitsAndBytesConfig to from_pretrained	No separate calibration in the guide	LLM.int8, QLoRA, quantized linear layers, accelerator-specific support
GPTQ	Create GPTQConfig with bits, dataset, and tokenizer, then load with quantization_config	Yes	Calibration time can be significant; AutoGPTQ is no longer supported in favor of GPT-QModel
AWQ	Load a checkpoint whose config.json has quant_method set to awq	Usually performed before loading	Other weights default to fp16; fused modules cannot be combined with FlashAttention2
HQQ	Pass HqqConfig for global or layer-specific linear-layer quantization	No	Supports many bit widths, PEFT, torch.compile, and multiple backends
torchao	Use torchao quantization techniques and module-level configuration	Depends on technique	Composable with torch.compile, sparsity, optimizer quantization, KV cache quantization, and FSDP2
Quanto	Pass QuantoConfig with a weights target	No for the Transformers integration	Transformers integration supports weight quantization; use Quanto directly for activations, calibration, or QAT

Execution Flow

A typical workflow starts with dependency installation, because each backend lives in its own package ecosystem. The bitsandbytes guide installs Transformers, Accelerate, and bitsandbytes together. GPTQ installs Accelerate, Optimum, Transformers, and then GPT-QModel. AWQ uses autoawq for the documented example, with a warning that installing AutoAWQ can downgrade Transformers. HQQ installs the hqq package and may build CUDA kernels on CUDA devices. torchao is installed from PyPI or a PyTorch package index. Quanto installs optimum-quanto along with Accelerate and Transformers.

After installing dependencies, choose whether to quantize a base model or consume a quantized checkpoint. For configuration-driven loading, create the backend configuration and pass it as the model’s quantization configuration. For GPTQ, include the target bit width, calibration dataset, and tokenizer, then use automatic device placement and optionally max memory limits when the model or dataset is too large. For AWQ, inspect checkpoint metadata and then load the model; dtype can control the non-quantized weights, and device placement moves the checkpoint to the accelerator. These flows keep preprocessing and generation code mostly unchanged.

Sources: docs/source/en/quantization/gptq.md, docs/source/en/quantization/awq.md, docs/source/en/quantization/quanto.md

from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig
 
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m")
config = GPTQConfig(bits=4, dataset="c4", tokenizer=tokenizer)
model = AutoModelForCausalLM.from_pretrained(
    "facebook/opt-125m",
    device_map="auto",
    quantization_config=config,
 )

API Components and Options

The method-specific configuration objects are the practical API surface to remember. GPTQConfig is shown with bits, dataset, and tokenizer because GPTQ needs representative text for calibration. HqqConfig accepts nbits and group_size for a uniform linear-layer policy, and also accepts a dynamic mapping so attention projections and MLP projections can use different bit widths or group sizes. QuantoConfig uses a weights target, with the guide naming float8, int8, int4, and int2 as supported linear weight formats in the backend description. AWQ uses AwqConfig for fused modules, including do_fuse and fuse_max_seq_len.

Sources: docs/source/en/quantization/gptq.md, docs/source/en/quantization/awq.md, docs/source/en/quantization/hqq.md, docs/source/en/quantization/quanto.md

Several edge cases in the method pages should shape production decisions. GPTQ quantization from scratch can be slow on large models, so checking for an existing quantized Hub checkpoint is recommended before spending hours on calibration. AWQ fused modules can improve accuracy and performance, but they cannot be combined with FlashAttention2, so teams must choose the optimization stack deliberately. Quanto’s Transformers integration is limited to weight quantization; activation quantization, calibration, or quantization-aware training belongs in the Quanto library directly. HQQ’s lower bit depths can be useful, but lower precision should be validated against task quality before deployment.

Sources: docs/source/en/quantization/gptq.md, docs/source/en/quantization/awq.md, docs/source/en/quantization/hqq.md, docs/source/en/quantization/quanto.md

Hardware and Backend Considerations

Hardware support is not uniform across methods, and the documentation makes this a first-class selection criterion. bitsandbytes lists NVIDIA CUDA, Intel XPU, Intel Gaudi, and CPU backends, with separate minimum hardware expectations for 8-bit optimizers, LLM.int8, and NF4 or FP4 quantization. HQQ can use pure PyTorch and custom CUDA dequantization kernels, with additional fast inference backends after quantization. torchao documents CUDA, XPU, and CPU compatibility and emphasizes composition with PyTorch features. Quanto is framed as modality and device compatible, making it appealing where hardware portability is more important than a single vendor-specific kernel path.

Sources: docs/source/en/quantization/bitsandbytes.md, docs/source/en/quantization/hqq.md, docs/source/en/quantization/torchao.md, docs/source/en/quantization/quanto.md

For inference serving, measure the final stack rather than assuming quantization always improves latency. The method pages emphasize memory reduction, fused kernels, compilation, and backend-specific kernels, but real performance depends on model architecture, sequence length, device placement, attention implementation, and whether the bottleneck is memory bandwidth or compute. AWQ’s FlashAttention2 and fused-module guidance is a good example: each can improve performance, but the documented warning says they are mutually exclusive in that context. Treat quantization as one part of an optimization plan that also includes batching, attention backend selection, cache policy, and accelerator placement.

Sources: docs/source/en/quantization/awq.md, docs/source/en/quantization/torchao.md, docs/source/en/quantization/hqq.md

Next Steps

Start by testing the least disruptive method that satisfies your hardware and accuracy requirements. For a simple memory-footprint experiment, try bitsandbytes or Quanto with the same model and task code you already use. For a deployable 4-bit language model, search for an existing GPTQ or AWQ checkpoint before quantizing a base model yourself. For lower-bit experimentation, model-wide versus layer-specific quantization, or PEFT-friendly flows, evaluate HQQ. For PyTorch-native optimization stacks that combine quantization with compilation, sparsity, optimizer state reduction, or training features, continue into torchao’s guide and validate the exact technique on your target hardware.

Ask a question...