Caching and KV Cache

Purpose and Scope

Caching in Transformers is an inference-time optimization for autoregressive generation. Autoregressive models predict one token at a time, and each next-token decision depends on the context that came before it. Without a cache, the model repeatedly recomputes key and value projections for previous tokens at every generation step. A key-value cache, usually shortened to KV cache, stores those attention-layer key and value tensors after they are computed so later steps can reuse them. This page explains the attention reason caching works, the cache strategies documented by Transformers, and how the strategy choice affects memory, speed, and generation workflows.

Sources: docs/source/en/cache_explanation.md, docs/source/en/kv_cache.md

The core warning is simple: caching is for inference, not training. The documentation explicitly warns that enabling caching during training can cause unexpected errors. That boundary matters because the optimization relies on the fact that, during causal generation, previously processed token representations do not change with respect to future tokens. Training and full-sequence forward passes have different correctness and gradient requirements, so the cache is presented as a generation-time mechanism rather than a general acceleration switch for all model execution.

Sources: docs/source/en/cache_explanation.md, docs/source/ko/cache_explanation.md

Caching is tightly connected to text generation strategy, but it is not itself a decoding strategy. Greedy search, sampling, and beam search decide which token or sequence candidate to choose next. The cache decides how efficiently the model computes the attention inputs needed to score those next tokens. The generation strategy documentation describes generate() as the API where decoding behavior is customized with parameters such as max_new_tokens, do_sample, num_beams, and num_return_sequences; the cache strategy documentation places cache control in the same generation workflow through use_cache, past_key_values, and GenerationConfig#cache_implementation. Sources: docs/source/en/generation_strategies.md, docs/source/en/kv_cache.md

Relevant Source Files

docs/source/en/cache_explanation.md - Explains why autoregressive generation repeats attention work, defines the KV cache concept, shows the scaled dot-product attention framing, and states the inference-only warning.
docs/source/en/kv_cache.md - Documents cache strategy selection, the default DynamicCache, use_cache=False, pre-initialized cache objects, past_key_values, and GenerationConfig#cache_implementation.
docs/source/ko/cache_explanation.md - Korean localization of the cache explanation, including additional visible details about the basic cache.update(k_t, v_t, layer_idx) interface, attention mask shape requirements, and layer storage examples.
docs/source/en/generation_strategies.md - Provides the generation context in which cache strategies operate, including GenerationConfig, generate(), greedy search, sampling, beam search, and common generation parameters.
docs/source/ja/generation_strategies.md - Japanese generation strategy documentation that reinforces the relationship between PreTrainedModel.generate(), model.generation_config, and customizable decoding parameters.
docs/source/ko/generation_strategies.md - Korean generation strategy documentation that reinforces the same generation API concepts and common parameters for multilingual documentation users.

Conceptual Model

Transformer attention uses query, key, and value tensors. The cache explanation describes scaled dot-product attention over a batch size, number of attention heads, sequence length so far, and dimension per head. The query, key, and value matrices are projections from input embeddings with a shape conceptually described as batch, heads, sequence length, and head dimension. In causal attention, the mask prevents attending to future tokens. Once a previous token is processed, its key and value representation can be reused because future tokens do not alter that past representation.

Sources: docs/source/en/cache_explanation.md

At step t, the model only needs the query for the current last token to produce the representation used to predict token t plus one. The prior keys and values can be treated as already-known context. The cache grows by storing the new key and value vectors and appending them to the past keys and values. The documentation emphasizes that attention is computed independently in each model layer, so caching is also per-layer. That per-layer structure is important when reasoning about memory: every layer maintains its own stored key and value tensors, not a single global attention buffer.

Sources: docs/source/en/cache_explanation.md, docs/source/ko/cache_explanation.md

The efficiency tradeoff follows directly from that structure. Without caching, each step recomputes all previous K and V tensors; with caching, each step computes only the current token's K and V and reuses the stored tensors from earlier steps. The documentation summarizes this as a move from quadratic attention work per step with respect to sequence length to linear per-step behavior for the cache update path, while memory still grows linearly with the amount of stored context. In practice, this is why KV caching is central to responsive long-form generation.

Sources: docs/source/en/cache_explanation.md

Cache Strategies

Transformers documents several Cache classes with different optimization goals. DynamicCache is the default for all models. It grows as generation progresses, storing more keys and values as new tokens are produced. This makes it convenient because users usually do not need to size the cache up front. The cache strategy page notes that for models with sliding window attention, such as Mistral and Gemma2, or chunked attention, such as Llama4, the cache stops growing once those layers reach their maximum sliding window or chunk size. That model-specific behavior keeps cache growth aligned with the attention mechanism.