Generation API

Purpose and Scope

The Generation API is the reference surface for turning prepared model inputs into generated token sequences. In Transformers, generation is centered on the model method exposed by the PyTorch generation mixin, with behavior controlled by a generation configuration object and extended by utility classes that shape logits, collect outputs, stream tokens, and support continuous batching. This page is for developers who already know how to load a model and tokenizer and now need to understand which public classes participate when they call generation, how to inspect returned data, and where lower-level helper APIs fit into application code. Sources: docs/source/en/main_classes/text_generation.md, docs/source/en/internal/generation_utils.md

The main generation documentation establishes the reader-facing contract: each framework exposes a text generation method through its generation mixin, and users parameterize that method with a generation configuration instance. The internal utilities page complements that by documenting the structured output types and helper utilities used by generation. Read together, these files describe both the application entry point and the supporting components that advanced users rely on for scoring, decoding constraints, streaming, and server-style scheduling. Sources: docs/source/en/main_classes/text_generation.md, docs/source/en/internal/generation_utils.md

Relevant Source Files

• docs/source/en/main_classes/text_generation.md — Defines the main Generation page, the autodoc entries for GenerationConfig and GenerationMixin, and the continuous batching classes exposed in the public generation reference.
• docs/source/en/internal/generation_utils.md — Defines the Utilities for Generation page, including generate output structures, logits processors, logits warpers, and other helper classes used around generation.

Core API Components

The top-level entry point is GenerationMixin, whose documented methods are generate and compute_transition_scores. generate is the method applications call after tokenization or preprocessing, and it is where length limits, decoding mode, score returns, streaming, and model-specific generation behavior converge. compute_transition_scores is the companion inspection method for users who need token-level transition scores after a generation run, especially when they request score tensors. The documentation source lists these methods under the GenerationMixin autodoc block, making them part of the official reference rather than incidental implementation details. Sources: docs/source/en/main_classes/text_generation.md

GenerationConfig is the configuration object that groups generation parameters. The docs explicitly expose from_pretrained, from_model_config, save_pretrained, update, validate, and get_generation_mode. These methods define a practical lifecycle: load generation defaults from a checkpoint, derive a generation configuration from a model configuration, persist a customized configuration, update values for an experiment, validate the resulting state, and resolve the generation mode implied by settings such as beam search or sampling. This design keeps generation options reusable and serializable instead of forcing every caller to pass a long list of keyword arguments on every invocation. Sources: docs/source/en/main_classes/text_generation.md

The same main API page also lists ContinuousMixin, ContinuousBatchingManager, Scheduler, FIFOScheduler, and PrefillFirstScheduler. These names separate ordinary one-shot generation from continuous batching, where a server or long-running application coordinates multiple requests over time. A scheduler determines how pending work is ordered, while the continuous batching manager provides a higher-level coordination point. The reference page does not require every generate user to adopt continuous batching, but it exposes these classes beside the standard generation mixin so serving-oriented applications can use the same family of generation primitives rather than a completely separate API surface. Sources: docs/source/en/main_classes/text_generation.md

GenerationConfig Reference

GenerationConfig is the first place to look when a generation call behaves differently than expected. The official docs describe it as the complete list of parameters that control the generation method. The supplied documentation excerpt highlights length controls such as max_length, max_new_tokens, min_length, min_new_tokens, and early_stopping. In practice, max_new_tokens is the clearer control for conversational or prompt-completion workloads because it specifies how many tokens may be added beyond the prompt, while max_length includes the prompt length and remains for backward compatibility. The important operational habit is to distinguish prompt length from generated length before debugging truncated or unexpectedly long outputs.

Configuration also controls the decoding strategy selected by the generation call. The reference source exposes get_generation_mode, which is a strong signal that Transformers treats generation mode as a resolved outcome of configuration values rather than a single manually selected flag. For example, a user may adjust beam-related settings, sampling-related settings, or stopping settings, and the generation machinery uses the resulting configuration to determine the appropriate mode. Because validate is also documented, users should prefer updating and validating a GenerationConfig when building reusable application defaults instead of storing loosely checked dictionaries of options. Sources: docs/source/en/main_classes/text_generation.md

A useful workflow is to start from the checkpoint defaults, make only the application-specific changes, and save the result next to the model or in a project artifact. That workflow maps directly to from_pretrained, update, validate, and save_pretrained. For experiments, temporary keyword arguments to generate may be sufficient, but for production, serializing the generation configuration makes behavior reviewable and repeatable. When teams compare model outputs, the generation configuration should be treated as part of the model behavior contract, because changes to stopping, sampling, beams, or penalties can alter results as much as a checkpoint change.

Generate Outputs and Score Inspection

When return_dict_in_generate is enabled, generate returns a subclass of ModelOutput rather than only a tensor of sequences. The utilities documentation explains that these output objects can be used as regular attribute containers, tuples, or dictionaries. That matters because generation can optionally return many expensive diagnostic fields, and absent values are represented as None when accessed as attributes. When the same object is viewed as a tuple or dictionary, the None fields are omitted. This lets application code inspect generated sequences and scores while avoiding misleading empty entries for attentions or hidden states that were never requested. Sources: docs/source/en/internal/generation_utils.md

The GenerateDecoderOnlyOutput example shows the most common diagnostic path for causal language models: tokenize a prompt, call generate with return_dict_in_generate and output_scores, and then read sequences and scores. In that example, scores are present because output_scores was requested, while hidden_states and attentions are absent because their corresponding output flags were not set. This is an important edge case for debugging: a missing field usually reflects the requested output options, not necessarily a model failure. If downstream code expects attentions, hidden states, or per-step scores, the generation call must request them explicitly.

The internal utilities page documents multiple output types: GenerateDecoderOnlyOutput, GenerateEncoderDecoderOutput, GenerateBeamDecoderOnlyOutput, and GenerateBeamEncoderDecoderOutput. These names encode two distinctions. The first is model architecture: decoder-only models generate continuations from a prompt, while encoder-decoder models generate from encoded source inputs. The second is decoding family: beam outputs carry beam-search-specific structure. Choosing the right mental model helps when reading returned fields, because sequence shapes, score organization, and beam metadata differ across these cases even though they share the broader ModelOutput convention. Sources: docs/source/en/internal/generation_utils.md

from transformers import GPT2Tokenizer, GPT2LMHeadModel
 
tokenizer = GPT2Tokenizer.from_pretrained("openai-community/gpt2")
model = GPT2LMHeadModel.from_pretrained("openai-community/gpt2")
 
inputs = tokenizer("Hello, my dog is cute and ", return_tensors="pt")
generation_output = model.generate(
    **inputs,
    return_dict_in_generate=True,
    output_scores=True,
    max_new_tokens=20,
 )
 
print(generation_output.sequences)
print(generation_output.scores)

Logits Processing, Warping, and Constraints

Logits processors and warpers operate between the model’s raw language-model-head scores and the final token selection. The utilities documentation defines a LogitsProcessor as an object used to modify prediction scores for generation, and it lists individual processors with their call methods. This API is where Transformers implements constraints and score transformations such as forced beginning or ending tokens, no-repeat n-gram restrictions, repetition penalties, classifier-free guidance, invalid value cleanup, and probability-tail filtering. The key idea is that generation behavior can be shaped without changing model weights, because the processor transforms scores at each step before the next token is chosen. Sources: docs/source/en/internal/generation_utils.md

The named processors in the documentation illustrate different categories of intervention. ForcedBOSTokenLogitsProcessor and ForcedEOSTokenLogitsProcessor enforce required boundary tokens. EncoderNoRepeatNGramLogitsProcessor and EncoderRepetitionPenaltyLogitsProcessor connect decoder behavior to source-side content in encoder-decoder generation. EpsilonLogitsWarper and EtaLogitsWarper are filtering-style utilities that reshape the candidate token distribution. ClassifierFreeGuidanceLogitsProcessor supports guidance-style generation. InfNanRemoveLogitsProcessor protects generation from invalid numerical values. These tools are usually selected through GenerationConfig parameters or internal generation setup, but knowing their names helps when reading traces, extending generation, or debugging unexpected token suppression.

Because processors mutate scores rather than decoded text, they are sensitive to tokenization details. A forced token is a token id, not a human word; a no-repeat rule applies over token n-grams, not necessarily whitespace-delimited phrases; and probability filtering occurs before decoding. Developers should inspect token ids and decoded fragments together when a constraint seems surprising. This is also why generation debugging often involves both tokenizer output and generation outputs with scores enabled. The generation utility classes provide the score-level hooks, while tokenizers and processors define the vocabulary units those hooks operate on.

Streaming and Application Integration

Streaming is the application-facing pattern for returning generated text before the full sequence is complete. The official generation features docs describe TextStreamer as an object passed through the streamer parameter of generate, and the internal utilities page is the reference home for generation utility classes. A streamer needs put and end methods: put receives newly generated tokens, and end signals completion. This contract is intentionally small, which allows user interfaces, command-line tools, notebooks, and servers to adapt generation progress to their own output mechanism without rewriting the core generation loop. Sources: docs/source/en/internal/generation_utils.md

Streaming does not replace GenerationConfig or output inspection. It changes how partial results are surfaced while generation is running. A chat interface may stream decoded words to reduce perceived latency, but the same call can still rely on max_new_tokens, stopping behavior, logits processing, and model defaults. If the application also needs final scores or structured return data, design the callback path and final result path separately. Streamers are best treated as side-effect handlers for incremental output, while the returned generation object or tensor remains the durable result for logging, evaluation, or follow-up computation.

Continuous Batching and Serving-Oriented Helpers

The inclusion of ContinuousMixin, ContinuousBatchingManager, Scheduler, FIFOScheduler, and PrefillFirstScheduler in the main generation reference signals that Transformers documents server-oriented generation alongside the standard generate method. Continuous batching is relevant when requests arrive over time and a runtime wants to combine compatible work for better throughput. The named schedulers indicate that request ordering is configurable: a FIFO strategy emphasizes arrival order, while a prefill-first strategy names a serving concern specific to the initial context-processing phase of autoregressive generation. Sources: docs/source/en/main_classes/text_generation.md

For most scripts, the ordinary generate call is enough. Continuous batching becomes important when generation is embedded in a service, where multiple prompts compete for compute and memory, and where latency is affected by how prefill and token-by-token decoding are interleaved. The reference classes give advanced users vocabulary for reading the serving API: the mixin adds continuous-generation behavior, the manager coordinates batches, and schedulers define request selection policy. Readers working on local experiments should first master GenerationConfig and output inspection; readers building long-running inference services should continue to the continuous batching and serve documentation.

Practical Debugging Checklist

Start by confirming the generation configuration actually used by the model. If output length is surprising, compare prompt length, max_new_tokens, max_length, and stopping behavior. If token choices are surprising, check whether sampling, beam search, penalties, forced tokens, or no-repeat constraints are enabled. If diagnostic fields are missing, confirm that output_scores, output_hidden_states, or output_attentions were requested and that return_dict_in_generate is enabled. If scores are present but hard to interpret, pair them with compute_transition_scores and decoded tokens rather than reading raw tensors in isolation. Sources: docs/source/en/main_classes/text_generation.md, docs/source/en/internal/generation_utils.md

A compact reference map for this page is: GenerationMixin.generate runs generation, GenerationMixin.compute_transition_scores inspects token transition scores, GenerationConfig stores and validates generation behavior, Generate*Output classes structure returned data, LogitsProcessor and warpers alter scores before token selection, streamer classes surface incremental output, and continuous batching classes coordinate multi-request serving. The next useful pages are Text Generation for strategy guidance, Caching and KV Cache for performance behavior during autoregressive decoding, Continuous Batching for serving architecture, and Serve CLI for local server workflows.