Text Generation

Purpose and Scope

Text generation is the workflow for asking a generative model to continue, complete, or answer from an input prompt. In Transformers, the central public entry point is generate(), exposed through GenerationMixin for models with generative capabilities. The source documentation presents this as the API layer that coordinates decoding behavior, generation configuration, optional application features, and lower-level scheduling interfaces. This page orients you to the API names you will see in the docs, the parameters that most directly change output behavior, and the source files that publish the English and localized documentation surface.

Sources: docs/source/en/main_classes/text_generation.md, docs/source/en/generation_strategies.md, docs/source/en/generation_features.md

A language model does not generate a whole response in one operation. It repeatedly predicts a next token from the prompt plus the tokens it has already produced, then stops when a length limit, stopping criterion, or end-of-sequence condition is reached. The official text-generation guide frames this as the most common large language model application, while the repository source page frames the stable reference around GenerationMixin.generate, GenerationConfig, and related generated API documentation. In practice, you start with model and tokenizer loading, tokenize prompts, call model.generate(...), and decode the returned token ids into text.

The generation documentation is split intentionally. main_classes/text_generation.md is the API reference index: it names the classes, methods, and autodoc entries that are part of the public surface. generation_strategies.md is the conceptual guide for choosing how the next token is selected. generation_features.md is the application guide for features layered onto generate(), including streaming and watermarking. Read these together: the reference tells you what exists, the strategies guide tells you how decoding choices affect results, and the features guide shows how to integrate generation into products.

Relevant Source Files

• docs/source/en/main_classes/text_generation.md - English API reference page for generation, including GenerationConfig, GenerationMixin, continuous batching classes, schedulers, and the explicit pointer to generation strategies.
• docs/source/ja/main_classes/text_generation.md - Japanese localized generation reference, preserving the GenerationConfig and GenerationMixin framing and including translated framework-oriented explanations.
• docs/source/ko/main_classes/text_generation.md - Korean localized generation reference, including GenerationConfig, GenerationMixin, WatermarkingConfig, and translated guidance toward generation strategies.
• docs/source/zh/main_classes/text_generation.md - Chinese localized generation reference, mirroring the core GenerationConfig and GenerationMixin API structure.
• docs/source/en/generation_features.md - English feature guide for capabilities built on generate(), especially streaming through a streamer object and watermarking through WatermarkingConfig and WatermarkDetector.
• docs/source/en/generation_strategies.md - English guide explaining decoding strategies such as greedy search, sampling, and beam search, with concrete AutoModelForCausalLM, AutoTokenizer, and model.generate(...) examples.

Core Primitives

The first primitive is the model class with generation support. For causal language modeling examples, the docs use AutoModelForCausalLM.from_pretrained(...) together with AutoTokenizer.from_pretrained(...). The tokenizer turns text prompts into tensors; the model consumes those tensors and returns generated token ids; decoding converts ids back to user-facing text. The generation strategies guide shows this pattern with return_tensors="pt", an accelerator-selected device, and model.generate(**inputs, max_new_tokens=...), making the tokenizer-model-generate-decode loop the baseline workflow for readers moving beyond pipeline.

Sources: docs/source/en/generation_strategies.md

The second primitive is GenerationMixin.generate. The English API page says PyTorch generate is implemented in GenerationMixin, and the localized pages retain the same basic teaching pattern. The Japanese and Korean pages also mention TensorFlow and Flax/JAX generation mixins, which reflects the multilingual documentation history, while the current English and Chinese snippets focus on PyTorch. For users, the important idea is that generation is not a separate standalone service in the reference docs; it is a method available on compatible model objects and controlled through method arguments or a configuration object.

Sources: docs/source/en/main_classes/text_generation.md, docs/source/ja/main_classes/text_generation.md, docs/source/ko/main_classes/text_generation.md, docs/source/zh/main_classes/text_generation.md

The third primitive is GenerationConfig. The API source lists from_pretrained, from_model_config, save_pretrained, update, validate, and get_generation_mode under the autodoc entry. This tells you that generation settings are loadable, derivable from a model config, mutable, validatable, and serializable. Use ad hoc keyword arguments for quick experiments, but prefer a saved generation configuration when you need reproducible behavior across scripts, evaluation jobs, demos, or deployed applications. The reference page deliberately sends readers to the strategies guide to learn defaults, overrides, and persistence.

Sources: docs/source/en/main_classes/text_generation.md

Execution Flow

A typical local generation flow starts by choosing a checkpoint that matches the task, loading its tokenizer and model, preparing a prompt, and calling generate(). The strategies guide uses a causal language model because next-token generation is the natural shape of the task. It also explicitly sets max_new_tokens in examples, which is a useful habit because the API reference notes that length control is part of GenerationConfig and official docs recommend max_new_tokens for controlling how many tokens are produced independently of prompt length. After generation, use tokenizer decoding or batch decoding and skip special tokens when you want display-ready text.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
 
tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2")
model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2")
 
inputs = tokenizer("The secret to baking a good cake is ", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=20)
text = tokenizer.batch_decode(outputs, skip_special_tokens=True)

Sources: docs/source/en/generation_strategies.md, docs/source/en/generation_features.md

Once the baseline works, decide whether the output should be deterministic, diverse, or globally searched. Greedy search is the documented default and selects the most likely next token at each step. It is simple and useful for short, constrained outputs, but the source guide warns that it can repeat itself on longer sequences. Sampling is enabled with do_sample=True and num_beams=1; it draws from the model distribution and can produce more varied responses. Beam search tracks multiple candidate sequences and chooses based on overall probability, which can be useful when the best complete sequence is not found by a one-step greedy choice.

Sources: docs/source/en/generation_strategies.md

Decoding Parameters and Strategies

Generation parameters are not just tuning knobs; they define the contract between a prompt and the kind of completion you expect. Length settings such as max_new_tokens, min_new_tokens, max_length, and min_length determine how long generation may continue. The official reference favors max_new_tokens because it ignores prompt length, which makes scripts easier to reason about across short and long prompts. Beam-related settings such as num_beams and early_stopping affect search behavior, while sampling settings such as do_sample=True turn deterministic next-token selection into probabilistic selection.

The strategies guide should be your starting point when outputs look wrong but the model loads successfully. Repetition, bland completions, overly short answers, and surprising creativity are often decoding issues rather than model issues. Greedy search can be too narrow, sampling can be too unconstrained if used carelessly, and beam search can prefer high-probability but generic text. GenerationConfig exists so these choices can be made explicit, validated, saved, and reused instead of being scattered as unexplained keyword arguments across notebooks and scripts.

Sources: docs/source/en/main_classes/text_generation.md, docs/source/en/generation_strategies.md

API Components Reference

The compact reference above reflects names exposed by the documentation source, not every internal generation utility in the repository. GenerationMixin is the method-bearing surface most users touch. GenerationConfig is the parameter object that keeps generation behavior portable. The feature guide adds application-facing objects, notably TextStreamer for progressive output and watermarking classes for generated-text detection. The English reference also includes continuous batching and scheduler classes, which connects this page to serving and throughput-oriented documentation rather than only single-request notebook usage.

Sources: docs/source/en/main_classes/text_generation.md, docs/source/en/generation_features.md

Generation Features for Applications

Streaming matters when generation is part of an interactive application. Without streaming, users wait until the full response is produced and decoded. The feature guide explains that streaming returns text as soon as it is generated, reducing perceived latency and making progress visible. Transformers exposes this through the streamer parameter to generate(). The built-in TextStreamer is created with a tokenizer, and custom streamers can participate as long as they implement put() to receive tokens and end() to signal completion.

from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
 
tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2")
model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2")
inputs = tokenizer(["The secret to baking a good cake is "], return_tensors="pt")
streamer = TextStreamer(tokenizer)
 
_ = model.generate(**inputs, streamer=streamer, max_new_tokens=20)

Sources: docs/source/en/generation_features.md

Watermarking is the other feature highlighted in the supplied generation feature source. The docs describe it as a way to detect whether text is generated by biasing a subset of tokens, called green tokens, during generation and later checking their proportion. The feature guide names WatermarkingConfig for configuring the bias and algorithm, and WatermarkDetector for detection. It also notes practical details: prompt text should be stripped when it is much longer than the generated text, and padding can affect detection. Treat watermarking as generation-time behavior plus a later analysis step, not as a classifier-only feature.

Sources: docs/source/en/generation_features.md, docs/source/ko/main_classes/text_generation.md

Next Steps

If you are new to the library, start with a simple causal language model and a short prompt, then vary only one generation parameter at a time. Set max_new_tokens first so every run has a clear budget. Compare default greedy search with do_sample=True, then add more advanced strategy settings only after you can explain the difference in outputs. When you move from experiments to reusable code, save or load a GenerationConfig and keep prompt preprocessing, generation parameters, and decoding in one visible flow.

For product features, read the feature guide after the strategy guide. Add TextStreamer when latency and user feedback matter, and evaluate watermarking only if generated-text detectability is part of the product requirement. For high-throughput or server-oriented work, follow the continuous batching and serving pages because the English reference exposes ContinuousBatchingManager, schedulers, and related mixins alongside the core generation API. Related OpenWiki pages: generation-api, chat-templates, kv-cache, inference-optimization, and continuous-batching.