Continuous Batching

Purpose and Scope

Continuous batching is the Transformers inference pattern for serving many text-generation requests without forcing every request to wait for the slowest item in a fixed batch. In static batching, a server groups requests, runs generation for the group, and only admits a new group after the current one completes. The continuous batching docs describe a different loop: at every generation step, the scheduler checks which requests are finished, removes them, and immediately admits waiting requests that fit the available scheduling budgets. The goal is to keep the GPU occupied while lowering average latency for mixed workloads with short and long generations.

Sources: docs/source/en/continuous_batching.md, docs/source/en/continuous_batching_architecture.md

This feature is primarily about causal language model generation, where inference alternates between loading prompt tokens into the key-value cache and decoding new tokens one step at a time. That incremental structure makes batching more dynamic than a single forward pass over fixed-size inputs. A short prompt that only needs a few tokens should leave the active batch as soon as it is done, while a long request can continue decoding. The user guide presents continuous batching both as a Python API and as a serving optimization, with production deployments directed to transformers serve and its OpenAI-compatible HTTP endpoint.

Sources: docs/source/en/continuous_batching.md

Relevant Source Files

• docs/source/en/continuous_batching_architecture.md — Explains the scheduler model, request lifecycle, chunked prefill, and the token, cache, and request-cap budgets that determine which work enters each forward pass.
• docs/source/en/continuous_batching.md — Provides the user guide, including generate_batch, ContinuousBatchingManager, model-loading examples, request submission, and the production serving tip.
• docs/source/en/main_classes/continuous_batching.md — Defines the API reference landing page for ContinuousMixin.generate_batch, ContinuousBatchingManager, and ContinuousBatchingConfig through autodoc entries.
• docs/source/_config.py — Supplies the English documentation notebook setup cell and doc-builder formatting substitutions used by examples in the docs build.
• docs/source/ar/_config.py — Mirrors the notebook setup cell and formatting substitutions for the Arabic documentation build, showing the continuous batching docs live inside the localized documentation system.

Core Primitives

The main primitives are requests, tokenized prompts, generation configuration, continuous batching configuration, a manager, and a scheduler. A request is a unit of generation work submitted by a caller. Its prompt is represented as input token IDs, not raw text, at the continuous batching API boundary. Generation settings such as max_new_tokens and eos_token_id come from the normal generation configuration path. Continuous batching settings control how much work the scheduler may place in each forward pass and how much memory pressure the loop may create.

Sources: docs/source/en/continuous_batching.md, docs/source/en/main_classes/continuous_batching.md

The guide gives two ways to use those primitives. ContinuousMixin.generate_batch is the convenience entry point when all prompts are known up front and the caller can block until every result is complete. It accepts a list of tokenized prompts, schedules them internally, and returns completed outputs keyed by request ID. ContinuousBatchingManager is the explicit serving-style interface. It runs a background generation thread, accepts requests over time, and lets callers retrieve results independently, which is important for streaming, real-time serving, and applications where new prompts arrive while older prompts are still decoding.

Sources: docs/source/en/continuous_batching.md, docs/source/en/main_classes/continuous_batching.md

ContinuousBatchingConfig is the configuration object named by the API reference for scheduling and memory limits. The architecture guide specifically connects max_batch_tokens to the token budget for a single forward pass and max_requests to the cap on the number of requests processed in one pass. It also describes a cache budget based on the total number of KV pages that can be read in a pass. Those constraints explain why continuous batching is not simply an unlimited queue: the scheduler must decide what fits before it can admit work.

Sources: docs/source/en/continuous_batching_architecture.md, docs/source/en/main_classes/continuous_batching.md

Request Lifecycle and Scheduler Architecture

The architecture guide defines four lifecycle states. A request starts as pending, meaning it is queued and waiting for the scheduler. It then enters prefilling, where prompt tokens are processed in a forward pass so their key-value cache entries are available for later decoding. After prefill, the request moves to decoding, where generated output is produced one token at a time. Finally, the request becomes finished, meaning generation is complete and the result can be returned to the caller. This lifecycle is the vocabulary used to reason about latency, throughput, and memory.

Sources: docs/source/en/continuous_batching_architecture.md

The scheduler moves a request from pending to prefilling only when there is enough token budget and cache space. If the prompt is too long to fit in one step, the architecture guide describes chunked prefill: the scheduler processes as many prompt tokens as fit, holds the rest, and continues later. This matters because a single very long prompt can otherwise block other requests from decoding. Chunked prefill turns that long prefill into smaller pieces that can interleave with decode work from already active requests, reducing time-to-first-token for other users.

Sources: docs/source/en/continuous_batching_architecture.md

The scheduling decision is governed by two budgets and one cap. The token budget limits the number of query tokens processed in a forward pass. The cache budget limits how many KV pages can be read in one pass and is bounded by total cache size. The request cap limits the number of requests in the pass. Together they form the operational contract for continuous batching: work can be admitted dynamically, but only when it fits both compute and memory constraints. Larger budgets may improve throughput, while tighter budgets reduce memory pressure and prevent long prompts from monopolizing the step.

Sources: docs/source/en/continuous_batching_architecture.md

Python Usage Flow

The user guide’s direct API example loads a causal language model with AutoModelForCausalLM.from_pretrained, uses attn_implementation="flash_attention_2", places the model on CUDA with device_map="cuda", and sets the model dtype to torch.bfloat16. It then loads the matching tokenizer with AutoTokenizer.from_pretrained, encodes several prompts with tokenizer.encode, creates a GenerationConfig with max_new_tokens and eos_token_id, and calls model.generate_batch(inputs=inputs, generation_config=generation_config). Results are decoded by reading each output’s generated tokens and passing them to the tokenizer.

Sources: docs/source/en/continuous_batching.md

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation import ContinuousBatchingConfig, GenerationConfig
 
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-4B",
    attn_implementation="flash_attention_2",
    device_map="cuda",
    dtype=torch.bfloat16,
 )
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B")
inputs = [tokenizer.encode(p) for p in ["Whats up?", "Name a cat breed."]]
generation_config = GenerationConfig(max_new_tokens=64, eos_token_id=tokenizer.eos_token_id)
outputs = model.generate_batch(inputs=inputs, generation_config=generation_config)

For serving-style use, the guide shows model.continuous_batching_context_manager(generation_config=generation_config) and then calls manager.add_request several times with variable-length prompts. One request asks for a detailed history of quantum mechanics with max_new_tokens=512, while shorter requests use max_new_tokens=32. The important behavior is that completed short requests leave the batch while longer requests keep generating. This is the difference between a batch as a fixed group and a batch as a continuously refreshed set of active work.

Sources: docs/source/en/continuous_batching.md

The documentation configuration files show how examples are framed in the built docs. Both the English and Arabic configs define INSTALL_CONTENT with a notebook cell that installs transformers datasets evaluate accelerate, and both include a commented source-install command using git+https://github.com/huggingface/transformers.git. They also set notebook_first_cells and define black_avoid_patterns for placeholder class names. Continuous batching is an inference feature, but these configuration fields explain the standard documentation environment readers encounter when running notebook-style examples.

Sources: docs/source/_config.py, docs/source/ar/_config.py

Compact API Reference

Sources: docs/source/en/main_classes/continuous_batching.md, docs/source/en/continuous_batching_architecture.md, docs/source/_config.py, docs/source/ar/_config.py

The API reference page is intentionally compact because it delegates implementation details to autodoc. It still establishes the public names that readers should search for in generated API docs: ContinuousMixin.generate_batch, ContinuousBatchingManager, and ContinuousBatchingConfig. The guide then supplies the missing usage context by showing how token IDs, GenerationConfig, and result decoding fit together. When writing application code, treat the guide as the task flow and the main-classes page as the reference surface for parameters, return shapes, and manager entry points.

Sources: docs/source/en/continuous_batching.md, docs/source/en/main_classes/continuous_batching.md

Implementation Details and Tradeoffs

Continuous batching improves throughput by reducing idle GPU time, but it does not remove the cost of long prompts or long outputs. Prefill still consumes compute and cache, and decoding still advances one token per request per step. The benefit comes from making the active batch elastic. Finished requests leave immediately, waiting requests enter when capacity exists, and chunked prefill prevents a single long prompt from turning an entire step into blocking prefill work. This is why the architecture guide emphasizes scheduling budgets rather than presenting continuous batching as a simple larger batch size.

Sources: docs/source/en/continuous_batching_architecture.md

The most important edge case is a mixed workload where some users submit short interactive prompts and others submit long prompts or request many new tokens. In static batching, short work can be trapped behind the long item because the whole group finishes together. In the manager example, the short prompts can complete independently while the long quantum-mechanics prompt continues. For latency-sensitive services, that independence is the practical reason to use the manager or the Serve CLI rather than hand-rolling a fixed batch loop around generate.

Sources: docs/source/en/continuous_batching.md, docs/source/en/continuous_batching_architecture.md

Choosing between generate_batch and the manager is mostly a question of request timing. Use generate_batch when the input set is finite and known before the call begins. Use ContinuousBatchingManager when requests should be submitted as they arrive, when results should be consumed as they finish, or when streaming is required. For a production HTTP interface, the guide points to transformers serve, which builds on ContinuousBatchingManager and exposes an OpenAI-compatible endpoint. That recommendation keeps embedded Python orchestration separate from the supported server path.

Sources: docs/source/en/continuous_batching.md

Next Steps

Start with generate_batch if you want the shortest local proof of concept: tokenize prompts, create a GenerationConfig, call the method, and decode output.generated_tokens. Move to continuous_batching_context_manager when you need dynamic submission, streaming, or independent result retrieval. Before tuning ContinuousBatchingConfig, read the architecture guide so the effects of max_batch_tokens, max_requests, cache pressure, chunked prefill, pending requests, prefilling, decoding, and finished states are clear. For deployment, continue to the Serve CLI material referenced by the guide and evaluate continuous batching with the same traffic mix your service will actually receive.

Sources: docs/source/en/continuous_batching.md, docs/source/en/continuous_batching_architecture.md, docs/source/en/main_classes/continuous_batching.md