Tool Use and Response Parsing

Purpose and Scope

This page explains the chat-layer features that sit between a model's raw token stream and the developer-facing message dictionaries used by Transformers. In chat workflows, applications usually work with messages that have roles such as user, assistant, system, and tool. The model, however, still sees and emits one sequence of tokens. Chat templates convert structured messages into model-specific tokens, while response templates convert generated text back into structured message objects. Understanding both directions matters when a chat model can call tools, return reasoning fields, or mix text with images, video, or audio.

Sources: docs/source/en/chat_response_parsing.md, docs/source/en/chat_content_patterns.md, docs/source/en/chat_extras.md, docs/source/en/chat_templating_multimodal.md, docs/source/en/chat_templating_writing.md

Tool use is the feature sometimes called function calling. A tool is a user-supplied function, or a JSON schema describing such a function, that a model may choose to call instead of answering from its parameters alone. Response parsing is the complementary feature for interpreting generated outputs that contain more than a plain assistant reply. A reasoning model may emit a thinking trace, and a tool-calling model may emit function names and arguments. Transformers documents these as message patterns and template-driven glue rather than as a separate protocol outside the tokenizer or processor APIs.

Sources: docs/source/en/chat_extras.md, docs/source/en/chat_response_parsing.md

Relevant Source Files

• docs/source/en/chat_response_parsing.md - Defines response templates, explains why generated LLM outputs need parsing, and introduces PreTrainedTokenizerBase.parse_response for turning generated tokens or text back into structured assistant messages.
• docs/source/ar/chat_templating.md - Localized chat templating documentation that reinforces the core concept: conversations are lists of role/content messages that apply_chat_template formats into the prompt expected by a particular checkpoint.
• docs/source/en/chat_content_patterns.md - Documents the supported chat message dictionary patterns for text, tools, multimodal content, batches, and multi-turn conversations.
• docs/source/en/chat_extras.md - Provides the tool-use guide, including tool definitions, passing tools to apply_chat_template, Google-style docstring parsing, and a tool-calling example.
• docs/source/en/chat_templating_multimodal.md - Explains multimodal chat histories, ProcessorMixin.apply_chat_template, ImageTextToTextPipeline chat mode, and mixed image/text content blocks.
• docs/source/en/chat_templating_writing.md - Describes how chat templates are written in Jinja, where templates are stored, how they are saved, and how multimodal templates should emit media placeholder tokens.

Core Message Primitives

The shared primitive is a conversation represented as a list of dictionaries. Each dictionary has a role and usually a content field. For text-only models, content can be a single string. For code that may later add images, audio, or video, the explicit block form is safer: content becomes a list of typed items, such as a text item with type text and text content. The documentation treats this explicit modality format as the unified representation because it lets text, tools, and media be combined without changing the outer message shape.

Sources: docs/source/en/chat_content_patterns.md, docs/source/en/chat_templating_multimodal.md

Tool calls extend the same role-based structure. The assistant message can contain a tool_calls field whose entries describe a function call, commonly with type function and a function object containing the tool name and arguments. The tool role then carries the result back to the model, and the documentation states that this result content should be a string. This is important for application loops: the assistant requests work, your code executes the selected function, and the tool result is appended as another conversational turn before generation continues.

Sources: docs/source/en/chat_content_patterns.md, docs/source/en/chat_extras.md

weather = {"name": "get_current_temperature", "arguments": {"location": "Paris, France", "unit": "celsius"}}
messages.append({"role": "assistant", "tool_calls": [{"type": "function", "function": weather}]})
messages.append({"role": "tool", "content": "22"})

Multimodal chat keeps the same message-list abstraction but changes the content field from a string to a list of typed media and text blocks. Image, video, and audio items specify their type and provide media by URL or local path. The multimodal template guide explains that processors, not tokenizers alone, handle preprocessing, tokenization, and chat template application for these models. This means a tool-aware application may need to choose between tokenizer-based and processor-based entry points depending on whether the selected checkpoint accepts only text or also accepts media.

Sources: docs/source/en/chat_content_patterns.md, docs/source/en/chat_templating_multimodal.md

Tool Definition and Prompt Formatting Flow

For models that support tool use, the prompt-formatting entry point is apply_chat_template with a tools argument. The tool list can contain JSON schemas or Python functions. When Python functions are supplied, Transformers parses the function name, argument names, argument types, and Google-style docstring to generate the schema presented to the model. The body of the Python function is not what the model sees; the useful contract is the signature and natural-language description. That distinction helps avoid assuming that registering a function executes it automatically during generation.

Sources: docs/source/en/chat_extras.md

def get_current_temperature(location: str, unit: str):
    """
    Get the current temperature at a location.
 
    Args:
        location: The location to get the temperature for, in the format "City, Country"
        unit: The unit to return the temperature in. (choices: ["celsius", "fahrenheit"])
    """
    return 22.
 
tools = [get_current_temperature]

A practical tool loop has four phases. First, prepare the chat history with system and user messages. Second, pass messages and tools into the chat template so the model receives the function signatures in the format it was trained to understand. Third, generate the assistant turn and inspect whether it contains a tool call. Fourth, execute the function in application code and append a tool role message with the string result. The model can then be called again with the augmented conversation, now able to use the tool output in a final answer.

Sources: docs/source/en/chat_extras.md, docs/source/en/chat_content_patterns.md

Chat templates are model-specific, which is why the documentation emphasizes not manually constructing control tokens. The Arabic chat templating page uses BlenderBot, Mistral Instruct, and Zephyr examples to show that models trained from similar bases may expect very different control-token layouts. The writing guide also shows that a template is a Jinja template stored on the tokenizer's chat_template attribute and used by apply_chat_template. Once set, the template is saved with the tokenizer, including as chat_template.jinja in the tokenizer directory.

Sources: docs/source/ar/chat_templating.md, docs/source/en/chat_templating_writing.md

Response Parsing Flow

Response parsing solves the reverse problem. A chat API wants a structured assistant dictionary, but the model produced raw tokens. The response parsing guide describes response templates as the inverse of chat templates: where a chat template maps messages into tokens, a response template maps generated output back into fields such as role, content, thinking, or tool-call structures. The main entry point documented for this workflow is PreTrainedTokenizerBase.parse_response, which accepts either a single sequence or a batch and returns structured message data when the tokenizer has a response_template.

Sources: docs/source/en/chat_response_parsing.md

The documented generation pattern is to call apply_chat_template with add_generation_prompt enabled, run model.generate, remove the prompt prefix from the generated tensor, decode the generated tokens, and then call parse_response. The prefix remains important even though parsing focuses on the final assistant message. Some chat templates start assistant messages or open thinking blocks before the model begins generating; without the prefix, the parser can miss prefilled fields and silently misinterpret the response. The guide therefore requires prefix when parsing with newer response-template behavior.

Sources: docs/source/en/chat_response_parsing.md

messages = [{"role": "user", "content": "Summarize the end of the Cold War, very briefly."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
input_ids = inputs["input_ids"].to(model.device)
outputs = model.generate(input_ids, max_new_tokens=1024)[0, input_ids.shape[1]:]
out_text = tokenizer.decode(outputs)
parsed = tokenizer.parse_response(out_text, prefix=input_ids[0])

When using response parsing in production code, treat it as part of the model contract rather than a generic JSON parser. A checkpoint must have a response_template that matches how its chat template and training format represent assistant fields. Parsing one message at a time also means the application should preserve conversation state explicitly: append the parsed assistant message, append any tool role results, and then render the updated list again for the next model call. This keeps the universal message API aligned with the model-specific token format.

Sources: docs/source/en/chat_response_parsing.md, docs/source/en/chat_templating_writing.md

Multimodal and Template Authoring Details

Multimodal tool and response workflows require an additional processor layer. The multimodal guide states that text-only chat templates live on tokenizers, while multimodal chat templates are handled by Processor classes through ProcessorMixin.apply_chat_template. The ImageTextToTextPipeline can enable chat mode when the prompt is properly formatted for a conversational model, and lower-level code can call the processor template and then GenerationMixin.generate. This is the same overall pattern as text generation, but media preprocessing must happen before the final tensors reach the model.

Sources: docs/source/en/chat_templating_multimodal.md

For authors writing templates, the Jinja template should usually emit control text and special placeholder tokens rather than directly processing raw media. The writing guide recommends that multimodal templates detect content item types and emit model-specific placeholders such as an image or video token. The processor later expands those placeholders into the required media token sequence. This separation keeps template rendering focused on conversation structure and leaves image, video, and audio tensor preparation to the processor implementation selected by AutoProcessor or a model-specific processor class.

Sources: docs/source/en/chat_templating_writing.md, docs/source/en/chat_templating_multimodal.md

The same caution applies to tools. A tool-aware template can be much more complex than a simple role/content loop because it may need to render tool schemas, assistant tool calls, tool results, and optional generation prompts. The writing guide recommends inspecting existing templates with print(tokenizer.chat_template) and beginning with simpler models before editing tool-use or retrieval-augmented templates. That advice is operationally important: a small mismatch in delimiters, roles, or generation prompts can reduce model quality even when the Python message dictionaries look correct.

Sources: docs/source/en/chat_templating_writing.md, docs/source/en/chat_extras.md

Compact API and Data Reference

Sources: docs/source/en/chat_response_parsing.md, docs/source/en/chat_content_patterns.md, docs/source/en/chat_extras.md, docs/source/en/chat_templating_multimodal.md

Use this reference to decide where a bug belongs. If the model receives the wrong prompt, inspect the chat_template or processor template. If a tool is never selected, verify the schema derived from the Python function name, annotations, and docstring. If generated output is hard to consume, check that the tokenizer has a matching response_template and that parse_response receives the prefix. If multimodal messages fail, check that content blocks use explicit types and that the processor, not a text-only tokenizer path, is responsible for template application.

Sources: docs/source/en/chat_response_parsing.md, docs/source/en/chat_content_patterns.md, docs/source/en/chat_extras.md, docs/source/en/chat_templating_multimodal.md

Next Steps

Start with the plain message patterns before adding tools or response parsing. Build the conversation as role/content dictionaries, render it with apply_chat_template, and verify the decoded prompt for one small example. Then register one simple Google-style Python function as a tool and trace the assistant tool call plus tool result turn. After that, add parse_response only for a tokenizer with a response_template and always pass the prompt prefix. For multimodal checkpoints, switch the same mental model to AutoProcessor and ProcessorMixin.apply_chat_template so media blocks are prepared by the correct component.

Sources: docs/source/en/chat_response_parsing.md, docs/source/en/chat_content_patterns.md, docs/source/en/chat_extras.md, docs/source/en/chat_templating_multimodal.md, docs/source/en/chat_templating_writing.md, docs/source/ar/chat_templating.md