Multimodal Tasks

Multimodal tasks in Transformers cover workflows where more than one kind of input or output participates in a single model call. The task guides separate these workflows by the modality mix: image and text to text, video and text to text, audio and text to text, visual question answering, and broader any-to-any multimodal generation. The practical reader problem is choosing the correct abstraction before writing inference or fine-tuning code. A vision-language model, a video language model, an audio-language model, and an omni-modal model can all use conversation-style inputs, but they differ in what the processor must prepare and what the model can generate.

Sources: docs/source/en/tasks/image_text_to_text.md, docs/source/en/tasks/video_text_to_text.md, docs/source/en/tasks/audio_text_to_text.md, docs/source/en/tasks/visual_question_answering.md, docs/source/en/tasks/any_to_any.md

Purpose and Scope

Use this page as a map across the multimodal task family rather than as a replacement for each full tutorial. The shared pattern is that raw media and text are packaged into structured messages, a processor turns those messages into tensors, and a model generates or classifies an answer. The important distinction is the output contract. Image-text-to-text, video-text-to-text, and audio-text-to-text guides emphasize text generation from multimodal context. Visual question answering includes both classification-style fine-tuning with ViLT and zero-shot generative inference with models such as BLIP-2. Any-to-any multimodal generation expands the contract so inputs and outputs may span text, images, audio, and video.

The image-text-to-text guide defines vision language models as language models that take image input and can handle general tasks such as visual question answering and segmentation. It also contrasts VLMs with narrower image-to-text models: image-to-text models often accept only images and target a specific task, while VLMs accept open-ended text and image inputs. This matters when designing prompts, because an instruction-tuned VLM can receive a user message containing both an image item and a text question. The guide centers inference with an instruction-tuned checkpoint and uses the auto processor and image-text model class to keep preprocessing aligned with the checkpoint.

Sources: docs/source/en/tasks/image_text_to_text.md

Relevant Source Files

• docs/source/en/tasks/image_text_to_text.md — documents vision language model inference, image-plus-text message structure, model and processor loading, and multi-image conversation patterns.
• docs/source/en/tasks/video_text_to_text.md — documents video language models, how they relate to image-text models, video frame considerations, interleaved inputs, and processor handling for video inference.
• docs/source/en/tasks/audio_text_to_text.md — documents audio-plus-text inputs that generate text, Audio Flamingo 3 usage, AudioCaps fine-tuning, LoRA setup, and audio sampling preparation.
• docs/source/en/tasks/visual_question_answering.md — documents VQA as open-ended image-question answering, ViLT classification fine-tuning, dataset loading, and generative zero-shot alternatives.
• docs/source/en/tasks/any_to_any.md — documents multimodal generation with any-to-any models, omni-modal chat templates, auto multimodal loading, and the any-to-any pipeline.

Core Primitives

The common primitive across these task guides is the processor. A processor is the checkpoint-specific component that knows how to combine text tokenization with media preprocessing. In the image-text guide it is loaded with the selected instruction-tuned VLM and receives messages containing image and text content. In the video guide the processor abstracts a video processor and accepts video-related inference arguments through the chat template call. In the audio guide the processor applies the chat template to a conversation containing a text instruction and an audio path. In any-to-any generation, the processor can structure mixed-modality conversations before the model generates an answer.

Chat templates are the second recurring primitive. The task guides use role-based message lists where each user or assistant turn has a content array. Each content item declares a media or text type, such as image, video, audio, or text. This structure lets a prompt refer to concrete media while preserving the conversational context that instruction-tuned models expect. The image guide explicitly shows alternating user and assistant roles to ground later responses. The any-to-any guide notes that multimodal models typically include chat templates to structure conversations across modalities, which is especially important when a single turn mixes audio and text or other supported inputs.

Sources: docs/source/en/tasks/image_text_to_text.md, docs/source/en/tasks/video_text_to_text.md, docs/source/en/tasks/audio_text_to_text.md, docs/source/en/tasks/any_to_any.md

System-to-Code Mapping

The video-text-to-text guide is the clearest example of why task selection matters. It explains that video data is essentially image frames with temporal dependencies, but that simply giving many images to an image-text model is not equivalent to video understanding. Video models are trained with vision modalities that may include videos, multiple videos, images, multiple images, and interleaved inputs. The guide also states that these models process videos with no audio, while any-to-any models can process videos that include audio. That distinction should guide model choice whenever the answer depends on sound, speech, or synchronized audiovisual events.

Sources: docs/source/en/tasks/video_text_to_text.md, docs/source/en/tasks/any_to_any.md

Execution Flow

A typical multimodal inference flow starts by installing the packages required by the selected task. The image-text guide installs Transformers and Accelerate, with FlashAttention as an additional performance dependency. The video guide adds TorchCodec for video handling. The any-to-any guide uses Transformers, Accelerate, and FlashAttention, while the audio fine-tuning guide adds datasets and PEFT for training. After dependencies are present, load the model checkpoint and its processor together. Keeping the processor and model from the same checkpoint is essential because the processor encodes the modality-specific conventions expected by that model.

pip install -q transformers accelerate flash_attn

from transformers import AutoProcessor, AutoModelForMultimodalLM, infer_device
import torch
 
device = torch.device(infer_device())
model = AutoModelForMultimodalLM.from_pretrained(
    "Qwen/Qwen2.5-Omni-3B",
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
).to(device)
processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-Omni-3B")

After loading, construct a message list that represents the user request. For image, audio, and any-to-any tasks, the examples use a user role whose content array contains one or more media items and a text instruction. The processor call usually tokenizes the chat template, returns tensors, and may add a generation prompt. The resulting dictionary is passed to the model generation method. The guides then decode only the generated portion, or decode the full generated ids depending on the example. This is the central workflow to reuse when moving from a tutorial checkpoint to a production checkpoint with the same task interface.

messages = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "url": "https://huggingface.co/datasets/raushan-testing-hf/audio-test/resolve/main/f2641_0_throatclearing.wav"},
            {"type": "text", "text": "What do you hear in this audio?"},
        ],
    },
]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
 )

Sources: docs/source/en/tasks/any_to_any.md, docs/source/en/tasks/audio_text_to_text.md, docs/source/en/tasks/image_text_to_text.md

Task-Specific Guidance

For visual question answering, decide early whether you need supervised classification or open-ended generation. The VQA guide defines the task as answering open-ended questions based on an image, but its ViLT section treats the training example as a classification problem because a classifier head is placed on the final hidden state of the classification token. That tutorial fine-tunes on a small sample of the Graphcore VQA dataset and encourages Hub login for sharing the model. By contrast, more recent BLIP-family and InstructBLIP-style models can answer VQA prompts in a zero-shot generative manner, which is often faster to prototype but less controlled for fixed-label evaluation.

Audio-text-to-text workflows sit between speech recognition and general language reasoning. The audio guide explicitly distinguishes these models from traditional automatic speech recognition systems: they can transcribe, but they can also reason about audio content, infer emotion, follow instructions, and produce contextual responses. Its example uses Audio Flamingo 3 with a conversation containing a text instruction and an audio path, then decodes the generated text. The same page also moves into fine-tuning on AudioCaps for captioning with LoRA, which means this task guide is useful both for simple inference and for parameter-efficient adaptation to domain-specific sound descriptions.

Any-to-any multimodal generation is the broadest workflow and should be chosen when a fixed input-output task is too narrow. The guide describes omni-modal models that can accept combinations of text, images, audio, and video and produce outputs in different modalities, depending on the model configuration. It also exposes the fastest route through the pipeline API by specifying the any-to-any task. Prefer the pipeline for quick experiments and demos, then move to the model and processor APIs when you need explicit device placement, attention implementation choices, decoding control, custom batching, or access to intermediate preprocessing results.

Sources: docs/source/en/tasks/visual_question_answering.md, docs/source/en/tasks/audio_text_to_text.md, docs/source/en/tasks/any_to_any.md

Practical Selection Checklist

Choose image-text-to-text when the main context is still imagery plus instructions and the desired answer is text. Choose video-text-to-text when temporal ordering matters, such as describing an action or comparing two clips, and remember that the documented video models do not use audio. Choose audio-text-to-text when speech, sound events, or acoustic reasoning are the primary evidence and the response should be textual. Choose VQA when your product interaction is specifically question answering over images, especially if you need a supervised classification tutorial. Choose any-to-any when the application needs flexible modality combinations or output modalities beyond generated text.

As next steps, read the processor and chat-template documentation before implementing a custom input layer, then open the task-specific guide that matches your modality mix. If the first prototype uses a pipeline, verify later that the lower-level model and processor call produces the same message formatting and tensor keys. For fine-tuning, start from the VQA or audio captioning guides because they include dataset loading and sharing patterns. For inference, start with the image-text, video-text, or any-to-any examples and adapt only the checkpoint, message content, and generation settings needed for your application.

Sources: docs/source/en/tasks/image_text_to_text.md, docs/source/en/tasks/video_text_to_text.md, docs/source/en/tasks/audio_text_to_text.md, docs/source/en/tasks/visual_question_answering.md, docs/source/en/tasks/any_to_any.md