Task Overview

Purpose and Scope

Transformers supports task workflows that span text, vision, audio, video, and combinations of those modalities. A task is the user-facing problem shape, such as text generation, image captioning, audio classification, visual question answering, or video question answering. The task overview helps readers decide whether they should start with a high-level pipeline, an Auto class plus processor, a fine-tuning example, or an inference-optimization path. In current documentation, many modern workflows are multimodal: image-text models can answer questions about images, audio-text models can reason over sound with a text prompt, and video-text models process visual frames and generate text responses.

The supplied repository sources for this page focus on the kernels layer, which is not a task by itself but is an important execution layer underneath many tasks. Kernels are optimized implementations of operations such as attention, normalization, matrix multiplication, and fused model layers. They matter when a task becomes compute-bound, for example long-context language generation, large multimodal generation, or high-throughput inference. This page therefore maps tasks first, then explains how the kernels documentation connects those tasks to practical performance decisions. Sources: docs/source/en/kernel_doc/overview.md, docs/source/en/kernels.md

Supported Task Families

Text tasks include classification, token classification, question answering, summarization, translation, and language modeling. Inference can often start from a task pipeline, while training usually moves toward a task-specific script or the Trainer API. Generation-centered text tasks use causal or sequence-to-sequence models and depend heavily on decoding configuration, KV cache behavior, attention implementation, and model loading choices. When the workload is dominated by attention or normalization, kernelized execution can reduce overhead without changing the task semantics.

Vision tasks cover image classification, object detection, segmentation, depth estimation, image captioning, and image-to-text reasoning. Some workflows use specialized vision models, while others use vision-language models that accept an image, image sequence, or interleaved prompt and produce text. The official IDEFICS task guide frames this second style as a way to use one large multimodal model for captioning, visual question answering, image classification, and story generation grounded in images. The practical choice is between smaller specialized models and larger general-purpose models that demand more memory and throughput planning.

Audio tasks include audio classification, automatic speech recognition, speech pretraining, text-to-speech, and newer audio-text-to-text workflows. Traditional ASR maps speech to transcription, while audio-text-to-text models accept audio plus a prompt and generate a contextual text answer. That distinction matters for API selection: a classifier or ASR model may use a feature extractor and task head, while an audio-language model commonly uses an AutoProcessor-style interface with chat-template formatting and generate(). Performance choices become more important as audio context, batch size, and model size increase.

Video and multimodal tasks combine multiple input forms. Video-text-to-text models process video frames and generate text, but they generally do not include audio unless the model is explicitly any-to-any. Image-text and video-text models often share a language-model-style generation loop, with modality-specific preprocessing handled by processors. This is why task selection should consider both the front-end data contract and the back-end generation or training path: the same visible task label can involve very different processor, model, cache, and kernel behavior.

Performance Layer for Task Workflows

The kernels documentation describes optimized kernels as drop-in replacements for standard PyTorch operations. When use_kernels=True is passed to PreTrainedModel.from_pretrained, Transformers identifies supported layers, loads available kernel implementations, and falls back to PyTorch when no implementation exists. This means task code can remain organized around models, processors, and generation APIs while selected operations run through device-specific implementations. The fallback behavior is important for broad task coverage because not every architecture, operation, or hardware target has a kernel available. Sources: docs/source/en/kernel_doc/loading_kernels.md, docs/source/en/kernel_doc/overview.md

Kernels are distributed through the Hub and selected according to the execution platform. The overview documentation lists NVIDIA CUDA, AMD ROCm, Apple Silicon Metal, and Intel XPU as supported platform families for precompiled kernel distribution. The loading guide also documents operation-level defaults such as RMSNorm, MoE MLP, SwiGLU or GeGLU MLPs, linear layers, activations, rotary embeddings, causal language-model loss, and deformable attention. For task users, this means the acceleration story is operation-oriented rather than task-oriented: a text-generation model and a multimodal model can both benefit if they use supported operations. Sources: docs/source/en/kernel_doc/overview.md, docs/source/en/kernel_doc/loading_kernels.md

Attention kernels are selected separately through attn_implementation, which is especially relevant for generation, video-language, and long-context workloads. The loading guide shows attn_implementation="kernels-community/flash-attn2" as a Hub-backed attention implementation. It also documents an explicit safety boundary: attention kernels outside the trusted kernels-community namespace require allow_all_kernels=True, because kernel loading can execute code on the host machine. That security model should be part of any production task review, alongside model trust, checkpoint provenance, and deployment isolation. Sources: docs/source/en/kernel_doc/loading_kernels.md

System-to-Code Mapping

The public API surface documented for kernels is compact. The main classes page lists kernelize and KernelConfig, while the conceptual kernels page shows KernelConfig(kernel_mapping={"RMSNorm": "kernels-community/rmsnorm"}) passed into AutoModelForCausalLM.from_pretrained. For users who are only choosing a task, this is usually a later-stage optimization. For maintainers and advanced users, it is the bridge between model architecture and optimized runtime behavior. Sources: docs/source/en/main_classes/kernels.md, docs/source/en/kernels.md

Execution Flow

A practical task workflow starts by identifying the modality and output type. For text classification or audio classification, choose a classification checkpoint and preprocess examples into tensors. For image-text, audio-text, or video-text generation, choose a model family that accepts the relevant modality through its processor and then call generate() for text output. Once the model is functionally correct, evaluate latency, memory use, and reproducibility. Only then introduce kernel options, because kernels should preserve the model-level contract while changing selected internal operations.

The normal kernel loading path is intentionally simple. Install the kernels package, load a model with use_kernels=True, and let Transformers pick the most performant available kernels for the detected device. If a required operation has no registered default, the model continues through PyTorch. If a team needs a specific implementation, it can provide a KernelConfig with a module-to-kernel mapping. If attention is the bottleneck, it can choose a Hub attention backend with attn_implementation. Sources: docs/source/en/kernel_doc/loading_kernels.md, docs/source/en/kernels.md

from transformers import AutoModelForCausalLM, KernelConfig
 
kernel_config = KernelConfig(
    kernel_mapping={
        "RMSNorm": "kernels-community/rmsnorm",
    }
)
 
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B",
    use_kernels=True,
    kernel_config=kernel_config,
    device_map="cuda",
)

For training-oriented task workflows, the kernels page documents a separate Liger integration. Installing liger-kernel and setting use_liger_kernel=True in TrainingArguments patches supported model layers with Liger kernels. The documented liger_kernel_config dictionary can enable or disable options such as rope, swiglu, cross_entropy, fused_linear_cross_entropy, and rms_norm, depending on model support. This makes the task overview actionable for fine-tuning: first choose the task and model, then choose Trainer settings, and finally add kernel patching when the model family supports it. Sources: docs/source/en/kernels.md, docs/source/zh/kernels.md

API Components

KernelConfig is the main user-facing configuration object for Hub kernels. It accepts a kernel mapping where keys refer to original module class names in the model and values identify the kernel implementation repository and, when needed, class name. The writing guide shows an advanced mapping such as {"RMSNorm": "owner/my-kernel:CustomRMSNorm"}. That example demonstrates that kernel selection is attached to model internals, not to top-level tasks, so the same task may or may not support a given kernel depending on its architecture. Sources: docs/source/en/kernel_doc/writing_kernels.md, docs/source/en/main_classes/kernels.md

The advanced authoring API supports parameter transformation and module fusion. A stateful kernel follows a strict two-class pattern: KernelName contains the forward pass, while KernelNameLayout is an nn.Module that holds parameters and patches the original module before checkpoint loading. The layout class may declare conversion_mapping to remap checkpoint keys when the optimized implementation expects different parameter names or shapes. This is relevant to task support because it allows optimized kernels to adapt to existing checkpoints rather than requiring task users to retrain models from scratch. Sources: docs/source/en/kernel_doc/writing_kernels.md

Relevant Source Files

• docs/source/en/kernel_doc/loading_kernels.md - Explains how use_kernels=True, KernelConfig, attn_implementation, device defaults, trusted kernel repositories, and PyTorch fallback behavior work during model loading.
• docs/source/en/kernel_doc/writing_kernels.md - Documents the advanced two-class kernel authoring pattern, parameter transformation, conversion_mapping, and module fusion for kernels that need state or layout changes.
• docs/source/en/kernels.md - Provides the main English kernels guide, including Hub kernel loading, Liger integration through TrainingArguments, and configurable liger_kernel_config options.
• docs/source/en/main_classes/kernels.md - Defines the kernels API reference surface by documenting kernelize and KernelConfig.
• docs/source/zh/kernels.md - Mirrors the kernels guide in Chinese, confirming the same Hub kernel and Liger concepts for localized documentation readers.
• docs/source/en/kernel_doc/overview.md - Introduces the platform problem that kernels solve, supported hardware families, runtime detection, caching, fallback behavior, and determinism considerations.

Next Steps

Use this overview to choose the task family first, then move to the page that matches the workflow. Read task-specific guides for text, vision, audio, video, or multimodal examples when you need data formats and model classes. Read the generation, processor, Trainer, and quantization pages when the workflow becomes implementation-specific. If the task already works but performance is the blocker, continue with the kernels, attention backend, Liger, and inference-optimization documentation; those pages explain how to speed up supported operations without changing the high-level task contract.