OpenWikihuggingface/transformers

Vision Tasks

Purpose and Scope

Transformers treats computer vision as a set of task workflows built around the same project conventions used elsewhere in the library: load a dataset, choose an Auto class or pipeline, preprocess raw inputs into tensors, fine-tune when needed, evaluate, and run inference with a saved checkpoint. The vision task docs cover image classification, object detection, semantic segmentation, and image captioning as separate guides because their labels and outputs differ substantially. Classification predicts one class for an image, detection predicts labeled boxes, segmentation predicts pixel-level masks, and captioning turns an image into generated text.

Sources: docs/source/en/tasks/image_classification.md, docs/source/en/tasks/object_detection.md, docs/source/en/tasks/semantic_segmentation.md, docs/source/en/tasks/image_captioning.md

These workflows are useful when you already know the kind of visual output your application needs but want to understand the corresponding Transformers entry points. The task guides emphasize end-to-end fine-tuning and inference, not only model architecture descriptions. They show how datasets from the Hugging Face Hub, processors, image processors, Trainer-based training, and task-specific postprocessing fit together. The example READMEs add a more script-oriented path for teams that want repeatable command-line training runs rather than notebook-style guide code.

Sources: docs/source/en/tasks/image_classification.md, examples/pytorch/image-classification/README.md, examples/pytorch/object-detection/README.md

Relevant Source Files

  • docs/source/en/tasks/image_classification.md - Official task guide for assigning one label to an image, using Food-101, ViT, image preprocessing, fine-tuning, and inference.
  • docs/source/en/tasks/object_detection.md - Official task guide for detecting multiple objects with bounding boxes and labels, using RF-DETR and a mobile UI dataset.
  • docs/source/en/tasks/semantic_segmentation.md - Official task guide for segmentation concepts and semantic segmentation fine-tuning, including pipeline inference that returns masks per predicted class.
  • docs/source/en/tasks/image_captioning.md - Official task guide for image-to-text captioning, using paired image and caption data plus an AutoProcessor for multimodal preprocessing.
  • examples/pytorch/image-classification/README.md - PyTorch example documentation for Trainer and no-Trainer image classification scripts based on AutoModelForImageClassification.
  • examples/pytorch/object-detection/README.md - PyTorch example documentation for Trainer and no-Trainer object detection scripts based on AutoModelForObjectDetection.

Core Primitives

The first shared primitive is a vision dataset with fields that match the task. Image classification examples use image and label fields, then build id-to-label and label-to-id mappings so the model configuration can translate between class names and integer labels. Object detection examples require richer annotations: every image may contain multiple objects, each with a category and bounding box, and the guide normalizes those annotations before training. Segmentation uses masks or per-pixel targets, while captioning uses image and text pairs. The data shape determines the model head, metrics, and postprocessing.

Sources: docs/source/en/tasks/image_classification.md, docs/source/en/tasks/object_detection.md, docs/source/en/tasks/semantic_segmentation.md, docs/source/en/tasks/image_captioning.md

The second primitive is preprocessing. Classification uses AutoImageProcessor to load the image processing rules associated with a checkpoint such as a ViT model, then applies image transforms such as cropping, resizing, tensor conversion, and normalization. Captioning uses AutoProcessor because it must handle both image preprocessing and text tokenization for captions. Detection and segmentation have their own target formatting requirements, so preprocessing is not just cosmetic augmentation; it is the place where raw image data and annotations become the structure that the model and Trainer expect.

Sources: docs/source/en/tasks/image_classification.md, docs/source/en/tasks/image_captioning.md, docs/source/en/tasks/object_detection.md

The third primitive is a task-specific model API. The image classification examples explicitly target AutoModelForImageClassification, which covers ViT, ConvNeXT, ResNet, Swin Transformer, and other supported backbones through a shared loading contract. The object detection examples target AutoModelForObjectDetection, covering DETR-family models and related architectures. Captioning may use multimodal processors and image-to-text architectures, while segmentation can be explored quickly through the image-segmentation pipeline before moving into fine-tuning. This division lets users choose a task first and swap compatible checkpoints later.

Sources: examples/pytorch/image-classification/README.md, examples/pytorch/object-detection/README.md, docs/source/en/tasks/semantic_segmentation.md, docs/source/en/tasks/image_captioning.md

Task-to-Workflow Mapping

For image classification, the guide starts with Food-101 and describes the task as assigning a label or class to an image based on pixel values. The recommended flow is to install Transformers, Datasets, Evaluate, Accelerate, Pillow, torchvision, scikit-learn, and Trackio; optionally authenticate with the Hub; load a subset of the dataset; split it; inspect the image and label fields; build label mappings; load an AutoImageProcessor; then train a ViT-style classifier. The PyTorch example README turns the same pattern into a reusable command-line script with dataset, output, evaluation, saving, and Hub upload flags.

Sources: docs/source/en/tasks/image_classification.md, examples/pytorch/image-classification/README.md

For object detection, the output is not a single class but a set of labeled bounding boxes. The official guide uses RF-DETR on a mobile app screenshot dataset and calls out that annotations include UI element categories such as group, image, rectangle, and text. It also notes that the dataset uses COCO-style boxes with coordinates and dimensions, so examples must convert category names to ids, compute areas, and filter invalid boxes. The PyTorch README gives a DETR command where evaluation behavior, image size, mixed precision, and mismatched head loading are explicit runtime concerns.

Sources: docs/source/en/tasks/object_detection.md, examples/pytorch/object-detection/README.md

For semantic segmentation, the output is dense rather than box-based: the model assigns labels at pixel level. The guide distinguishes semantic, instance, and panoptic segmentation, then demonstrates fast inference through the image segmentation pipeline using a SegFormer checkpoint. The pipeline returns a list of predicted classes with masks, which is a different interface from classification logits or detection boxes. This makes segmentation a good example of why downstream code must read task outputs carefully instead of assuming every vision model returns the same structure.

Sources: docs/source/en/tasks/semantic_segmentation.md

For image captioning, the task becomes multimodal generation. The guide uses a dataset of image-caption pairs and explains that many captioning datasets can contain multiple captions per image, with random caption sampling as a common training strategy. Preprocessing is deliberately different from the pure vision tasks because the processor prepares both image pixels and caption text. Official model documentation for vision-encoder-decoder systems also frames captioning as an encoder-decoder pattern: a vision encoder represents the image and an autoregressive decoder generates the caption.

Sources: docs/source/en/tasks/image_captioning.md

Scripted Training Patterns

The example READMEs are the bridge from tutorial concepts to repeatable jobs. Image classification documents two PyTorch scripts, one using Trainer and one without Trainer, and the Trainer path shows how to run on a Hub dataset such as beans with train and eval enabled, column retention adjusted, batch sizes set, best-model loading enabled, and Hub upload configured. It also documents a practical edge case: if the classification head shape does not match the dataset labels, the ignore-mismatched-sizes option can adapt the checkpoint for fine-tuning.

Sources: examples/pytorch/image-classification/README.md

Object detection scripting has a similar structure but different operational pitfalls. The README highlights that the Trainer-based script runs in distributed environments and shows a DETR fine-tuning command on CPPE-5. It explicitly recommends disabling concatenation of evaluation batches for correct detection evaluation and enabling mismatched-size loading when changing the number of classes. It also points readers toward tuning image size, learning rate, batch size, warmup, optimizer choice, and augmentations if model quality is insufficient, which is especially important for detection where annotation geometry and image scale influence metrics.

Sources: examples/pytorch/object-detection/README.md

python run_image_classification.py --dataset_name beans --output_dir ./beans_outputs/ --remove_unused_columns False --label_column_name labels --do_train --do_eval --push_to_hub --learning_rate 2e-5 --num_train_epochs 5 --per_device_train_batch_size 8
python run_object_detection.py --model_name_or_path facebook/detr-resnet-50 --dataset_name cppe-5 --do_train true --do_eval true --output_dir detr-finetuned-cppe-5-10k-steps --remove_unused_columns false --eval_do_concat_batches false --ignore_mismatched_sizes true

API and Configuration Reference

Use this compact reference to choose the correct abstraction before opening a task guide or script. For image classification, prefer AutoImageProcessor plus AutoModelForImageClassification when training a classifier, and keep label mappings aligned with the dataset. For object detection, use AutoModelForObjectDetection-compatible checkpoints and preserve annotation fields needed for boxes, categories, and areas. For segmentation inference, the image-segmentation pipeline returns masks per predicted label, while fine-tuning guides require pixel-level targets. For captioning, use AutoProcessor when both image and text preprocessing are required by the checkpoint.

Sources: docs/source/en/tasks/image_classification.md, docs/source/en/tasks/object_detection.md, docs/source/en/tasks/semantic_segmentation.md, docs/source/en/tasks/image_captioning.md, examples/pytorch/image-classification/README.md, examples/pytorch/object-detection/README.md

  • Image classification task output: one class label per image.
  • Object detection task output: one or more labeled bounding boxes per image.
  • Semantic segmentation task output: predicted class masks or pixel labels.
  • Image captioning task output: generated text conditioned on image inputs.
  • Classification script API: AutoModelForImageClassification-compatible checkpoints and Trainer arguments.
  • Detection script API: AutoModelForObjectDetection-compatible checkpoints plus detection-specific evaluation settings.

Practical Next Steps

Start with the highest-level guide that matches your desired output, then move to the PyTorch example README when you need a reproducible training command. If you are evaluating a checkpoint quickly, segmentation and other vision tasks may be accessible through pipelines, but fine-tuning generally requires careful dataset formatting and preprocessing. When changing datasets, revisit label mappings, annotation formats, and model head dimensions before training. For broader context, read the preprocessing, processors, pipelines, Trainer, fine-tuning, and examples pages so the vision-specific instructions connect to the common Transformers workflow.

Ask a question...