OpenWikihuggingface/transformers

Audio Tasks

Purpose and Scope

Audio tasks in Transformers cover workflows where sound is either the model input, the model output, or both. The repository documentation separates the most common developer paths into audio classification, automatic speech recognition, and text to speech. Audio classification assigns labels to raw waveforms, automatic speech recognition converts speech signals into text, and text to speech creates natural sounding speech from text prompts. These tasks share a practical pattern: load audio-capable dependencies, prepare datasets with waveform arrays and sampling rates, choose an audio model family, then use either a high-level pipeline or a task-specific training script.

Sources: docs/source/en/tasks/audio_classification.md, docs/source/en/tasks/asr.md, docs/source/en/tasks/text-to-speech.md

The task guides are written for readers who want a complete fine-tuning and inference path rather than only API signatures. Audio classification and ASR both use the MInDS-14 dataset in the docs, but they focus on different fields: classification keeps the waveform and intent label, while speech recognition keeps the waveform and transcription. This distinction matters because it changes the model head, the metric, the preprocessing target, and the output interpretation. Text to speech inverts the direction of the workflow by accepting text or conversational content and returning generated audio arrays plus a sampling rate.

Sources: docs/source/en/tasks/audio_classification.md, docs/source/en/tasks/asr.md, docs/source/en/tasks/text-to-speech.md

Relevant Source Files

  • docs/source/en/tasks/audio_classification.md - User-facing task guide for fine-tuning Wav2Vec2 on MInDS-14 intent classification, installing audio dependencies, logging in for Hub upload, loading the dataset, and using audio and label fields.
  • docs/source/en/tasks/asr.md - User-facing task guide for automatic speech recognition with Wav2Vec2 on MInDS-14, including the speech-to-text framing, dependency list, dataset split, and transcription-focused preprocessing.
  • docs/source/en/tasks/text-to-speech.md - User-facing guide for text-to-speech inference with the text-to-audio pipeline, CSM and Dia examples, voice cloning through chat-style content, and notes on fine-tunable TTS model families.
  • examples/pytorch/audio-classification/README.md - Runnable PyTorch examples for fine-tuning Wav2Vec2-style models on audio classification datasets, including single-GPU and multi-GPU commands, Trainer flags, expected accuracy, and Hub sharing guidance.
  • examples/pytorch/speech-recognition/README.md - Runnable PyTorch examples for ASR with CTC, CTC adapters, and sequence-to-sequence models, including preprocessing behavior, vocabulary creation, resampling, normalization, padding, and a multiprocessing troubleshooting note.

Core Audio Workflows

For audio classification, the input is a one dimensional speech or sound waveform and the output is a class id or class label. The docs describe examples such as speaker intent, language classification, and animal sound recognition, then demonstrate intent classification with Wav2Vec2 on MInDS-14. The dataset includes metadata such as paths, transcriptions, language identifiers, audio arrays, and intent classes, but the guide narrows the supervised signal to the audio input and intent class. That narrowing is an important modeling decision because unused columns should not silently drive the task or confuse the training batch.

Sources: docs/source/en/tasks/audio_classification.md

For automatic speech recognition, the input is again a waveform, but the output is a text sequence. The ASR guide frames the task as mapping a sequence of audio inputs to text outputs, with virtual assistants, live captions, and meeting notes as representative applications. In the MInDS-14 walkthrough, the guide loads a smaller subset, splits it for training and testing, and removes columns unrelated to the transcription target. Compared with classification, ASR preparation must preserve text labels and account for speech-specific processing such as resampling, normalization, padding, and, in CTC workflows, vocabulary construction.

Sources: docs/source/en/tasks/asr.md, examples/pytorch/speech-recognition/README.md

For text to speech, the model produces audio instead of consuming it as the only primary signal. The guide presents the task as generating natural sounding speech from text, potentially across multiple languages and speakers. The simplest interface is the text-to-audio pipeline, also available through the text-to-speech alias, which returns audio data and a sampling rate suitable for playback in notebooks. The same guide shows that some models accept richer conversational inputs, including reference audio for voice cloning or speaker tags and nonverbal cues for more expressive speech generation.

Sources: docs/source/en/tasks/text-to-speech.md

Training and Inference Flow

A typical audio training workflow begins with dependencies, because audio examples rely on packages beyond the core library. The classification guide installs Transformers, Datasets, Evaluate, SoundFile, Librosa, TorchCodec, and a backend such as PyTorch. The ASR guide adds JiWER for word error rate style evaluation. After installation, the task guides encourage authenticating with Hugging Face Hub so trained models can be uploaded and shared. The dataset is then loaded through Datasets, optionally split into smaller train and test partitions, and simplified to the columns needed for the selected supervised objective.

Sources: docs/source/en/tasks/audio_classification.md, docs/source/en/tasks/asr.md

pip install transformers datasets evaluate soundfile librosa torchcodec
pip install transformers datasets evaluate jiwer soundfile librosa torchcodec

The example READMEs show where the concise task guides connect to maintainable training recipes. Audio classification examples use Wav2Vec2 and related unsupervised speech-pretrained models such as HuBERT and XLSR-Wav2Vec2, emphasizing that these models can perform well with relatively little annotated audio. The single-GPU keyword spotting command configures model and dataset names, mixed precision, maximum audio length, evaluation and save strategies, best-model loading, accuracy as the selection metric, and Hub upload. A multi-GPU language identification command follows the same contract with different dataset columns and batch sizing.

Sources: examples/pytorch/audio-classification/README.md

python run_audio_classification.py \
    --model_name_or_path facebook/wav2vec2-base \
    --dataset_name superb \
    --dataset_config_name ks \
    --output_dir wav2vec2-base-ft-keyword-spotting \
    --remove_unused_columns False \
    --do_train \
    --do_eval \
    --fp16 \
    --metric_for_best_model accuracy \
    --push_to_hub

ASR examples expose two major implementation families. CTC examples fine-tune pretrained speech models through a workflow that creates a vocabulary from unique training and evaluation characters, preprocesses the speech recognition dataset, and trains with CTC loss. The README also documents adapter-based CTC workflows for Massive Multilingual Speech and sequence-to-sequence workflows for Whisper or warm-started speech encoder-decoder models. This breadth is important for choosing a recipe: CTC is often a strong fit when labels are transcripts aligned only at the sequence level, while sequence-to-sequence recipes are natural for encoder-decoder speech models.

Sources: examples/pytorch/speech-recognition/README.md

API Components and Configuration

At the high level, the pipeline API is the fastest inference entry point for text to speech. The guide uses the text-to-audio task name with a CSM checkpoint and notes that text-to-speech is an alias. The returned object contains generated audio and the sampling rate, which can be passed to notebook playback utilities. For CSM voice cloning, the input can be a chat template style list whose content includes both text and a reference audio array. For Dia, the prompt can include speaker tags and nonverbal annotations such as clearing the throat, laughing, sighing, crying, or music-like cues.

Sources: docs/source/en/tasks/text-to-speech.md

from transformers import pipeline
 
pipe = pipeline("text-to-audio", model="sesame/csm-1b")
output = pipe("Hello from Sesame.")

Lower-level fine-tuning flows use the model families and script arguments exposed by the examples rather than a single uniform pipeline call. The audio classification README fine-tunes Wav2Vec2-style encoders with a classification head and includes a practical flag for mismatched classifier dimensions: when the checkpoint head does not match the dataset label count, the user can pass the ignore mismatched sizes option. The speech recognition README points to AutoModelForCTC-compatible models and describes preprocessing steps that are specific to audio text alignment. These examples are designed to be adapted by replacing model names, dataset names, column names, batch sizes, and output directories.

Sources: examples/pytorch/audio-classification/README.md, examples/pytorch/speech-recognition/README.md

Implementation Details and Edge Cases

Sampling rate and column selection are recurring edge cases in the audio docs. Dataset examples represent audio as a dictionary containing an array, a path, and a sampling rate, and the guide text explicitly calls out that the audio field must be loaded and resampled. If a dataset includes both transcriptions and class labels, selecting the wrong target column changes the task completely. Classification should map waveform examples to label ids, whereas ASR should map waveform examples to normalized text. In custom datasets, mirror that separation by naming audio, label, and transcript columns clearly before adapting the example scripts.

Sources: docs/source/en/tasks/audio_classification.md, docs/source/en/tasks/asr.md, examples/pytorch/audio-classification/README.md

The ASR example README also documents a concrete preprocessing failure mode. If data preprocessing freezes when using more than one preprocessing worker, it recommends setting the OMP thread count to one before launching the CTC training command. This note is tied to multiprocessing behavior in audio preprocessing rather than model quality. Treat it as a runtime stability control: first confirm the command works with conservative worker settings, then increase workers only after the dataset loading, resampling, and padding path is stable. This is especially useful when adapting examples to new machines or custom audio corpora.

Sources: examples/pytorch/speech-recognition/README.md

OMP_NUM_THREADS=1 python run_speech_recognition_ctc.py ...

Choosing the Right Page Next

Use this page as the audio task router. If your output is a label, continue with the audio classification guide and the PyTorch audio classification examples. If your output is transcript text, continue with ASR and choose between CTC, adapter, and sequence-to-sequence examples based on the checkpoint family. If your output is generated speech, continue with text to speech and start with the pipeline examples before investigating fine-tunable families such as SpeechT5, FastSpeech2Conformer, Dia, and CSM. For shared preprocessing concepts like padding, truncation, batching, and processors, read the preprocessing and processor pages next.

Sources: docs/source/en/tasks/audio_classification.md, docs/source/en/tasks/asr.md, docs/source/en/tasks/text-to-speech.md, examples/pytorch/audio-classification/README.md, examples/pytorch/speech-recognition/README.md

Ask a question...