Token Classification

Purpose and Scope

Token classification is the Transformers task pattern for assigning a label to each token position in an input sequence. The most common example is named entity recognition, or NER, where the model marks spans such as people, locations, organizations, products, and creative works. The official task guide frames the reader goal as a complete fine-tuning and inference path: fine-tune DistilBERT on WNUT 17 to detect emerging entities, then use the fine-tuned model to make predictions. This page explains that flow as a developer guide, with emphasis on the parts that are easy to get wrong: label schemas, tokenizer word alignment, ignored labels, and the handoff from preprocessed datasets to model training.

Sources: docs/source/en/tasks/token_classification.md, docs/source/ar/tasks/token_classification.md, docs/source/ja/tasks/token_classification.md, docs/source/ko/tasks/token_classification.md, docs/source/pt/tasks/token_classification.md, docs/source/zh/tasks/token_classification.md

Unlike sequence classification, token classification preserves a one-label-per-token relationship through preprocessing, batching, training, and inference. That means the dataset labels must continue to line up with model input positions after the tokenizer adds special tokens and splits words into subwords. The WNUT 17 examples in the documentation make this concrete: the dataset contains a tokens field with word-like units and a parallel ner_tags field with integer labels. The guide then converts those integers through the dataset feature names so the model’s numeric outputs can be interpreted as labels such as B-location, I-location, or O.

Relevant Source Files

• docs/source/en/tasks/token_classification.md — English source for the task guide, including the DistilBERT and WNUT 17 workflow, installation prerequisites, dataset inspection, label schema, preprocessing, and inference goals.
• docs/source/ar/tasks/token_classification.md — Arabic localization of the same task guide, preserving the NER framing, WNUT 17 example, tokenizer setup, and label-alignment concepts.
• docs/source/ja/tasks/token_classification.md — Japanese localization that includes the special-token and subword-alignment explanation for [CLS], [SEP], word_ids, and -100 ignored labels.
• docs/source/ko/tasks/token_classification.md — Korean localization that explicitly lists the three alignment rules: map tokens to words, ignore special tokens, and label only the first subtoken of a word.
• docs/source/pt/tasks/token_classification.md — Portuguese localization focused on the same WNUT 17 NER tutorial and the DistilBERT tokenizer preprocessing path.
• docs/source/zh/tasks/token_classification.md — Chinese localization that includes the full label-realignment function pattern and the truncation-aware tokenizer call.

The multilingual source files are not separate implementations; they are the localized publishing surface for the same official task. Read the English file as the canonical task narrative if you are working in the repository, and use the localized files to confirm that concepts such as is_split_into_words=True, word_ids, and -100 are stable documentation terminology rather than incidental wording. The repeated structure across languages is useful for contributors because it shows which parts of the task are considered essential: task definition, compatible checkpoints, installation, optional Hub login, dataset loading, label interpretation, preprocessing, fine-tuning, and inference.

Sources: docs/source/en/tasks/token_classification.md, docs/source/ar/tasks/token_classification.md, docs/source/ja/tasks/token_classification.md, docs/source/ko/tasks/token_classification.md, docs/source/pt/tasks/token_classification.md, docs/source/zh/tasks/token_classification.md

Core Primitives

The first primitive is the labeled token dataset. In the documented workflow, load_dataset("wnut_17") returns examples with an id, a tokens list, and a ner_tags list. Each position in ner_tags corresponds to the token at the same position in tokens before model tokenization. The label list is read from wnut["train"].features[f"ner_tags"].feature.names, which is important because it avoids hard-coding the mapping between integer IDs and string labels. That mapping includes O for non-entity tokens plus B- and I- labels for entity boundaries and continuations.

The second primitive is the tokenizer, loaded in the guide with AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased"). The dataset appears pre-tokenized because tokens is already a list, but Transformers tokenizers still need to convert those words into model vocabulary IDs. Passing is_split_into_words=True tells the tokenizer that the input is a list of words rather than a raw string. For DistilBERT, one visible result is subword splitting: a handle such as @paulwalk can become @, paul, and ##walk, and the model input also receives special tokens such as [CLS] and [SEP].

The third primitive is the alignment rule that reconciles dataset words with tokenizer output positions. The guide uses BatchEncoding.word_ids to map each tokenized position back to the original word index. Positions that do not correspond to an original word, such as special tokens, receive -100 so PyTorch’s cross-entropy loss ignores them. For words split into multiple subtokens, the documented recipe labels only the first subtoken and assigns -100 to the rest. This creates a training target that remains compatible with one original word label while using a subword model.

Sources: docs/source/en/tasks/token_classification.md, docs/source/ja/tasks/token_classification.md, docs/source/ko/tasks/token_classification.md, docs/source/zh/tasks/token_classification.md

Execution Flow

Start by installing the libraries used by the guide. Transformers provides the model, tokenizer, and training APIs; Datasets loads WNUT 17; Evaluate provides metric integration; and seqeval supplies standard sequence-labeling metrics for NER-style spans. The documentation also encourages logging in through huggingface_hub.notebook_login() so the trained model can be uploaded and shared. That login is not required to understand the local preprocessing flow, but it is part of the intended end-to-end tutorial because task guides generally finish with reusable or shareable checkpoints.

pip install transformers datasets evaluate seqeval

>>> from huggingface_hub import notebook_login
>>> notebook_login()

Load the dataset and inspect a single row before building the training pipeline. This is more than a sanity check: token classification failures often come from mismatched label order, unexpected dataset columns, or misunderstanding whether the input has already been tokenized. In the official example, wnut["train"][0] shows an English sentence broken into tokens and a parallel list of ner_tags. The label vocabulary then explains how to decode numeric tags into names. Prefixes matter: B- starts an entity span, I- continues the same span, and O marks text outside any entity.

>>> from datasets import load_dataset
>>> wnut = load_dataset("wnut_17")
>>> label_list = wnut["train"].features[f"ner_tags"].feature.names

Preprocessing is the most task-specific step. The tokenizer must be called with truncation=True and is_split_into_words=True, and then each example’s labels must be transformed into token-level labels matching the tokenizer output length. Conceptually, the function loops over each example, asks the tokenized batch for word_ids(batch_index=i), and builds a new label list. If the word ID is None, use -100; if it is a new word, copy the original label; if it is a repeated word ID caused by subword splitting, use -100. The result can be passed to training as the labels field.

Sources: docs/source/en/tasks/token_classification.md, docs/source/ko/tasks/token_classification.md, docs/source/zh/tasks/token_classification.md

Implementation Details and Edge Cases

The label scheme is a span encoding, not just a flat classification vocabulary. B-location and I-location are separate labels because they communicate whether a token begins or continues a location entity. This is why an entity like Empire State Building can be represented across multiple positions rather than as one sentence-level label. During inference, the model predicts a label for each model token; downstream grouping logic can combine adjacent B- and I- predictions into entity spans. Keeping the label list from the dataset feature metadata helps preserve the exact ordering expected by the model head and metric computation.

Subword tokenization is the most important edge case for fine-tuning. If every subtoken of a split word receives the same label, long or frequently split words can overweight the loss and distort metrics. If labels are not expanded at all, the model receives label arrays with a different length from the input IDs. The documentation’s -100 convention solves both problems by producing a label list with the same length as the tokenized input while telling the loss function to ignore special tokens and non-first subtokens. This convention is also why inspecting converted tokens is useful before mapping the whole dataset.

Truncation is another practical constraint. DistilBERT has a maximum input length, so the task guide’s preprocessing pattern truncates tokenized sequences to remain within model limits. In token classification, truncation removes both input positions and their aligned labels, so the alignment function should be the only place where tokenized inputs and labels are built together. Padding is usually delegated to a data collator in the later training step, but the key invariant remains the same: every retained input position must have either a valid class ID or -100.

Sources: docs/source/en/tasks/token_classification.md, docs/source/ja/tasks/token_classification.md, docs/source/ko/tasks/token_classification.md, docs/source/zh/tasks/token_classification.md

Compact Reference

Use this reference as a checklist when adapting the task to another token-level labeling problem such as part-of-speech tagging, chunking, or custom entity extraction. Replace WNUT 17 with your dataset, but keep the same invariant: a stable label vocabulary, tokenizer-aware label alignment, ignored labels for positions that should not train the model, and a final inference step that maps predicted IDs back to human-readable labels. If your dataset starts as raw text rather than a list of words, you need a consistent word-level annotation and tokenization strategy before reusing the WNUT alignment pattern.

Sources: docs/source/en/tasks/token_classification.md, docs/source/pt/tasks/token_classification.md, docs/source/zh/tasks/token_classification.md

Next Steps

After you understand the alignment contract, continue with the broader training pages for batching, evaluation, checkpoint saving, and Hub upload. The token classification task guide is intentionally task-focused: it teaches the data shape and preprocessing decisions that make NER training correct. For production work, pair it with the Trainer and fine-tuning documentation so you can configure evaluation cadence, choose batch sizes, push a model to the Hub, and run inference through a token-classification pipeline or lower-level model calls. When contributing docs changes, update the English task page first and keep localized pages semantically aligned with the same reader workflow.