Language Modeling

Purpose and Scope

Language modeling in Transformers is documented as two related task guides: causal language modeling and masked language modeling. Causal language modeling teaches a model to predict the next token from the tokens that precede it, which makes it the training objective most closely associated with open-ended text generation, coding assistants, and interactive writing applications. Masked language modeling teaches a model to recover hidden tokens from both left and right context, which makes it useful when the application needs a strong representation of the whole sequence rather than strictly left-to-right continuation.

Sources: docs/source/en/tasks/language_modeling.md, docs/source/en/tasks/masked_language_modeling.md

The repository keeps the reader-facing walkthroughs in English and several localized versions. The English causal page is centered on DistilGPT2 and the text-generation task page, while the English masked page is centered on DistilRoBERTa and the fill-mask task page. The Spanish page presents both causal and masked language modeling together as forms of language modeling, and the Arabic and Japanese files mirror the same tutorials for localized audiences. Treat these files as the source of the public workflow, terminology, and recommended first experiment.

Sources: docs/source/en/tasks/language_modeling.md, docs/source/en/tasks/masked_language_modeling.md, docs/source/es/tasks/language_modeling.md, docs/source/ar/tasks/language_modeling.md, docs/source/ar/tasks/masked_language_modeling.md, docs/source/ja/tasks/language_modeling.md

Relevant Source Files

• docs/source/en/tasks/language_modeling.md - English causal language modeling guide, including the DistilGPT2 example, setup commands, dataset loading, train and test split, and task-page pointer for text generation.
• docs/source/en/tasks/masked_language_modeling.md - English masked language modeling guide, including the DistilRoBERTa example, setup commands, dataset loading, train and test split, and task-page pointer for fill-mask.
• docs/source/es/tasks/language_modeling.md - Spanish task guide that frames language modeling as causal and masked objectives in a single localized page and calls out both model choices.
• docs/source/ja/tasks/language_modeling.md - Japanese causal language modeling guide that follows the ELI5 loading and preprocessing sequence and emphasizes that labels are derived from the text itself.
• docs/source/ar/tasks/language_modeling.md - Arabic causal language modeling guide with the same task intent, notebook affordance, setup, login, and ELI5 experiment structure.
• docs/source/ar/tasks/masked_language_modeling.md - Arabic masked language modeling guide that localizes the bidirectional objective, DistilRoBERTa fine-tuning path, and fill-mask task pointer.

Choosing the Objective

Choose causal language modeling when the model should continue text from a prompt. The guide describes this objective as next-token prediction over a sequence where the model can only attend to tokens on the left, so future tokens are not visible while learning or generating. GPT-2 is the conceptual example, and DistilGPT2 is the tutorial checkpoint. This left-to-right constraint is the reason causal language models are used for creative writing and code-completion style products: the model repeatedly extends the context it has already seen.

Sources: docs/source/en/tasks/language_modeling.md, docs/source/ar/tasks/language_modeling.md, docs/source/ja/tasks/language_modeling.md

Choose masked language modeling when the model should learn bidirectional contextual understanding. The guide defines the task as predicting a masked token in a sequence while attending to tokens on both sides of the mask. BERT is the conceptual example, and DistilRoBERTa is the tutorial checkpoint. This objective is not the same as autoregressive generation: it is better suited to fill-mask behavior and representation learning because the model is trained to infer missing content from the entire surrounding sentence rather than generate only from preceding text.

Sources: docs/source/en/tasks/masked_language_modeling.md, docs/source/ar/tasks/masked_language_modeling.md, docs/source/es/tasks/language_modeling.md

Tutorial Workflow

Both guides start with a small, repeatable experiment before a full training run. The documented setup installs Transformers together with Datasets and Evaluate, then encourages logging in with a Hugging Face account so the fine-tuned model can be uploaded and shared. The dataset step intentionally loads only the first five thousand ELI5 examples, giving the reader a fast validation loop before spending more time on the complete dataset. This sequencing is important because most language-modeling errors surface during preprocessing, tokenization, batching, or label construction.

Sources: docs/source/en/tasks/language_modeling.md, docs/source/en/tasks/masked_language_modeling.md

pip install transformers datasets evaluate

>>> from huggingface_hub import notebook_login
>>> notebook_login()

The central dataset operation is the same in the English task guides: load an ELI5 category split, then divide the result into training and test subsets with the dataset train test split method. The causal guide uses the dany0407 ELI5 category dataset in the current English documentation, while the masked guide uses the ELI5 category dataset. Localized pages show earlier or alternate dataset identifiers, but the reader task remains consistent: use a manageable ask-science subset, inspect an example, and identify the answer text that will become training material.

Sources: docs/source/en/tasks/language_modeling.md, docs/source/en/tasks/masked_language_modeling.md, docs/source/es/tasks/language_modeling.md, docs/source/ja/tasks/language_modeling.md

>>> from datasets import load_dataset
>>> eli5 = load_dataset("dany0407/eli5_category", split="train[:5000]")
>>> eli5 = eli5.train_test_split(test_size=0.2)

Source-to-Code Mapping

The source pages map documentation concepts to the public APIs a user actually runs. Dataset loading comes from the Datasets library, while model and tokenizer loading are presented through Transformers auto classes later in the task flow. The examples deliberately start from pretrained checkpoints rather than building architectures from scratch. For causal language modeling, the checkpoint choice is DistilGPT2; for masked language modeling, it is DistilRoBERTa. That pairing teaches the reader that the checkpoint architecture must match the learning objective and downstream inference behavior.

Sources: docs/source/en/tasks/language_modeling.md, docs/source/en/tasks/masked_language_modeling.md, docs/source/ar/tasks/language_modeling.md, docs/source/ar/tasks/masked_language_modeling.md

A practical edge case appears in the ELI5 examples: the useful training text is nested inside answer structures rather than provided as a flat text column. The Spanish and Japanese pages explicitly call attention to the answer text field as the part that matters for preprocessing. This matters for custom datasets too. If a dataset stores prose inside nested dictionaries, lists, question fields, or metadata-heavy records, extract the natural language content first, then tokenize it according to the selected objective instead of passing the raw record shape directly to the model.

Sources: docs/source/es/tasks/language_modeling.md, docs/source/ja/tasks/language_modeling.md

Compact Reference

The minimal decision rule is simple: use a causal language model when inference will extend a prompt, and use a masked language model when inference will recover deliberately hidden spans from a complete sentence. Both task guides use fine-tuning followed by inference as the end-to-end shape of the tutorial. Both also point readers to the corresponding Hugging Face task page for compatible architectures and checkpoints, which is the right next stop when replacing the tutorial checkpoint with a larger model, a domain-specific checkpoint, or a multilingual model.

Sources: docs/source/en/tasks/language_modeling.md, docs/source/en/tasks/masked_language_modeling.md

Next Steps

After completing the language modeling tutorial, continue with lower-level pages that explain the mechanics hidden by the task guide. Read Tokenizers to understand padding, truncation, special tokens, and conversion from raw text into model inputs. Read Trainer and Fine-tuning to connect the preprocessing output to training arguments, evaluation, saving, and model upload. Read Text Generation when the causal model will be used with generation parameters, decoding strategies, or streaming output. Read Pipelines when the masked model will be exposed through a fill-mask style inference interface.

Sources: docs/source/en/tasks/language_modeling.md, docs/source/en/tasks/masked_language_modeling.md