Expressions, Input Format, and Notation

Purpose and Scope

This page connects three Reference-facing ideas to the Rust repository implementation: input format, notation, and expressions. Input format is the question of how a Rust source file becomes characters and tokens. Notation is the grammar vocabulary the Reference uses to describe lexer and syntax productions. Expressions are the language constructs that produce values and may have effects. The compiler does not implement the Reference as prose; it represents parsed syntax, macro-specific fragments, lowered intermediate forms, formatting tools, and MIR type checks in specialized modules. The goal here is to help readers move from the formal documentation wording to concrete source areas that preserve or transform syntax information.

The official Reference describes source as UTF-8 input that is interpreted as a sequence of Unicode characters, normalized for details such as byte order marks and CRLF pairs before tokenization. It also explains grammar notation such as lexer tokens, syntactic productions, optional or repeated items, alternation, and hard cuts. For expressions, the Reference frames every expression as producing a value and possibly having effects; expression structure determines evaluation structure. In repository terms, those concepts appear as token and AST data, syntax-preserving spans, macro-specific parsers, lowering from AST to HIR, formatting tools that consume source ranges, and borrow checking that validates the types flowing into and out of lowered bodies.

Sources: compiler/rustc_ast/src/format.rs, compiler/rustc_ast_lowering/src/format.rs, compiler/rustc_parse_format/src/lib.rs, src/tools/rustfmt/src/format-diff/main.rs, src/doc/unstable-book/src/language-features/cfg-target-object-format.md, compiler/rustc_borrowck/src/type_check/input_output.rs

Relevant Source Files

• compiler/rustc_parse_format/src/lib.rs defines the parser-facing data model for Rust format strings, including parse modes, literal pieces, argument positions, and formatting specifications.
• compiler/rustc_ast/src/format.rs defines the AST representation of format_args!, including template pieces, placeholders, arguments, raw format string storage, and source-literal tracking.
• compiler/rustc_ast_lowering/src/format.rs lowers parsed format arguments into HIR expression form and contains optimization-oriented logic for flattening and inlining literal arguments.
• src/tools/rustfmt/src/format-diff/main.rs shows a source-input tool path: it reads diff text, extracts changed file ranges, filters Rust files, and invokes rustfmt --file-lines with JSON ranges.
• src/doc/unstable-book/src/language-features/cfg-target-object-format.md documents the unstable cfg_target_object_format language feature, showing syntax that depends on target object format through #[cfg] and cfg!.
• compiler/rustc_borrowck/src/type_check/input_output.rs checks MIR input and output types against expected function or closure signatures after earlier parsing, lowering, and type normalization stages.

These files cover different depths of the same language pipeline. The format-string crates are narrow but very concrete: they take a token-level language inside a macro invocation and preserve enough structure to diagnose and lower it correctly. The rustfmt diff tool operates outside the compiler pipeline, but it demonstrates how Rust tooling still depends on precise source file paths and line ranges. The unstable-book page is documentation rather than compiler code, yet it records a language surface that the implementation must understand as configuration syntax. The borrow checker file is later in the pipeline, where expressions have already become MIR and the remaining question is whether body inputs and outputs match the signature-level contract.

Sources: compiler/rustc_parse_format/src/lib.rs, compiler/rustc_ast/src/format.rs, compiler/rustc_ast_lowering/src/format.rs, src/tools/rustfmt/src/format-diff/main.rs, src/doc/unstable-book/src/language-features/cfg-target-object-format.md, compiler/rustc_borrowck/src/type_check/input_output.rs

From Input Text to Syntax Structures

A useful way to read the Reference chapters is as a layered contract. The input-format chapter describes the raw material: source files become character streams and then tokens. The notation chapter describes how grammar fragments are written down: uppercase names are lexer products, CamelCase names are syntactic productions, quoted strings are exact characters, and repetition or alternation controls how larger forms are recognized. The expressions chapter then describes a semantic category built from that syntax: expressions always produce a value and may also cause effects during evaluation. Each layer constrains the next one, but the repository source usually handles them in separate places rather than in one monolithic parser file.

Format strings are a compact example of this layering because they are parsed inside Rust source but have their own mini-language. compiler/rustc_parse_format/src/lib.rs distinguishes ParseMode::Format, ParseMode::InlineAsm, and ParseMode::Diagnostic, which means the same textual braces and placeholders are interpreted under different contracts depending on the macro or attribute context. The crate emits a stream of Piece values, separating literal text from NextArgument entries. That mirrors the Reference distinction between literal input and syntactic productions: a string is not just bytes once it is inside a formatting macro; it becomes a structured sequence with argument positions and formatting options.

The AST representation in compiler/rustc_ast/src/format.rs is the next important bridge. FormatArgs is described in the source as the parsed form of a complete format_args!() invocation. It stores a span, a vector of FormatArgsPiece values, FormatArguments, the raw unsplit format string literal, and a flag indicating whether the literal was actually written in the source. Those details matter for diagnostics and suggestions. A parser can know that a placeholder exists, but a diagnostic often needs to know whether it can safely point into the original literal or whether the text came from something like concat! or include_str!. Sources: compiler/rustc_parse_format/src/lib.rs, compiler/rustc_ast/src/format.rs

Expression-Oriented Lowering and Format Arguments

The Reference treats expressions as value-producing constructs whose structure drives evaluation. In the implementation, source-level expression forms are not the final representation used by every compiler pass. compiler/rustc_ast_lowering/src/format.rs shows one targeted lowering path: LoweringContext::lower_format_args takes a span and parsed FormatArgs, marks the span as a format-literal desugaring, optionally flattens or inlines formatting arguments under an unstable option, and calls into expansion that produces a HIR expression kind. The public-facing expression remains format_args!(...), but the compiler prepares a lower-level expression tree that later stages can type-check and codegen.

This lowering step also preserves a distinction between written syntax and generated structure. The implementation marks the span with DesugaringKind::FormatLiteral and carries whether the format literal was source-written. That is important because Rust diagnostics routinely refer back to source spans, and the compiler should avoid pretending that generated strings have the same editability as user-written source. The inline-literal optimization described in the source turns a pattern such as format_args!("Hello, {}! {} {}", "World", 123, x) into a shape closer to format_args!("Hello, World! 123 {}", x) when it can prove the literal arguments are simple displayable constants. That is an expression transformation, but it is constrained by syntax and span information.

The format AST also captures argument lookup rules that are easy to miss if one only reads examples. FormatArguments tracks the argument vector, unnamed counts, explicit counts, and a map from names to indexes. Position in the parser crate distinguishes implicit index arguments, explicit numeric arguments, and named arguments. The AST source comments illustrate format pieces, placeholders, and positions in a single format_args! call. In other words, notation such as {abc:.xyz$} becomes a typed structure before it becomes executable formatting logic. This is how a grammar fragment becomes a compiler-maintained contract rather than an ad hoc string search.

Sources: compiler/rustc_ast_lowering/src/format.rs, compiler/rustc_ast/src/format.rs, compiler/rustc_parse_format/src/lib.rs

Configuration Syntax and Target-Dependent Expressions

Not all syntax questions are about ordinary expression grammar. Conditional compilation is a good example because it affects which items or expression branches are relevant for a target. The unstable-book page for cfg_target_object_format documents a feature allowing code to branch on the current target object file format. Its examples use attributes such as #[cfg(target_object_format = "elf")], #[cfg(target_object_format = "mach-o")], and the expression-like cfg!(target_object_format = "wasm") macro. This is still part of the language surface that developers write in source files, but it depends on target configuration rather than local expression evaluation alone.

#![feature(cfg_target_object_format)]
 
#[cfg(target_object_format = "elf")]
fn a() {
    // ...
}
 
fn b() {
    if cfg!(target_object_format = "wasm") {
        // ...
    }
}

This example is useful when reading the Reference because it shows that input notation eventually interacts with compilation environment. A cfg predicate has tokens and grammar just like other Rust syntax, but the meaningful value of target_object_format is supplied by the selected target. The page does not by itself describe the compiler implementation of every target query, so it should be read as a language-feature signal rather than a full implementation map. For this topic, its main value is showing how target-sensitive syntax can appear both as an outer attribute on an item and as a macro form inside an expression context.

Sources: src/doc/unstable-book/src/language-features/cfg-target-object-format.md

Tooling That Consumes Source Ranges

The Rust repository also contains tools that depend on source input without being the compiler parser. src/tools/rustfmt/src/format-diff/main.rs implements rustfmt-format-diff, a small command-line tool inspired by clang-format-diff. It defines a default file pattern of .*\.rs, parses options with clap, reads diff text from standard input, scans for changed files and hunk line ranges, serializes those ranges to JSON, and invokes rustfmt with --file-lines. This path reinforces that Rust tooling treats source files as structured locations, not just opaque text blobs.

The scan_diff function is especially relevant to input format because it turns a non-Rust input format, a unified diff, into Rust file ranges. It builds a regex based on the requested prefix-skipping count, extracts file paths from +++ lines, parses hunk headers for added line ranges, and filters files through a caller-supplied regular expression. The output is a HashSet<String> of files and a vector of Range values containing file names and line bounds. The next stage passes those ranges to rustfmt. Even though this is a formatter utility, it depends on accurate mapping from external text notation to Rust source positions.

This source also illustrates a common boundary between compiler semantics and developer experience. The Reference defines what Rust source means, while rustfmt decides how Rust source should be laid out. The diff formatter does not decide expression semantics, but it must respect changed ranges and file filtering so formatting can be applied safely in review workflows. For readers studying notation, this is a reminder that formal grammar is only one consumer of source structure. Editors, formatters, diagnostics, and CI scripts all need stable ways to identify files, spans, and ranges.

Sources: src/tools/rustfmt/src/format-diff/main.rs

Later Pipeline: Inputs, Outputs, and MIR Type Checking

By the time code reaches borrow checking, the compiler is no longer working primarily with the Reference's surface grammar. compiler/rustc_borrowck/src/type_check/input_output.rs explains that it equates input and output types appearing in MIR with expected input and output types from the function signature. The module comment notes that expected types can arrive before normalization and may contain opaque impl Trait instances, while the MIR locals for RETURN_PLACE and arguments are fully normalized and contain revealed impl Trait values. This is a later-stage counterpart to expression parsing: the surface expression has become a body with typed inputs, outputs, and local places.

The function check_signature_annotation focuses on closure-like definitions and explicit closure signature annotations, such as a closure parameter annotated with a complex type. It first ensures the body is closure-like and returns early if the MIR body is tainted by earlier errors. That defensive behavior matters in a syntax-to-semantics pipeline: if parsing or match checking already failed, later passes should avoid panicking because body arguments may not match the user-provided signature. The code then instantiates canonical variables, replaces bound items with fresh variables, and handles async or generator closure cases through coroutine-specific signature construction.

For readers of the expressions chapter, this file is a useful endpoint. Expressions produce values, but the compiler ultimately checks those values in a normalized intermediate representation with region variables, closure signatures, coroutine closures, and return places. The original notation has not disappeared; it has been translated through AST, HIR, and MIR into obligations that can be checked. When debugging language behavior, it is therefore important to ask which layer owns the bug: tokenization and parsing, AST representation, lowering and desugaring, target configuration, formatting tooling, or MIR type validation.

Sources: compiler/rustc_borrowck/src/type_check/input_output.rs

Compact Reference: Source Concepts and Code Anchors

Use this page as a map rather than as a substitute for the Reference. If you are learning the language, start with the Reference expression, input-format, and notation chapters to understand the formal vocabulary. If you are reading compiler code, follow the transformation from parsed FormatArgs through AST lowering and finally into MIR type checks. If you are working on developer tooling, compare compiler-owned syntax handling with the rustfmt diff utility's source-range pipeline. The most productive next step is to choose one surface construct, such as a format placeholder or cfg!(...), and trace which layer first parses it, which layer lowers it, and which later pass validates or consumes its meaning.

Sources: compiler/rustc_parse_format/src/lib.rs, compiler/rustc_ast/src/format.rs, compiler/rustc_ast_lowering/src/format.rs, src/tools/rustfmt/src/format-diff/main.rs, src/doc/unstable-book/src/language-features/cfg-target-object-format.md, compiler/rustc_borrowck/src/type_check/input_output.rs