Language Reference Overview
Purpose and Scope
The Rust Reference is the specification-oriented companion to Rust’s tutorial material. A tutorial such as The Book or a runnable example collection teaches concepts in a learning order; a reference answers exact questions about what the language accepts, how constructs are classified, and which vocabulary should be used when discussing the language precisely. In this repository, that distinction is important because the same source tree is presented as the main Rust repository and contains the compiler, standard library, and documentation rather than only end-user prose.
Sources: README.md
A good mental model is to treat the Reference as the reader-facing description of Rust source text and the compiler syntax crates as implementation structures that must represent that source text. The requested compiler evidence centers on rustc_ast, which its README describes as containing the abstract syntax tree, token and token stream definitions, traits and data structures for mutating ASTs, and shared syntax-related definitions used by compiler components such as the lexer and macro expansion. That makes it the most direct source-backed bridge between formal language questions and compiler code.
Sources: compiler/rustc_ast/README.md
This page does not try to restate the full Rust Reference. Instead, it explains where reference-style reading fits in the repository’s documentation spine and how to navigate from a formal language concept into the syntax-oriented implementation when that is useful. If you are learning Rust for the first time, begin with installation, The Book, and examples. If you already know the construct and need exact terminology, use the Reference first, then inspect rustc_ast only when implementation behavior or compiler contribution work requires it.
Sources: README.md, compiler/rustc_ast/README.md
Relevant Source Files
- README.md - Establishes this repository as the main Rust source tree, states that it contains the compiler, standard library, and documentation, and links readers to the website, getting started material, learning resources, documentation, and contribution guidance.
- compiler/rustc_ast/README.md - Defines the scope of the
rustc_astcrate as syntax-focused: AST definitions, tokens, token streams, AST mutation support, and shared definitions for the lexer and macro expansion. - compiler/rustc_ast/src/lib.rs - Declares the Rust Abstract Syntax Tree crate, notes that its API is unstable, exposes syntax modules, configures crate-level compiler features, and reexports selected AST definitions and traits.
Documentation Spine and Reader Workflow
The top-level README gives a compact map of the official Rust reader journey: website, getting started, learn, documentation, and contributing. That layout explains why the Reference is not the first stop for most beginners. Getting started material answers how to install Rust and run a first program. Learning material builds intuition about ownership, types, functions, modules, and error handling. Reference material becomes most valuable after that foundation, when the reader can ask a focused question such as whether a construct is an expression, an item, an attribute, or a macro-related form.
Sources: README.md
Official Rust By Example reinforces the difference between examples and reference prose. It presents Rust as a systems programming language focused on safety, speed, and concurrency, then teaches through runnable examples covering primitives, custom types, expressions, flow control, functions, modules, crates, attributes, generics, traits, macros, and error handling. Those examples are excellent for experimentation because they show a feature in use. The Reference-oriented step is different: once the example is familiar, the reader checks the formal grammar, allowed positions, and terminology for the same construct.
The repository source helps keep those layers separate. The README links to documentation and learning resources for users, while rustc_ast is explicitly a compiler crate concerned with syntax. That means the compiler source is not the stable contract a Rust programmer writes against. The stable reader-facing contract is the language documentation. The source is still valuable because it shows how rustc organizes syntax internally, which is essential when diagnosing parser behavior, reviewing compiler changes, or understanding why a documented construct needs token-level and tree-level representation.
Sources: README.md, compiler/rustc_ast/README.md
System-to-Code Mapping
At the highest level, the Reference describes Rust programs as source-language constructs, while rustc_ast provides the compiler-side vocabulary for those constructs after parsing-related processing. The crate root documents itself as the Rust Abstract Syntax Tree and immediately warns that the API is completely unstable and subject to change. That note should shape how readers use the code: it is not a stable public library API, but it is the authoritative in-tree place to understand the compiler’s syntax data model at this commit.
Sources: compiler/rustc_ast/src/lib.rs
The module list in compiler/rustc_ast/src/lib.rs is a compact map of syntax responsibilities. Modules named ast, ast_traits, attr, token, and tokenstream correspond to common reference questions about tree nodes, attributes, lexical units, and token-sequence forms. Modules named visit and mut_visit indicate the traversal and transformation patterns used by compiler passes that inspect or modify syntax. Utility modules for comments, literals, parser support, Unicode, classification, and case handling show that syntax work includes both grammar-shaped nodes and smaller representation details.
Sources: compiler/rustc_ast/src/lib.rs
Macro expansion is a useful example of why the implementation cannot be reduced to only parsed tree nodes. The rustc_ast README explicitly calls out token streams and shared definitions for macro expansion, and the crate root exposes both expand and tokenstream. A macro system must reason about input before it has become ordinary post-expansion syntax, and it can produce syntax that later phases consume. When reading Reference material about macros or attributes, keep this intermediate token-stream layer in mind instead of assuming every rule maps directly to one final AST node.
Sources: compiler/rustc_ast/README.md, compiler/rustc_ast/src/lib.rs
Compact Source-Level Reference
The following table names the concrete compiler-facing components visible in the requested sources. These names are useful landmarks when moving from formal language prose into implementation code, but they should be read as unstable rustc internals rather than as APIs for ordinary Rust programs.
| Component or declaration | Source-level contract visible here | Reader use |
|---|---|---|
rustc_ast crate | Crate documentation says it is the Rust Abstract Syntax Tree and that the API is completely unstable and subject to change. | Treat as internal compiler vocabulary for syntax, not as stable user API. |
pub mod ast | Public module exposed from the crate root and reexported with pub use self::ast::*. | Start here when looking for primary AST node definitions. |
pub mod ast_traits | Exposes shared AST traits; crate root reexports AstNodeWrapper, HasAttrs, HasNodeId, and HasTokens. | Use these trait names to understand common node capabilities such as attributes, IDs, and stored tokens. |
pub mod attr | Attribute-related syntax module. | Connect Reference questions about attributes to compiler representation. |
pub mod token and pub mod tokenstream | Token and token-stream modules, matching the README’s token-related scope. | Inspect these when a feature depends on lexical units, macro input, or preserved tokens. |
pub mod visit and pub mod mut_visit | Traversal and mutation modules for syntax trees. | Look here for compiler patterns that walk or rewrite AST structures. |
pub mod expand | Expansion-related module exposed from the crate root. | Relevant when syntax is produced or transformed by macro expansion. |
pub mod util::{case, classify, comments, literal, parser, unicode} | Utility submodules grouped under util. | Use for support logic around literals, comments, parser helpers, Unicode, classification, and case handling. |
#![recursion_limit = "256"] | Crate-level recursion limit configured in the crate root. | Shows that this compiler crate has internal build-time configuration, not only syntax type declarations. |
#![feature(...)] entries | Crate-level use of unstable Rust features such as associated_type_defaults, deref_patterns, iter_order_by, and macro_metavar_expr. | Signals that rustc internals may rely on nightly/compiler-only capabilities. |
Sources: compiler/rustc_ast/src/lib.rs
Reading Flow for Language Questions
For everyday language questions, begin with the documented construct instead of beginning with a source file. Ask what kind of thing the source text is: an expression, item, pattern, literal, attribute, module declaration, macro invocation, or other syntactic category. Then use the Reference to learn its formal boundaries and terminology. Only after that should you map the construct into rustc_ast, because the implementation may preserve tokens, attach attributes, assign node identifiers, or support traversal in ways that are not part of the language contract itself.
Sources: README.md, compiler/rustc_ast/README.md
For parser-facing or input-format questions, separate source text, tokens, token streams, and AST nodes. Source text is what a Rust programmer writes. Token-oriented representation is the lexer-facing layer named in the rustc_ast README and represented by the crate’s token and tokenstream modules. The AST is the structured syntax tree represented by the crate as a whole. Macro expansion can operate between these layers, so code involving macros and attributes often needs both token-level and tree-level reasoning.
Sources: compiler/rustc_ast/README.md, compiler/rustc_ast/src/lib.rs
For compiler contribution work, use the Reference as the design-facing language description and use rustc_ast as the syntax data boundary. A syntax change may require changes to AST definitions, token handling, attribute representation, traversal code, mutation code, parser utilities, and expansion support. The crate README’s scope statement is a reminder that syntax is shared infrastructure: before type checking, borrow checking, code generation, or diagnostics can reason about a construct, the frontend must represent it consistently enough for later phases to consume.
Sources: compiler/rustc_ast/README.md
The unstable API note in the crate root also prevents a common mistake. It is tempting to treat visible pub mod declarations and reexports as stable APIs because they are public within the crate graph. In rustc, that visibility is an implementation boundary, not a promise to external users. The Reference is the place for stable language meaning; rustc_ast is the current compiler implementation’s organization of syntax data. Contributors can rely on it for source navigation, but downstream projects should not treat it as a compatibility guarantee.
Sources: compiler/rustc_ast/src/lib.rs
Next Steps
If you are learning Rust, continue through tutorial and example-driven documentation until you can identify the construct you want to understand, then use the Reference for precise rules and terminology. If you are investigating compiler behavior, read compiler/rustc_ast/README.md before diving into individual modules because it states the crate boundary and points to parser and macro-expansion background in the rustc development guide. Nearby OpenWiki pages to read next are expressions-and-input-format for syntax layering, rust-by-example for example-led learning, and compiler-architecture for the broader pipeline after AST construction.
Sources: README.md, compiler/rustc_ast/README.md