OpenWikigolang/go

Language Syntax and Built-ins

Purpose and Scope

This page connects the Go language surface that programmers write every day with the repository files that describe and implement it. The language surface includes declarations, expressions, literals, names, operators, automatic semicolon insertion, and the special identifiers that exist without an import. The implementation surface begins in the compiler front end, where source text is scanned into tokens and then parsed into structured syntax. It also includes the documentation-only builtin package, which is how predeclared identifiers are presented to users even though they are not ordinary imported package members.

Sources: src/builtin/builtin.go, src/cmd/compile/internal/syntax/scanner.go, src/cmd/compile/internal/syntax/parser.go

The important distinction for readers is that Go syntax is not implemented by the documentation package alone. The file for builtin identifiers documents language-defined names such as numeric types, boolean constants, nil, iota, aliases, and constraints. The scanner and parser are compiler machinery. The scanner reads characters, classifies them into tokens, reports lexical errors, and performs newline-to-semicolon translation where the language requires it. The parser then consumes those tokens according to the grammar used by the compiler front end.

Relevant Source Files

  • src/builtin/builtin.go — Documents Go predeclared identifiers, including basic types, aliases, constants, nil, iota, any, comparable, and documentation stand-ins used by godoc.
  • src/cmd/compile/internal/syntax/scanner.go — Implements the compiler scanner, the lexical tokenizer that advances through Go source one token at a time and records token metadata.
  • src/cmd/compile/internal/syntax/parser.go — Provides the companion parser stage in the compiler syntax package, consuming scanner tokens to build the compiler's syntactic representation.

Core Language Primitives

Go's predeclared identifiers are part of the language environment, not names imported from a normal package. The builtin documentation file makes that explicit: it says the documented items are not actually in package builtin, but their descriptions allow documentation tools to present the language's special identifiers. That design lets user-facing reference material live in ordinary Go-shaped documentation while preserving the compiler rule that these names are already available in every package block unless shadowed by a local declaration.

Sources: src/builtin/builtin.go

The documented primitive types cover booleans, signed and unsigned integers, floating-point numbers, complex numbers, strings, pointer-sized integers, and conventional aliases. The definitions explain ranges for fixed-width integer types, the minimum size and distinct identity of int and uint, string immutability, and the fact that uintptr can hold the bit pattern of a pointer. The aliases byte and rune are documented as exactly equivalent to uint8 and int32, while still carrying conventional meaning for bytes and character values in source code.

The same file records the language's special values and generic-era predeclared names. The constants true and false are untyped boolean values, and iota is the zero-indexed ordinal for a const specification. The identifier nil is the zero value for pointer, channel, function, interface, map, or slice types, which is why it cannot be assigned where a concrete non-nilable value is expected. The aliases any and comparable document two important type-parameter conveniences: any is interface{} in all ways, while comparable is constrained to use as a type parameter constraint.

Scanner Responsibilities

The scanner is the first compiler component that gives structure to raw source text. After initialization, repeated calls advance one token at a time. Its state records the current line and column, whether the line was blank up to the current column, the current token, literal text, literal kind, operator kind, and operator precedence. This makes tokenization more than simply splitting text: it preserves enough source-position and token-detail information for diagnostics, parsing decisions, and later compiler stages that need precise source locations.

Sources: src/cmd/compile/internal/syntax/scanner.go

The scanner exposes modes for comment handling. With no mode flag, comments are ignored for tokenization. With the comments flag, every comment is reported to the error handler with its position and text. With the directives flag, only directive-bearing comments such as line directives and go directives are reported. The implementation deliberately routes those comment reports through the same handler channel as lexical diagnostics, while guaranteeing that ordinary error messages are non-empty and do not begin with a slash. Consumers can distinguish comments by their leading slash.

Semicolon insertion is also visible in scanner state. The scanner stores a newline-semi flag that causes a newline or end of file to translate to a semicolon token after tokens that end a statement. In the token loop, whitespace is skipped differently depending on whether that flag is set. When end of file is reached after a token requiring insertion, the scanner emits a semicolon with a literal indicating EOF rather than immediately returning the end token. This implements a language rule before the parser sees the stream.

Parsing and Source-to-Code Mapping

The front-end flow is best read as a pipeline. Source bytes enter the syntax package through the scanner, where Unicode-aware identifier handling, whitespace skipping, literal recognition, operator classification, and diagnostic reporting happen. Tokens then pass to the parser source in the same internal syntax package. The parser is responsible for recognizing declarations and expressions from those tokens, while the scanner has already resolved lower-level questions such as whether input is a name, literal, operator, parenthesis, newline semicolon, or end marker.

Sources: src/cmd/compile/internal/syntax/scanner.go, src/cmd/compile/internal/syntax/parser.go

This separation matters when debugging language behavior. If source text is rejected because of malformed numeric, string, raw string, or rune literal spelling, the scanner is a likely place to inspect because literal recognition and the bad-literal marker are scanner responsibilities. If tokenization succeeds but the sequence cannot form a declaration, statement, or expression, the parser stage is the better starting point. Separating these concerns keeps lexical encoding and grammar structure independent enough for precise diagnostics and maintainable compiler code.

Compact Reference

Predeclared identifier groups documented by the repository:

  • Boolean values and type: true, false, bool.
  • Integer types: int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64, uintptr.
  • Floating and complex types: float32, float64, complex64, complex128.
  • Text and aliases: string, byte, rune.
  • Generic and interface aliases: any and comparable.
  • Special declaration and zero-value identifiers: iota and nil.
  • Documentation stand-ins: Type and related placeholder names used to describe builtin functions consistently.

Scanner state and behavior exposed in the implementation:

  • Initialization accepts a source reader, an error handler, and a mode bitset.
  • The current token record includes line, column, token kind, literal text, literal kind, operator, and precedence.
  • Literal scanning marks malformed literals while still retaining the literal text for diagnostics.
  • Comment reporting is optional and can include all comments or only directive comments.
  • Newline and end-of-file semicolon insertion is handled before parsing.

Practical Reading Guide

When reading Go source or compiler behavior, start by classifying what kind of rule you are studying. The meaning of a name such as string, nil, or comparable is documented in the builtin file because it is part of the language environment. The spelling of identifiers, literals, comments, operators, and statement-ending semicolons belongs to scanner behavior. The arrangement of those tokens into declarations and expressions belongs to the parser. This source mapping helps avoid looking for language rules in the wrong layer of the repository.

Sources: src/builtin/builtin.go, src/cmd/compile/internal/syntax/scanner.go, src/cmd/compile/internal/syntax/parser.go

For everyday Go users, these files explain why normal programs can use names such as int, error-shaped builtin functions, nil, and any without an import, and why source formatting can omit most semicolons. For compiler contributors, they show where documentation-facing language names meet implementation-facing token streams. A useful next step is to compare this page with generics concepts for type-parameter constraints, Go doc comments for comment parsing and presentation, and compiler architecture for how parsed syntax continues into type checking and later compilation phases.

Ask a question...