Metrics, Traces, and Health Checks
Purpose and Scope
This page orients operators and platform engineers around the monitoring surfaces that Airflow documents together: metrics, traces, health checks, and the logging signals that help connect runtime symptoms back to user actions. In Airflow operations, “metrics” are numeric time-series signals, “traces” are spans that describe a request or task path through components, and “health checks” are fast status probes for automation. The source evidence for this page sits at the boundary between product documentation, the UI dashboard, FastAPI logging, and the logging configuration layer that determines how task logs are resolved. Sources: docs/images/documentation_architecture.py, airflow-core/src/airflow/ui/src/pages/Dashboard/HistoricalMetrics/index.ts, airflow-core/src/airflow/api_fastapi/logging/decorators.py, airflow-core/src/airflow/logging_config.py
Airflow’s official documentation groups metrics, traces, health checks, callbacks, and error tracking under “Logging & Monitoring,” which is a useful mental model for production work. Monitoring is not only a dashboard problem: it is also an evidence problem. When a scheduler appears slow, an API endpoint returns an error, or task logs are unavailable, the operator needs correlated signals from the UI, API access and audit logs, task log configuration, and documented health probes. The repository paths here show the source-side entry points that support that observability story rather than a single monolithic monitoring subsystem.
Relevant Source Files
- docs/images/documentation_architecture.py — generates the diagram for how Airflow documentation is published from the repository and served to users, which matters because monitoring procedures are maintained as first-party docs.
- airflow-core/src/airflow/ui/src/pages/Dashboard/HistoricalMetrics/index.ts — exports the
HistoricalMetricsdashboard page used by the Airflow UI surface for historical operational metrics. - airflow-core/src/airflow/api_fastapi/logging/init.py — marks the FastAPI logging package namespace used by API logging helpers.
- airflow-core/src/airflow/api_fastapi/logging/decorators.py — implements action logging helpers, audit-log masking, request-body handling, user identification, and log-injection hardening for API actions.
- airflow-core/src/airflow/logging_config.py — loads and caches Airflow logging configuration, resolves remote task log handlers, and exposes helpers for remote logging state.
Monitoring Model
Airflow deployments normally expose several kinds of monitoring data at once. Metrics provide aggregate behavior such as counts, gauges, and timings. Traces add request- or task-scoped causality, especially when OpenTelemetry is configured. Health checks give load balancers, supervisors, and humans a quick answer about whether a component is responsive. Logs provide the human-readable and audit-oriented trail that explains why a state changed. The official monitoring docs call out StatsD, OpenTelemetry, metric allow/block lists, renamed metrics, custom spans in tasks, and component-specific health checks as separate but related setup tasks.
The HistoricalMetrics UI export is intentionally small, but it is still an important source signal: the UI has a named dashboard page dedicated to historical metrics. That page is not the collector or backend; it is the operator-facing presentation layer. In practice, this means an Airflow deployment can collect and export metrics through configured backends while also surfacing selected historical information in the web UI. Operators should treat the UI as a navigation and diagnosis surface, then use their metrics backend for long retention, alerting, and cross-service correlation. Sources: airflow-core/src/airflow/ui/src/pages/Dashboard/HistoricalMetrics/index.ts
Logging is the bridge between raw signals and an explanation. The FastAPI logging decorator code shows Airflow logging user actions with an event name, request context, and a resolved user identity, defaulting to an anonymous identity when no user is present. It also includes explicit defensive behavior: user-supplied values are stripped of carriage returns and newlines before being sent to standard-library logging, and sensitive connection and variable fields are masked before persistence. Those details matter for monitoring because audit logs are often shipped into the same security and observability systems as metrics and traces. Sources: airflow-core/src/airflow/api_fastapi/logging/decorators.py
The logging configuration module handles another operationally important monitoring path: task log access. It imports the configured logging dictionary, resolves remote task log configuration through the providers manager, and caches the active remote task log handler and default remote connection ID. If a task is running but logs cannot be displayed, downloaded, or correlated with failures, this configuration layer is one of the first source-backed places to understand the control flow. Monitoring runbooks should therefore include both metrics/traces checks and log-resolution checks rather than treating task logs as unrelated. Sources: airflow-core/src/airflow/logging_config.py
System-to-Code Mapping
| Monitoring concern | Source-backed code surface | Operational meaning |
|---|---|---|
| Historical metric presentation | HistoricalMetrics export | UI route/component boundary for viewing historical metrics. |
| API action audit trail | `action_logging(event: str | None = None)` |
| Log-injection hardening | _sanitize_for_stdlib_log(value: str) -> str | Removes CR/LF from user-supplied values before plain-text stdlib logging. |
| Secret-safe audit fields | _mask_connection_fields(extra_fields) and _mask_variable_fields(extra_fields) | Prevents connection extras and variable values from being written in clear text. |
| Remote task log lookup | get_remote_task_log() | Lazily loads and returns the active remote task log handler. |
| Default remote log connection | get_default_remote_conn_id() | Reads [logging] remote_log_conn_id or the provider-resolved default. |
| Logging config import | _get_logging_config() -> dict[str, Any] | Imports and validates the configured logging dictionary. |
| Remote logging initialization | _load_logging_config() -> None | Resolves remote logging once and stores active state atomically. |
Metrics and Traces in Practice
When enabling metrics, start by deciding where Airflow should emit time-series data and how that data will be named. The official docs distinguish StatsD setup, OpenTelemetry setup, HTTPS support, allow/block lists, metric renaming, and other options, which implies that production deployments should standardize metric names before writing alerts. Airflow operators usually care about scheduler liveness, DAG processing latency, task state changes, executor pressure, and API/web responsiveness. The UI historical metrics page then gives a first-party view into selected historical behavior, while the external backend remains the durable monitoring store.
Traces solve a different problem: they explain flow. The official traces documentation highlights OpenTelemetry and adding custom spans in tasks. That is especially useful when a DAG task talks to external services or when a slow user action crosses the API server, database, and scheduler-facing state changes. Traces should be configured with the same privacy posture as logs because span attributes may contain identifiers or request metadata. The API logging decorator reinforces this design concern by redacting variable values and connection extras, demonstrating that observability data must be useful without leaking secrets. Sources: airflow-core/src/airflow/api_fastapi/logging/decorators.py
Airflow’s monitoring stack is also documentation-driven. The checked-in documentation architecture generator creates a diagram showing release managers, committers, Airflow GitHub repositories, a live docs S3 bucket, CloudFront, and the public https://airflow.apache.org site. For operators, this matters because monitoring behavior is split between implementation and official procedures: the health-check commands, metrics setup variants, and trace configuration guidance are expected to be consumed from the published docs. Keeping runbooks aligned with the stable docs is part of operating Airflow safely across upgrades. Sources: docs/images/documentation_architecture.py
Health Checks and Operational Runbooks
Health checks should be treated as component-specific probes rather than full correctness tests. The official docs identify webserver health, scheduler health, database checks, and Celery worker or cluster checks as distinct tasks. A passing health check generally says that a component can answer the probe; it does not prove that DAGs are scheduled quickly, all workers have capacity, or remote task logs are readable. For that reason, production monitoring should pair health checks with metrics alerts, trace sampling, and log access tests. This avoids a common failure mode where the service looks alive but user-visible workflow progress is degraded.
A practical runbook starts with a symptom and then selects the right evidence. For a blank or stale dashboard, check whether the web UI is reachable and whether the historical metrics page is available. For a task that is stuck or failed, inspect task logs and confirm the remote logging configuration has loaded as expected. For unexpected API-side changes, search action logs for the event name and user identity while relying on Airflow’s masking behavior for sensitive fields. For broad latency or scheduler concerns, use metrics and traces to decide whether the problem is component health, database pressure, executor capacity, or DAG parsing workload. Sources: airflow-core/src/airflow/ui/src/pages/Dashboard/HistoricalMetrics/index.ts, airflow-core/src/airflow/logging_config.py, airflow-core/src/airflow/api_fastapi/logging/decorators.py
API Components and Reference
The public-looking entry points in the requested source are small but operationally meaningful. action_logging(event: str | None = None) returns an async dependency-style logger for FastAPI actions; it derives the event name from the explicit argument or endpoint name, handles absent users as anonymous, and prepares request body data for logging. _sanitize_for_stdlib_log(value: str) -> str removes newline characters from user-controlled values. _mask_connection_fields(extra_fields) masks connection extra content by key presence, and _mask_variable_fields(extra_fields) masks val and value unconditionally. Sources: airflow-core/src/airflow/api_fastapi/logging/decorators.py
The logging configuration helpers form the reference surface for task log resolution. get_remote_task_log() lazily loads configuration if needed and returns the active remote log handler or None. get_default_remote_conn_id() first honors [logging] remote_log_conn_id, then falls back to the provider-resolved default. _get_logging_config() imports [logging] logging_config_class, validates that it is a dictionary, and raises an import error with context if loading fails. _load_logging_config() resolves remote logging through ProvidersManager and stores the result atomically on _ActiveLoggingConfig. Sources: airflow-core/src/airflow/logging_config.py
Next Steps
Use this page as the operational bridge between Airflow’s monitoring docs and the source modules that make monitoring evidence trustworthy. For setup instructions, follow the stable documentation pages for metrics, traces, and health checks because they enumerate supported backends and component probes. For implementation-level troubleshooting, read the logging architecture and task logs pages next, then inspect airflow-core/src/airflow/logging_config.py when remote logs do not resolve and airflow-core/src/airflow/api_fastapi/logging/decorators.py when audit events, masking, or API action logs are central to the incident.