Logging Architecture
Purpose and Scope
Airflow logging is an operational architecture concern rather than a single isolated subsystem. Task logs are produced while tasks execute, read back by operators through the API server and UI, and often stored outside the local process that produced them. The official documentation separates this area into logging and monitoring architecture, task logging, advanced logging configuration, metrics, traces, callbacks, health checks, and error tracking. This page focuses on the logging architecture layer: which Airflow components participate, where log-producing work happens, how readers reach logs, and how provider extensions fit into that model.
The repository evidence for this page is a set of source files that generate architecture diagrams. Those diagrams identify the runtime actors that shape logging behavior: API server processes, schedulers, workers, triggerers, DAG processors, the metadata database, DAG files, plugins, installed packages, auth managers, and the documentation publishing flow. They do not define handler classes directly, but they ground the component topology used by the logging documentation. When reading the task logging guide, map every configuration choice back to this topology: a handler runs in the component that emits or serves logs, and a UI request reaches the component boundary through the API server.
Sources: airflow-core/docs/img/diagram_basic_airflow_architecture.py, airflow-core/docs/img/diagram_distributed_airflow_architecture.py, airflow-core/docs/img/diagram_dag_processor_airflow_architecture.py
Relevant Source Files
- docs/images/documentation_architecture.py — Generates the documentation-site architecture image, including repository, package-doc publishing, S3-backed live docs, CloudFront cache, and the live Airflow website. It matters here because the logging pages themselves are published through this documentation flow.
- airflow-core/docs/img/diagram_auth_manager_airflow_architecture.py — Generates the auth-manager architecture image, showing a user reaching webserver processes, provider auth-manager code, and the core auth-manager interface. It grounds the access-control boundary for viewing operational pages such as logs.
- airflow-core/docs/img/diagram_basic_airflow_architecture.py — Generates the basic Airflow architecture image with user-authored DAG files, scheduler, metadata database, API server, plugins, and installed packages. It is the smallest topology needed to explain where logs are produced and where users read them.
- airflow-core/docs/img/diagram_dag_processor_airflow_architecture.py — Generates the DAG-processor architecture image with separated security perimeters for scheduling/UI and DAG-code execution, including workers, triggerers, DAG processors, and API servers. It is important when explaining which components may execute user code and therefore emit user-controlled logs.
- airflow-core/docs/img/diagram_distributed_airflow_architecture.py — Generates the distributed Airflow architecture image with schedulers, workers, triggerers, API servers, metadata database, DAG files, plugins, and installed packages. It grounds multi-process and multi-host log routing concerns.
System-to-Code Mapping
The basic diagram shows an Airflow user authoring DAG files, a scheduler reading those files, an API server presenting the UI, and the metadata database shared between scheduler and API server. It also shows plugins and installed packages being installed for both scheduler and API server. For logging, this means the UI is not the place where most task log lines originate. Instead, the UI is a reader and coordinator. The logs usually come from execution-time components, while the API server gives operators a controlled way to find, render, search, download, or link to those logs.
The distributed architecture expands that model by adding workers and triggerers next to schedulers. This is where the task logging guide becomes operationally important. A worker that runs a Python task, a triggerer that manages deferred asynchronous work, and scheduler-side orchestration can all produce useful diagnostic output, but task logs are primarily associated with task execution. In a distributed deployment, a reader may not be on the same machine as the writer. The architecture therefore needs a log storage or serving strategy that survives process boundaries, host boundaries, retries, and task-state transitions.
The DAG processor architecture adds a security dimension that is directly relevant to log trust. It divides the environment into a perimeter with no DAG code execution for scheduling and UI, and a perimeter with DAG code execution for workers, triggerers, DAG processors, and DAG files. Logs generated from DAG code should be treated as workload output, not as privileged control-plane truth. This distinction helps administrators decide where to expose logs, how to mask secrets, and how to troubleshoot parser or task failures without granting broad access to execution hosts.
Sources: airflow-core/docs/img/diagram_basic_airflow_architecture.py, airflow-core/docs/img/diagram_distributed_airflow_architecture.py, airflow-core/docs/img/diagram_dag_processor_airflow_architecture.py
| Source-level item | Kind | Logging architecture meaning |
|---|---|---|
| generate_basic_airflow_diagram | Diagram generator function | Defines the compact topology of Airflow user, DAG files, scheduler, metadata database, API server, plugins, and installed packages. |
| generate_distributed_airflow_diagram | Diagram generator function | Adds workers and triggerers, making remote task-log collection and serving part of the operational model. |
| generate_dag_processor_airflow_diagram | Diagram generator function | Separates non-DAG-code scheduling/UI components from DAG-code execution components, which is important for log trust boundaries. |
| generate_auth_manager_airflow_diagram | Diagram generator function | Shows user access through webserver processes and provider auth-manager code, grounding authorization around log viewing. |
| generate_documentation_architecture_diagram | Diagram generator function | Shows how Airflow documentation, including logging documentation, is published from repositories to the live site. |
| graph_attr and edge_attr | Diagram configuration dictionaries | Record layout choices such as direction, splines, concentration, and edge spacing; useful when regenerating or reviewing the architecture images. |
| PYTHON_MULTIPROCESS_LOGO, PACKAGES_IMAGE, DATABASE_IMAGE, MULTIPLE_FILES_IMAGE | Image constants | Represent the recurring component types that appear in the architecture: processes, packages, database, and DAG files. |
Runtime Logging Flow
A useful way to reason about Airflow logging is to start with the person reading the log. The operations user interacts with API server processes, and those processes are shown as the UI-facing entry point in the diagrams. The reader wants to understand why a task ran, failed, deferred, retried, or produced unexpected output. The API server can display task state and related metadata because it is connected to the metadata database. The log bytes themselves may be local, remotely stored, streamed, or served from another component, depending on the configured logging backend.
The writer side begins where work runs. In the distributed diagram, workers are connected with schedulers through an executor relationship and with the metadata database through dotted state-oriented edges. Triggerers are also present, reflecting Airflow’s support for deferred work that waits asynchronously. A task log handler must therefore behave well when the component that emitted a line is not the component that later serves it. Administrators should design logging so that restarting workers, replacing pods, or scaling triggerers does not make historical task logs disappear during incident review.
DAG parsing introduces another log category that can be confused with task execution. The DAG processor diagram shows DAG processors in the perimeter that executes DAG code, separate from API servers and schedulers in the no-DAG-code perimeter. Parse-time errors and import-time warnings can originate before any task instance runs. Those messages help authors fix DAG files, but they are architecturally different from task logs emitted by a running task. This is why a complete troubleshooting workflow should check both DAG-processor health and task-instance logs instead of assuming every problem is visible in only one log stream.
Sources: airflow-core/docs/img/diagram_distributed_airflow_architecture.py, airflow-core/docs/img/diagram_dag_processor_airflow_architecture.py
Configuration and Extension Reference
The official logging documentation exposes several reader tasks: configuring logging, writing to task logs from task code, grouping log lines, interleaving logs, troubleshooting missing logs, advanced configuration, serving logs from workers and triggerer, implementing a custom file task handler, and adding external links. The provider documentation lists community task-log handlers such as OSSTaskHandler, S3TaskHandler, CloudwatchTaskHandler, HdfsTaskHandler, ElasticsearchTaskHandler, GCSTaskHandler, StackdriverTaskHandler, and Azure-oriented handlers. These names are important because they show the extension model: core Airflow defines the logging surfaces, while providers can supply storage-specific implementations.
In practical terms, a deployment chooses where task logs land and how readers retrieve them. Local development can rely on local file-oriented behavior, but distributed production usually needs remote or component-aware serving. If task execution happens on workers and triggerers that may be ephemeral, then storing logs only on local disks can make investigation fragile. A remote handler supplied by a provider lets the workload component write logs to object storage, a search service, or a cloud logging system. The API server can then present a stable reader experience without requiring every old worker process to remain alive.
| Area | Concrete names or options from evidence | Operational guidance |
|---|---|---|
| Task log configuration | Configuring logging; Advanced logging configuration | Choose a storage and serving pattern that matches the deployment topology rather than only the local developer experience. |
| Task code output | Writing to task logs from your code; Grouping of log lines; Interleaving of logs | Treat task logs as workload output that may be merged, grouped, or interleaved when multiple execution streams contribute lines. |
| Serving path | Serving logs from workers and triggerer | Verify that readers can reach logs produced by execution components, especially in distributed or ephemeral environments. |
| Custom handler contract | Implementing a custom file task handler | Extend logging by implementing handler behavior compatible with Airflow task-log reading and writing expectations. |
| Provider handlers | OSSTaskHandler, S3TaskHandler, CloudwatchTaskHandler, HdfsTaskHandler, ElasticsearchTaskHandler, GCSTaskHandler, StackdriverTaskHandler | Prefer provider handlers when the log destination is an integration-specific storage or observability service. |
| Access boundary | User, Webserver or API Server, auth manager, core auth-manager interface | Ensure log viewing follows the same authenticated and authorized UI/API path as other operational actions. |
Sources: airflow-core/docs/img/diagram_auth_manager_airflow_architecture.py, airflow-core/docs/img/diagram_distributed_airflow_architecture.py
Security, Access, and Documentation Signals
The auth-manager diagram shows the user accessing webserver processes and those processes asking a provider auth manager whether the user is authorized, with that provider implementation inheriting from a core auth-manager interface. For logging, the main lesson is that log visibility is part of the operational console, not merely a filesystem permission. Logs can contain sensitive context, rendered parameters, exception details, and links to external systems. Administrators should combine proper authentication, role design, secret masking, and careful handler configuration before exposing logs broadly to teams or automation.
The documentation architecture diagram is also relevant because it explains how these logging guides reach users. It shows the Airflow repository publishing package docs, the Airflow site repository publishing the website, S3-backed live documentation, a CloudFront live cache, and the public Airflow website. That publishing path matters for maintainers: architecture images and official logging pages are not decorative extras. They are part of the supported operational contract that users consult when configuring production deployments, troubleshooting log retrieval, or choosing provider logging implementations.
Sources: airflow-core/docs/img/diagram_auth_manager_airflow_architecture.py, docs/images/documentation_architecture.py
Implementation Guidance and Next Steps
When implementing or reviewing a logging setup, first identify the component that writes each log category. Task execution points to workers and, for deferred work, triggerers. DAG import and parsing diagnostics point to DAG processors. Scheduler and API server logs support control-plane operations. Next identify where each category is stored and which process serves it back to the user. Finally, check the authorization and documentation path: the person reading the log should use the supported API/UI surface, and the chosen handler should be documented well enough for operations staff to recover logs during an outage.
A good next step is to pair this architecture page with task-log documentation and deployment documentation. Task-log pages explain user-facing details such as grouping, interleaving, troubleshooting, external links, and custom handlers. Deployment pages explain why local files, containers, Kubernetes pods, remote object storage, and provider integrations behave differently. If logs disappear, appear incomplete, or require direct host access, return to the diagrams: the usual cause is a mismatch between where execution occurs, where log bytes are persisted, and where the API server is expected to retrieve them.