Airflow Components

Purpose and Scope

Apache Airflow is operated as a set of cooperating runtime components rather than as a single long-running process. The important boundary for operators is that each component has a distinct responsibility, lifecycle, and deployment shape: the scheduler decides what can run, the API server exposes user and integration surfaces, the DAG file processor parses workflow definitions, and the triggerer handles asynchronous waiting for deferrable workloads. This page explains that component model from the repository perspective, so readers can connect official deployment guidance to the concrete packages, commands, templates, and migrations that make those services work.

Sources: airflow-core/src/airflow/cli/commands/scheduler_command.py, airflow-core/src/airflow/jobs/scheduler_job_runner.py

The source tree reinforces the component split by keeping core runtime behavior under airflow-core, while deployment-specific packaging lives elsewhere. The scheduler entry point is a Python CLI command that constructs a scheduler job runner and starts supporting subprocesses when configured. The Helm chart then turns that runtime concern into Kubernetes objects with labels, replicas, pod security context, node placement, and executor-sensitive behavior. A migration that removes the old scheduler_lock column shows that component boundaries also affect metadata-database evolution, not just process startup or Kubernetes YAML.

Sources: chart/templates/scheduler/scheduler-deployment.yaml, airflow-core/src/airflow/migrations/versions/0043_3_0_0_remove_scheduler_lock_column.py

Relevant Source Files

• docs/images/documentation_architecture.py - Generates an architecture diagram for Airflow documentation publishing; it is useful here mainly as a contrast between documentation-site architecture and runtime component architecture.
• chart/templates/scheduler/scheduler-deployment.yaml - Helm template for deploying the scheduler as a Kubernetes Deployment or StatefulSet, depending on executor and persistence settings.
• airflow-core/src/airflow/cli/commands/scheduler_command.py - CLI entry point that starts the scheduler, configures multiprocessing behavior, delegates to SchedulerJobRunner, and optionally serves logs and health checks.
• airflow-core/src/airflow/jobs/scheduler_job_runner.py - Scheduler job runner implementation path used by the CLI entry point to execute scheduling work.
• airflow-core/src/airflow/migrations/versions/0043_3_0_0_remove_scheduler_lock_column.py - Alembic migration for Airflow 3.0.0 that removes and restores the dag.scheduler_lock column across upgrade and downgrade paths.

Component Model

An Airflow deployment is easiest to reason about when each process is treated as a service with a contract. The scheduler contract is to evaluate DAG state and coordinate task execution through the metadata database and configured executor. The API server contract is to present control-plane access for users and clients. The DAG processor contract is to load and parse Python DAG files in a controlled environment. The triggerer contract is to run asynchronous trigger workloads so deferrable operators can wait without occupying normal worker slots. Those contracts are separable even when a local development installation runs several of them together.

Sources: airflow-core/src/airflow/cli/commands/scheduler_command.py, airflow-core/src/airflow/jobs/scheduler_job_runner.py

The scheduler source makes that contract concrete. scheduler(args: Namespace) is the CLI action for starting the scheduler. It prints the Airflow banner, validates that --only-idle is only used with a positive --num-runs, supports hot reload when the CLI utilities enable it, and otherwise delegates daemon handling to run_command_with_daemon_option. The actual scheduler process path calls _run_scheduler_job(args), sets the multiprocessing start method for the scheduler component, creates SchedulerJobRunner(job=Job(), num_runs=args.num_runs, only_idle=args.only_idle), and invokes run_job with the runner’s execute callable.

Sources: airflow-core/src/airflow/cli/commands/scheduler_command.py

Supporting services are started around the scheduler job rather than hidden inside the job runner. _serve_logs(skip_serve_logs=False) imports the default executor class and starts a serve_logs subprocess only when the executor declares that it serves logs and the caller has not skipped log serving. _serve_health_check(enable_health_check=False) starts a separate health-check subprocess when the scheduler configuration enables scheduler.ENABLE_HEALTH_CHECK. These context managers terminate their subprocesses on exit, which gives the scheduler command a clear lifecycle for auxiliary runtime endpoints.

Sources: airflow-core/src/airflow/cli/commands/scheduler_command.py

Deployment Mapping

The Helm scheduler template translates the scheduler component into Kubernetes primitives. It is gated by .Values.scheduler.enabled, labels the workload with tier: airflow and component: scheduler, and derives the workload name from the chart’s full Airflow name plus -scheduler. It also records the executor value as a normalized Kubernetes label. That labeling matters operationally because schedulers are usually monitored, scaled, and debugged independently from workers, web/API services, DAG processors, and triggerers, even though they share the same Airflow metadata database and configuration.

Sources: chart/templates/scheduler/scheduler-deployment.yaml

The template also captures an important deployment distinction: in Local executor mode, the scheduler assumes the role of the worker. The chart computes $local from the executor value and then decides whether the scheduler should be a StatefulSet when local mode combines with Celery worker persistence settings. It also computes $localOrDagProcessorDisabled, allowing DAG mounts to be skipped on the scheduler when a separate DAG processor is enabled, except in local mode. This is a practical example of how the component model changes deployment topology without changing the user-facing concept of a scheduler.

Sources: chart/templates/scheduler/scheduler-deployment.yaml

Kubernetes concerns such as replicas, revisionHistoryLimit, update strategy, pod security context, container security context, lifecycle hooks, node selectors, affinity, tolerations, and topology spread constraints are all surfaced through chart values. The scheduler therefore has both an Airflow runtime identity and a Kubernetes operations identity. When debugging a deployment, check whether the problem belongs to the Python process contract, such as CLI arguments or health-check subprocesses, or to the chart contract, such as placement, persistence, security context, or whether the scheduler is rendered at all.

Sources: chart/templates/scheduler/scheduler-deployment.yaml, airflow-core/src/airflow/cli/commands/scheduler_command.py

Metadata and Runtime Evolution

Airflow components coordinate through metadata, so schema migrations can change how components interact. The migration 0043_3_0_0_remove_scheduler_lock_column.py declares revision 486ac7936b78, down revision d59cbbef95eb, and airflow_version = "3.0.0". Its upgrade path batch-alters the dag table and drops the scheduler_lock column. Its downgrade path adds the column back as a nullable SQLAlchemy boolean. This is a narrow change, but it is operationally meaningful because scheduler coordination state historically lived in the database schema.

Sources: airflow-core/src/airflow/migrations/versions/0043_3_0_0_remove_scheduler_lock_column.py

For maintainers and operators, this migration is a reminder that component architecture is not only about processes. Scheduler behavior, DAG parsing, API operations, and asynchronous trigger handling all depend on shared persistent state. Before upgrading Airflow, review migrations as part of the same readiness process as chart values and command-line entry points. If a deployment has multiple schedulers or customized database automation, schema changes that touch scheduler-facing columns deserve special attention during staging validation and rollback planning.

Sources: airflow-core/src/airflow/migrations/versions/0043_3_0_0_remove_scheduler_lock_column.py

Documentation Architecture Versus Runtime Architecture

The repository also contains code that generates diagrams for Airflow’s documentation publishing architecture. documentation_architecture.py uses the diagrams package to create a left-to-right diagram with clusters such as Airflow GitHub repos and Live Docs, nodes for apache-airflow, apache-airflow-site, S3, CloudFront, and the live website. That diagram is about how documentation and sites are published, including release-manager and committer paths. It should not be read as a diagram of scheduler, API server, DAG processor, or triggerer runtime traffic.

Sources: docs/images/documentation_architecture.py

This distinction is useful when navigating a large repository. Airflow has product runtime code, generated documentation assets, Helm deployment templates, provider packages, SDK packages, and development tooling in the same monorepo. A file under docs/images can accurately describe the documentation-delivery system while saying little about how tasks are scheduled. Conversely, a file under airflow-core/src/airflow/cli/commands can be a direct runtime entry point even though it is not a user guide. Treat file location and package purpose as part of the evidence when reading source.

Sources: docs/images/documentation_architecture.py, airflow-core/src/airflow/cli/commands/scheduler_command.py

Compact Reference

Surface	Concrete names	What to look for
Scheduler CLI	scheduler(args: Namespace), _run_scheduler_job(args)	Starts the scheduler command, validates CLI options, constructs SchedulerJobRunner, and runs the job.
Scheduler auxiliary processes	_serve_logs(skip_serve_logs), _serve_health_check(enable_health_check)	Starts log-serving and health-check subprocesses only when configured or supported.
Helm scheduler workload	.Values.scheduler.enabled, .Values.scheduler.replicas, .Values.executor, component: scheduler	Controls whether the scheduler workload is rendered and how it is labeled and scaled.
Local executor deployment branch	$local, $stateful, $localOrDagProcessorDisabled	Encodes scheduler-as-worker behavior and DAG-mount decisions when DAG processor separation is used.
Scheduler schema migration	upgrade(), downgrade(), dag.scheduler_lock	Removes the scheduler lock column on upgrade and restores it on downgrade.

Operational Reading Flow

When you need to understand an Airflow deployment, start from the component that owns the symptom. For a scheduling delay, read the scheduler command and job runner path first, then inspect the Helm scheduler template that actually launched the process. For health probes or log-serving behavior, check whether the scheduler command enabled the relevant subprocess and whether the executor supports served logs. For upgrade concerns, inspect metadata migrations that touch scheduler-facing tables before changing chart values or rolling multiple scheduler replicas.

Sources: airflow-core/src/airflow/cli/commands/scheduler_command.py, airflow-core/src/airflow/jobs/scheduler_job_runner.py, chart/templates/scheduler/scheduler-deployment.yaml, airflow-core/src/airflow/migrations/versions/0043_3_0_0_remove_scheduler_lock_column.py

Next, read the dedicated pages for the components that are only summarized here. The scheduler page should be used for scheduling loops, task selection, and operational behavior. The DAG file processing page should be used for parser isolation and DAG processor separation. The metrics, traces, and health checks page should be used for observability signals. The Kubernetes and Helm page should be used when the question is about rendered workloads, chart values, pod placement, persistence, or executor-specific deployment topology.