Scheduler

Purpose and Scope

The Airflow scheduler is the runtime component that turns authored DAG definitions into scheduled work. A DAG defines dependencies and scheduling intent, but the scheduler is the process that repeatedly evaluates that intent against the metadata database, creates or advances DAG runs, and decides when task instances are ready to be handed to an executor. For operators running Airflow, this page explains where the scheduler lives in the source tree, how it is started, how it is deployed in Kubernetes through the Helm chart, and what operational controls are exposed by the checked-in code.

The official scheduler documentation frames this component as an administration and deployment topic, not just a DAG-authoring concept. That distinction matters: scheduler behavior depends on database health, executor behavior, task log serving, DAG file processing, and the number of scheduler replicas in a deployment. In this repository, that operational boundary is reflected by separate code for the CLI command, the job runner, chart templates, and database migrations. The scheduler should therefore be understood as a long-running service with lifecycle, health, and deployment concerns, rather than a library function invoked by DAG code.

Sources: airflow-core/src/airflow/cli/commands/scheduler_command.py, airflow-core/src/airflow/jobs/scheduler_job_runner.py, chart/templates/scheduler/scheduler-deployment.yaml

Relevant Source Files

airflow-core/src/airflow/cli/commands/scheduler_command.py - Defines the airflow scheduler command path, including process startup, daemon handling, hot reload handling, health-check serving, and optional task-log serving.
airflow-core/src/airflow/jobs/scheduler_job_runner.py - Contains the scheduler job runner class used by the CLI entry point to execute scheduler work.
chart/templates/scheduler/scheduler-deployment.yaml - Defines how the Helm chart renders the scheduler as a Kubernetes Deployment or StatefulSet, including replicas, labels, scheduling constraints, security contexts, and executor-dependent behavior.
airflow-core/src/airflow/migrations/versions/0043_3_0_0_remove_scheduler_lock_column.py - Records a scheduler-related metadata schema change for Airflow 3.0.0 by removing the legacy dag.scheduler_lock column and restoring it on downgrade.
docs/images/documentation_architecture.py - Generates the documentation architecture diagram used to publish Airflow package documentation, which is relevant when mapping the official scheduler documentation page back to repository artifacts.

System-to-Code Mapping

The CLI entry point is the most direct way to see how the scheduler service is assembled. The scheduler function is decorated as an Airflow CLI action and with provider configuration loading, then prints the Airflow banner, validates command options, optionally runs under the hot-reload wrapper, and finally delegates process management to run_command_with_daemon_option. The actual scheduler work is isolated in _run_scheduler_job, which constructs SchedulerJobRunner(job=Job(), num_runs=args.num_runs, only_idle=args.only_idle) and passes the runner's execution callable to run_job. This separation keeps command-line concerns, process lifecycle, and scheduling execution distinct.

Sources: airflow-core/src/airflow/cli/commands/scheduler_command.py, airflow-core/src/airflow/jobs/scheduler_job_runner.py

Two helper context managers make the scheduler process more than just a tight scheduling loop. _serve_logs imports the configured default executor class and starts a child process for serve_logs when the executor declares serve_logs support and the user has not opted out with skip_serve_logs. _serve_health_check starts a separate health-check subprocess when the [scheduler] ENABLE_HEALTH_CHECK configuration value is enabled. Both helpers terminate their subprocesses in finally blocks, which makes shutdown behavior part of the scheduler command contract rather than an incidental side effect.

Sources: airflow-core/src/airflow/cli/commands/scheduler_command.py

The Helm chart maps those service-level concerns into Kubernetes primitives. The scheduler template is gated by .Values.scheduler.enabled, names the workload with the scheduler component label, and sets replicas from .Values.scheduler.replicas. It chooses StatefulSet instead of Deployment when the executor contains Local and Celery worker persistence is enabled, with an inline comment explaining that in local mode the scheduler assumes the role of the worker. The template also computes node selectors, affinity, tolerations, topology spread constraints, revision history, pod security context, container security context, lifecycle hooks, and log-groomer security context from scheduler-specific values or global chart defaults.

Sources: chart/templates/scheduler/scheduler-deployment.yaml

Execution Flow

A typical scheduler startup begins when an operator runs the Airflow CLI command or a container entrypoint invokes it inside a scheduler pod. The CLI validates that --only-idle is only used with a positive --num-runs, because an idle-only scheduler pass must have a bounded number of runs to make sense. It then decides whether hot reload is enabled for the invocation. Without hot reload, run_command_with_daemon_option is responsible for running the scheduler callback either in the foreground or with daemon-style process management, while also setting up logging for the scheduler process.

Sources: airflow-core/src/airflow/cli/commands/scheduler_command.py

Inside the callback, _run_scheduler_job first calls set_component_mp_start_method("scheduler"), which identifies scheduler-specific multiprocessing behavior before constructing the job runner. The runner receives a fresh Job object plus the bounded-run and idle-only options from the CLI namespace. The scheduler then enters nested context managers for log serving and health serving before invoking run_job. This order is important operationally: the auxiliary services are present while the scheduler job is executing, and they are torn down when the scheduler exits, even if execution stops because of a bounded run count, an error, or process shutdown.

Sources: airflow-core/src/airflow/cli/commands/scheduler_command.py, airflow-core/src/airflow/jobs/scheduler_job_runner.py

In Kubernetes deployments, the execution flow is wrapped by the rendered chart. The workload kind, replica count, scheduling placement rules, annotations, labels, and pod-level security controls all come from Helm values. Because the template has explicit executor-aware branching, a LocalExecutor-style installation can cause the scheduler workload to behave differently from a distributed executor installation. The template also notes that DAG mounts can be skipped on the scheduler when the DAG processor is enabled, except in local mode. That reflects the broader Airflow architecture in which parsing and scheduling can be separated for scale and isolation.

Sources: chart/templates/scheduler/scheduler-deployment.yaml

Operational Considerations

Running more than one scheduler is an official deployment topic, and the codebase shows why database-backed coordination matters. Scheduler instances are long-running jobs that operate through the metadata database and must coexist with schema evolution. The migration 0043_3_0_0_remove_scheduler_lock_column.py removes the scheduler_lock column from the dag table for Airflow 3.0.0 and defines a downgrade that adds the nullable boolean column back. That migration is a useful signal that scheduler coordination has evolved away from an older DAG-table lock field and that operators should treat scheduler upgrades as database migrations, not only container image changes.

Sources: airflow-core/src/airflow/migrations/versions/0043_3_0_0_remove_scheduler_lock_column.py

Scheduler performance tuning should start from the resources that the scheduler actually consumes: metadata database throughput, CPU for scheduling decisions, process slots for DAG-related work, executor responsiveness, and the number of scheduler replicas. The Helm chart exposes placement and scaling levers through .Values.scheduler.replicas, .Values.scheduler.nodeSelector, .Values.scheduler.affinity, .Values.scheduler.tolerations, and .Values.scheduler.topologySpreadConstraints. These settings do not change the scheduling algorithm, but they determine whether scheduler pods have predictable compute capacity and whether multiple replicas are spread across nodes or zones in a way that supports availability goals.

Sources: chart/templates/scheduler/scheduler-deployment.yaml

Health and observability should be configured deliberately. The CLI only starts the scheduler health-check subprocess when [scheduler] ENABLE_HEALTH_CHECK is true, so deployment probes must line up with Airflow configuration rather than assuming a health endpoint is always served. Likewise, task log serving from the scheduler depends on both the skip_serve_logs CLI option and the default executor class exposing serve_logs. In environments with remote logging, sidecars, or external log aggregation, operators should verify which component is expected to serve or collect task logs before relying on scheduler-local behavior.

Sources: airflow-core/src/airflow/cli/commands/scheduler_command.py, chart/templates/scheduler/scheduler-deployment.yaml

Compact Reference

Surface	Concrete names and behavior	Source
CLI command	`scheduler(args: Namespace)` starts Airflow Scheduler as a CLI action after provider configuration is loaded.	`airflow-core/src/airflow/cli/commands/scheduler_command.py`
Runner construction	`_run_scheduler_job` creates `SchedulerJobRunner(job=Job(), num_runs=args.num_runs, only_idle=args.only_idle)`.	`airflow-core/src/airflow/cli/commands/scheduler_command.py`
Bounded execution	`--only-idle` requires `--num-runs` to be positive; otherwise the command exits with `SystemExit`.	`airflow-core/src/airflow/cli/commands/scheduler_command.py`
Hot reload	`cli_utils.should_enable_hot_reload(args)` switches execution to `run_with_reloader(..., process_name="scheduler")`.	`airflow-core/src/airflow/cli/commands/scheduler_command.py`
Log serving	`_serve_logs(skip_serve_logs)` starts `serve_logs` in a subprocess when the default executor class has `serve_logs`.	`airflow-core/src/airflow/cli/commands/scheduler_command.py`
Health serving	`_serve_health_check(enable_health_check)` starts `serve_health_check` in a subprocess when enabled by config.	`airflow-core/src/airflow/cli/commands/scheduler_command.py`
Helm enablement	`.Values.scheduler.enabled` gates rendering of the scheduler workload.	`chart/templates/scheduler/scheduler-deployment.yaml`
Helm scaling	`.Values.scheduler.replicas` sets the scheduler workload replica count.	`chart/templates/scheduler/scheduler-deployment.yaml`
Workload kind	The chart renders `StatefulSet` for the local-executor-plus-persistent-worker condition; otherwise it renders `Deployment`.	`chart/templates/scheduler/scheduler-deployment.yaml`
Schema migration	Revision `486ac7936b78` drops `dag.scheduler_lock` on upgrade and restores it on downgrade.	`airflow-core/src/airflow/migrations/versions/0043_3_0_0_remove_scheduler_lock_column.py`

Deployment Guidance and Next Steps

For a local or development setup, begin with the CLI behavior: decide whether the scheduler should run in the foreground, under daemon management, or under hot reload, and use --num-runs and --only-idle only when you intentionally want bounded scheduler execution. For production, start from the Helm chart values and confirm replica count, pod placement, security context, executor mode, log-serving assumptions, and health-check configuration. Then verify database migrations before upgrade, because scheduler behavior and metadata schema are coupled across Airflow releases.

Sources: airflow-core/src/airflow/cli/commands/scheduler_command.py, chart/templates/scheduler/scheduler-deployment.yaml, airflow-core/src/airflow/migrations/versions/0043_3_0_0_remove_scheduler_lock_column.py

Read this page alongside airflow-components for the role of scheduler relative to the API server, DAG processor, and triggerer; dag-file-processing for parser separation and DAG processing behavior; production-deployment for operational readiness; kubernetes-and-helm for chart-level deployment patterns; and metrics-traces-and-health-checks for monitoring and probe configuration. If you are tuning throughput, also review pools-and-priority-weights, because scheduler decisions are constrained by pool slots, priorities, and executor capacity as well as by scheduler process resources.

Ask a question...