Production Deployment
Purpose and Scope
Production deployment in Airflow means operating the platform as a multi-component service rather than treating it as a single local process. The official documentation places production deployment under Administration and Deployment, alongside database backend, multi-node cluster, logging, configuration, scheduler uptime, production container images, Helm chart for Kubernetes, live upgrades, Kerberos-authenticated workers, and cloud service-access topics. In practice, that means a production plan must cover component separation, database readiness, rolling updates, authentication, observability, and safe runtime customization. This page focuses on how those concerns show up in the repository evidence, especially the Helm chart templates used to run Airflow components on Kubernetes.
Sources: chart/templates/api-server/api-server-deployment.yaml, chart/templates/dag-processor/dag-processor-deployment.yaml, chart/templates/flower/flower-deployment.yaml
Airflow’s Kubernetes chart models production as separate Deployments for distinct responsibilities. The API server is its own scalable service, the DAG processor is deployed separately for parsing and processing DAG files, and Flower is conditionally deployed only for CeleryExecutor environments. This separation matters operationally because each workload has a different scaling profile, readiness expectation, security posture, and upgrade risk. A production Airflow deployment should therefore start by deciding which components are required, which executor is in use, and how each component should be scheduled, secured, rolled, and monitored.
Sources: chart/templates/api-server/api-server-deployment.yaml, chart/templates/dag-processor/dag-processor-deployment.yaml, chart/templates/flower/flower-deployment.yaml
Relevant Source Files
docs/images/documentation_architecture.py— Generates an architecture diagram for the public Airflow documentation publishing path, including GitHub repositories, release manager actions, S3-hosted package docs, CloudFront caching, and the publicairflow.apache.orgsite.chart/kustomize-overlays/kerberos/kdc-deployment.yaml— Provides a Kerberos KDC overlay explicitly marked as throwaway test infrastructure, with comments that warn against using it for production workloads.chart/templates/api-server/api-server-deployment.yaml— Defines the Helm template for the Airflow API server Deployment, including enablement, replicas, rolling update strategy, scheduling controls, security contexts, service metadata, and checksum annotations.chart/templates/dag-processor/dag-processor-deployment.yaml— Defines the Helm template for the DAG processor Deployment, including replicas, scheduling controls, security contexts, checksum annotations, optional eviction annotation, and wait-for-migrations context.chart/templates/flower/flower-deployment.yaml— Defines the Helm template for Flower, gated by Flower enablement and a CeleryExecutor value, with its own service account, security context, scheduling options, and configuration checksum annotations.
Production Readiness Model
The first production decision is topology. The API server template is wrapped in an apiServer.enabled condition and uses .Values.apiServer.replicas unless an HPA is enabled, which makes it clear that the chart supports both fixed replica counts and autoscaled API capacity. Its default strategy is a rolling update with maxSurge: 1 and maxUnavailable: 0, a production-oriented choice that preserves serving capacity during upgrades. Operators can override the strategy, but the checked-in default communicates the expected operational posture: keep user-facing API capacity available while replacing pods.
Sources: chart/templates/api-server/api-server-deployment.yaml
The second decision is workload placement. The API server, DAG processor, and Flower templates each compute component-specific nodeSelector, affinity, tolerations, and topologySpreadConstraints, falling back to global chart values when component values are not provided. This pattern lets a production cluster express broad scheduling policy once while still allowing exceptions for specific Airflow workloads. For example, DAG processing can be isolated from API-serving pods, Flower can be scheduled only where Celery monitoring should run, and topology spread constraints can reduce correlated failures across nodes or zones.
Sources: chart/templates/api-server/api-server-deployment.yaml, chart/templates/dag-processor/dag-processor-deployment.yaml, chart/templates/flower/flower-deployment.yaml
The third decision is upgrade and configuration propagation. The API server and DAG processor templates include checksum annotations for metadata connection secrets, PgBouncer configuration, and Airflow configuration. The DAG processor also includes checksums for extra config maps and extra secrets, while Flower includes checksums for the Airflow configuration and Flower secret. These checksum annotations are a deployment signal: when underlying config or secret templates change, Kubernetes sees the pod template as changed and rolls the workload. Production operators should account for this behavior when changing credentials, database connectivity, or component-specific configuration.
Sources: chart/templates/api-server/api-server-deployment.yaml, chart/templates/dag-processor/dag-processor-deployment.yaml, chart/templates/flower/flower-deployment.yaml
System-to-Code Mapping
| Production concern | Source-backed implementation signal | Operational interpretation |
|---|---|---|
| API availability | apiServer.enabled, apiServer.replicas, optional apiServer.hpa.enabled, and default rolling update fields in chart/templates/api-server/api-server-deployment.yaml | Size API serving capacity deliberately and avoid upgrade strategies that remove all available API pods at once. |
| DAG processing capacity | dagProcessor.enabled, dagProcessor.replicas, and component scheduling values in chart/templates/dag-processor/dag-processor-deployment.yaml | Treat DAG parsing and processing as its own workload with independent placement, resources, and rollout planning. |
| Celery monitoring | Flower deployment gated by flower.enabled and contains "CeleryExecutor" .Values.executor in chart/templates/flower/flower-deployment.yaml | Deploy Flower only when the executor architecture makes it useful and secure it as a separate operational surface. |
| Secure pod defaults | airflowPodSecurityContext and containerSecurityContext includes across component templates | Apply pod-level and container-level security settings consistently, with component overrides where required. |
| Config changes | checksum/* annotations in component pod templates | Expect configuration and secret changes to trigger pod replacement, and schedule such changes like deployments. |
| Kerberos testing | Explicit warning in chart/kustomize-overlays/kerberos/kdc-deployment.yaml | Do not reuse the sample in-cluster KDC as production Kerberos infrastructure. |
Kubernetes and Helm Deployment Flow
A practical Helm-based production flow starts by enabling only the components required by the target architecture. The API server is the user-facing service and should be sized for web and REST API demand. The DAG processor should be sized for DAG volume, parsing cost, and file-distribution strategy. Flower should be enabled only for CeleryExecutor deployments, because the template itself checks both .Values.flower.enabled and whether the executor value contains CeleryExecutor. This conditional design prevents a monitoring component from being deployed in executor modes where it does not apply.
Sources: chart/templates/api-server/api-server-deployment.yaml, chart/templates/dag-processor/dag-processor-deployment.yaml, chart/templates/flower/flower-deployment.yaml
After selecting components, define rollout expectations. The API server template documents a default rolling update strategy and explains the role of maxSurge and maxUnavailable, including that setting maxUnavailable to zero preserves capacity during the update. That comment is worth treating as production guidance rather than incidental YAML. User-facing components should be upgraded in a way that maintains enough pods to serve traffic, while background components should be rolled with awareness of parsing lag, migration dependencies, and the impact of restarting pods that own in-memory work.
Sources: chart/templates/api-server/api-server-deployment.yaml, chart/templates/dag-processor/dag-processor-deployment.yaml
Then set placement and identity. Each component template exposes or derives node selection, affinity, tolerations, topology spread constraints, service account names, image pull secrets, restart policy, and security context. In production, those fields connect Airflow architecture to cluster governance: which nodes may run Airflow, which service account a component uses, what image registry credentials it needs, and which pod security constraints apply. These values also provide a clean boundary between chart defaults and environment-specific policy, letting platform teams enforce cluster rules without editing component templates.
Sources: chart/templates/api-server/api-server-deployment.yaml, chart/templates/dag-processor/dag-processor-deployment.yaml, chart/templates/flower/flower-deployment.yaml
Finally, plan for operational change. Config and secret checksum annotations are repeated across workloads because Airflow production behavior depends heavily on database configuration, broker connectivity, web/API settings, and provider credentials. A change to metadata connection configuration or Airflow config can roll multiple components. Treat those changes as coordinated deployment events: verify database migrations, confirm connection pool capacity, observe component health, and communicate expected restarts. The DAG processor template’s wait-for-migrations security-context variable also reflects the common production need to coordinate component startup with metadata database state.
Sources: chart/templates/api-server/api-server-deployment.yaml, chart/templates/dag-processor/dag-processor-deployment.yaml, chart/templates/flower/flower-deployment.yaml
Security and Sensitive Deployment Concerns
The Kerberos overlay is an important negative example. It creates an in-cluster MIT Kerberos KDC for testing the overlay pattern, but the file comments explicitly say not to point production workloads at it. The warning explains why: the admin password is a fixed literal, and the database lives in an emptyDir. It also notes that the image should be pinned to an immutable digest before use outside a local kind cluster. Production Kerberos deployments should therefore use real infrastructure, managed credentials, durable state, and pinned artifacts instead of copying the test overlay unchanged.
Sources: chart/kustomize-overlays/kerberos/kdc-deployment.yaml
The same overlay also demonstrates a readiness principle that applies beyond Kerberos. Its readiness probe uses a TCP socket on port 88 because checking only that a principal file existed could mark the pod ready before the KDC was actually accepting connections. The comment describes a failure mode where kinit immediately after readiness could hit a closed port. For production Airflow dependencies, readiness should verify the service behavior that clients depend on, not merely the presence of initialization artifacts. A pod that is initialized is not necessarily ready to serve.
Sources: chart/kustomize-overlays/kerberos/kdc-deployment.yaml
Security also appears in the main chart templates through service accounts and security contexts. The Flower template sets serviceAccountName from a Flower-specific helper, enables or disables service links from chart values, and applies pod and container security context helpers. The API server and DAG processor templates similarly derive pod and container security contexts, including separate wait-for-migrations contexts. In production, this design encourages least-privilege identities and component-specific hardening rather than a single shared runtime identity for every Airflow pod.
Sources: chart/templates/api-server/api-server-deployment.yaml, chart/templates/dag-processor/dag-processor-deployment.yaml, chart/templates/flower/flower-deployment.yaml
Operational Documentation and Release Awareness
The repository also includes a generated documentation architecture diagram that shows how Airflow documentation moves from GitHub repositories to live public documentation. The diagram code represents the Apache Airflow repo, the Apache Airflow site repo, release-manager actions, manual package-doc publishing, an S3 bucket for live docs, CloudFront caching, and the public https://airflow.apache.org web server. For production operators, this matters because Airflow deployment guidance is versioned product documentation, not only chart YAML. Match the deployed Airflow version to the corresponding documentation version before applying upgrade, migration, or security instructions.
Sources: docs/images/documentation_architecture.py
This is especially important for live upgrades and production container-image decisions. The official documentation navigation places Production Deployment near database backend, multi-node cluster, logging, scheduler uptime, production container images, Helm chart for Kubernetes, and live-upgrading Airflow. The chart templates show how those topics become concrete Kubernetes mechanisms: Deployments, replicas, scheduling rules, security contexts, checksum-triggered rollouts, service accounts, and executor-specific optional components. Use the docs for sequencing and policy, then use the chart values and rendered manifests to confirm exactly what will run in the cluster.
Sources: docs/images/documentation_architecture.py, chart/templates/api-server/api-server-deployment.yaml, chart/templates/dag-processor/dag-processor-deployment.yaml, chart/templates/flower/flower-deployment.yaml
Production Checklist and Next Steps
Before promoting an Airflow installation to production, verify that each enabled component has an intentional replica count or autoscaling policy, an upgrade strategy, resource requests and limits, placement constraints, service account, image pull policy, and security context. Confirm that config and secret changes trigger the rollouts you expect, and rehearse those rollouts in a staging environment. If you use CeleryExecutor, decide whether Flower is required and protect it as an operational dashboard. If you use Kerberos, replace the sample KDC overlay with production-grade Kerberos infrastructure and durable credential management.
Sources: chart/kustomize-overlays/kerberos/kdc-deployment.yaml, chart/templates/api-server/api-server-deployment.yaml, chart/templates/dag-processor/dag-processor-deployment.yaml, chart/templates/flower/flower-deployment.yaml
Recommended next pages are kubernetes-and-helm for chart installation and values, metadata-database-and-migrations for database readiness and migration planning, scheduler and dag-file-processing for runtime behavior, logging-architecture and metrics-traces-and-health-checks for observability, and security-model for trust boundaries and hardening. Read those together rather than treating production deployment as a single install command. A production Airflow system is a coordinated set of services, persistent state, credentials, configuration, and operational procedures, and the checked-in chart templates are the implementation surface where many of those decisions become enforceable.