OpenWikiapache/airflow

Docker Compose Quick Start

Purpose and Scope

This page explains the local Docker Compose path for trying Apache Airflow without first designing a production deployment. The goal is to get the main Airflow services running on a developer workstation, understand what the quick-start environment is for, and know where it fits in the wider project. Airflow’s official quick start positions the first experience as bootstrapping a local instance; Docker Compose is the containerized variant of that same learning workflow, useful when you want the scheduler, API/UI, workers, and metadata database to behave like separate services instead of one local Python process.

The Docker Compose quick start should be treated as a learning and evaluation environment. It is convenient for running tutorial DAGs, testing simple operators, checking provider behavior, and observing how Airflow components cooperate. It is not a production architecture by itself: production deployments need durable configuration, explicit secrets handling, executor and database choices, monitoring, backups, and upgrade planning. The repository evidence available for this page also shows that Airflow’s documentation is published as a maintained product surface, so the compose workflow should be read together with the official docs for the Airflow version being evaluated.

Sources: docs/images/documentation_architecture.py

Quick Start Flow

A typical Compose-based local start begins by creating a clean working directory, downloading or copying the Airflow docker-compose.yaml for the selected Airflow documentation version, creating local mount points, and writing an .env file with the current user ID. The user ID step matters on Linux because containers write logs, DAG-generated files, and plugin artifacts into bind-mounted directories. After that, run the initialization service once, then start the full stack. The Airflow UI is normally reached at http://localhost:8080, and the default tutorial credentials in the documented quick start are intended only for local use.

mkdir airflow-docker
cd airflow-docker
curl -LfO https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml
mkdir -p ./dags ./logs ./plugins ./config
echo "AIRFLOW_UID=$(id -u)" > .env
docker compose up airflow-init
docker compose up

Once the services are up, the first useful task is not to customize everything at once. Open the UI, verify that example DAGs or tutorial DAGs are visible, trigger a small DAG run, and inspect the Grid and task logs. This validates that the scheduler can parse DAG files, the metadata database is reachable, and task execution produces retrievable logs. If a tutorial asks you to add a DAG file, place it under the mounted dags directory and wait for the DAG processor and scheduler to notice the change rather than rebuilding the image immediately.

The official Airflow quick start also calls out supported Python versions and installation methods for local Python installs. That matters even when using Compose because many users eventually move from “run Airflow” to “develop DAG code.” DAG code, custom plugins, and provider dependencies may still be edited and tested on the host, while the Compose services execute them inside containers. Keep the Airflow image version, constraints, and provider versions aligned with the documentation version you are following so that examples match the runtime you are actually running.

Core Primitives

The local Compose stack is easiest to understand as a small Airflow deployment. The scheduler decides which task instances are ready to run. The web UI and API expose DAGs, runs, task state, logs, and administrative actions. The metadata database stores Airflow state, including DAG runs and task instances. Worker or task-execution containers run the actual task code depending on the executor configured by the compose file. Local directories such as dags, logs, plugins, and sometimes config are bind-mounted so that changes on the host become visible to the containers.

DAGs are Python files that define workflows. Operators and TaskFlow-decorated Python functions define the task units inside those workflows. Connections and variables are runtime configuration objects that should usually be created through the UI, CLI, environment variables, or secrets backends rather than hardcoded into DAG files. Logs are written by task execution and then served back through the UI. Understanding these primitives helps you debug the quick start: if a DAG is absent, look at parsing; if a task is queued, look at the executor and worker; if a log is absent, look at task execution and log volume mounts.

Docker-specific Airflow integrations are separate from running Airflow itself in Docker. The Docker provider includes APIs such as DockerOperator, which executes a command inside a Docker container, and helper functions for Docker log streams. That provider is useful when an Airflow task needs to launch containerized work, but it is not the same thing as the Compose file that runs the Airflow control plane locally. In other words, Compose is the development deployment wrapper; the Docker provider is an integration surface available to DAG authors.

Relevant Source Files

  • docs/images/documentation_architecture.py — Generates an architecture diagram for how Airflow documentation moves from GitHub repositories through publishing infrastructure to the live documentation site; this supports why the quick-start instructions should be matched to the published docs version.
  • .apache-magpie-overrides/README.md — Documents repository-scoped agent override behavior and the rule that local override files live under .apache-magpie-overrides, giving context for project-maintenance files adjacent to developer documentation.
  • .apache-magpie-overrides/pr-management-config.md — Defines Airflow-specific pull-request triage labels, grace windows, and feedback delivery settings, which are relevant when proposing corrections to quick-start or compose documentation.
  • .apache-magpie-overrides/pr-management-triage-ci-check-map.md — Maps CI check name patterns to contributor documentation URLs, including docs builds, static checks, unit tests, Helm, Kubernetes, and image build categories that can be triggered by documentation or container changes.
  • .apache-magpie-overrides/pr-management-triage-comment-templates.md — Provides Airflow-specific triage comment templates and project URLs used when maintainers guide contributors toward quality criteria, static checks, testing, and documentation-building resources.

Sources: docs/images/documentation_architecture.py, .apache-magpie-overrides/README.md, .apache-magpie-overrides/pr-management-config.md, .apache-magpie-overrides/pr-management-triage-ci-check-map.md, .apache-magpie-overrides/pr-management-triage-comment-templates.md

System-to-Code Mapping

The supplied repository evidence for this page is strongest around the documentation and contributor-maintenance path rather than the compose runtime itself. docs/images/documentation_architecture.py builds a diagram with Airflow GitHub repositories, a release manager, committers, S3-backed live docs, CloudFront, and the live airflow.apache.org web server. That structure explains why a quick-start page should point readers to versioned official documentation instead of treating a copied command as timeless. When Airflow changes its supported versions, images, or installation guidance, the documentation-publishing pipeline is the mechanism that makes the updated instructions visible.

The .apache-magpie-overrides files show another part of the same developer experience: how Airflow maintainers keep contribution feedback consistent. The configuration file records project-specific labels and grace windows; the CI-check map turns failing GitHub checks into human-readable categories with documentation links; and the comment templates centralize URLs for pull-request quality criteria, static checks, testing, docs building, and provider testing. For a reader changing quick-start documentation, these files are not runtime dependencies, but they forecast the review signals a documentation or container-related change may encounter.

Sources: docs/images/documentation_architecture.py, .apache-magpie-overrides/pr-management-config.md, .apache-magpie-overrides/pr-management-triage-ci-check-map.md, .apache-magpie-overrides/pr-management-triage-comment-templates.md

Working with the Local Stack

After the stack starts, iterate through small changes. Add one DAG file under ./dags, wait for it to appear, trigger it manually, and inspect each task instance. Use docker compose ps to see which services are healthy, docker compose logs for service-level diagnostics, and the Airflow UI for scheduler and task-level state. If the UI loads but DAGs do not appear, the issue is usually parsing, imports, permissions, or a file not being mounted where the scheduler expects it. If tasks do not run, inspect the scheduler, worker, and executor-related containers.

docker compose ps
docker compose logs scheduler
docker compose logs api-server
docker compose down

Be careful with dependency changes. Installing Python packages interactively inside a running container is fast for experiments but disappears when containers are recreated. For repeatable local development, build a derived image or follow the documented mechanism for adding requirements to the Compose environment for the Airflow version you use. The same principle applies to connections, variables, and users: quick-start defaults help you learn, but durable team environments should make configuration explicit and reviewable. Treat local Compose state as disposable unless you have intentionally externalized it.

CI/CD and Documentation Signals

If you contribute changes around quick-start instructions, container examples, or documentation text, expect checks that map to the repository’s contributor guidance. The triage CI map categorizes static checks, Ruff, mypy, unit tests, documentation builds, Helm tests, Kubernetes tests, image builds, provider tests, and a catch-all group. That categorization is useful because a Compose documentation change may fail either because the prose build is broken or because an example interacts with image, provider, or deployment expectations. The comment-template file reinforces that maintainers prefer actionable links to the right contributor documentation rather than ad hoc review comments.

The practical next step is to choose the path that matches your goal. If you are learning Airflow, finish a fundamentals tutorial in the local Compose environment, then read the DAGs and Tasks pages. If you are validating deployment behavior, move next to Docker Stack, Kubernetes and Helm, and Production Deployment. If you are proposing documentation changes, first run the relevant docs checks and use the contributor URLs surfaced by the triage configuration to understand failures before asking reviewers to diagnose them.

Sources: .apache-magpie-overrides/pr-management-triage-ci-check-map.md, .apache-magpie-overrides/pr-management-triage-comment-templates.md

Ask a question...