Overview
Purpose and Scope
Apache Airflow is the Apache Software Foundation project in this repository for building, scheduling, and operating workflows. In Airflow terminology, a workflow is usually authored as a DAG, and the running platform is made up of services that parse DAG files, schedule work, expose APIs and UI surfaces, and coordinate runtime execution. This page orients repository readers before they jump into installation, tutorials, administration, provider integrations, or SDK references. It explains which parts of the source tree represent the core distribution, which parts support integrations and deployment artifacts, and how the official documentation structure maps back to the checked-in project layout.
The root README identifies the project as Apache Airflow and presents it as a distributable open source project with PyPI, container, license, and community surfaces. That matters because Airflow is not a single script or library-only package: it is published as Python packages, shipped as container images, extended through provider packages, and operated as a service in development and production environments. The core package README narrows the repository map by naming the Airflow runtime components that live in the core package: scheduler, API server, DAG file processor, and triggerer. Sources: README.md, airflow-core/README.md
Use this overview as the entry point when you need to answer three practical questions. First, where should a user look to install or run Airflow? Second, where should a DAG author look to understand public authoring surfaces? Third, where should an operator, provider maintainer, or contributor look when connecting the documentation to source modules? The official documentation spine separates quick start, installation, tutorials, core concepts, authoring and scheduling, administration and deployment, integration, public interfaces, and references; this wiki follows the same reader path while grounding each topic in repository source files.
Relevant Source Files
README.md- The root project README names the repository as Apache Airflow and exposes the outward-facing distribution signals for license, PyPI, containers, and community.airflow-core/README.md- The core package README states that the package contains functionality for the scheduler, API server, DAG file processor, and triggerer.
These two files are intentionally small but important. The root README is the front door for users arriving from GitHub, PyPI, or a release artifact, so it anchors the project identity and distribution channels. The core README is the shortest direct map from the repository to the runtime services an Airflow deployment actually runs. When reading deeper pages, treat these files as the project-level index and the core-package index rather than as exhaustive implementation references. Sources: README.md, airflow-core/README.md
Repository and Distribution Map
Airflow is organized as a multi-distribution repository. The airflow-core area contains the main Python package and web assets for the central Airflow application, including the components called out by its README: the scheduler, API server, DAG file processor, and triggerer. Those names describe separate responsibilities rather than optional documentation labels. The scheduler decides what work should run, the API server exposes programmatic and UI-facing access, the DAG file processor turns user-authored DAG files into metadata the system can reason about, and the triggerer supports asynchronous waiting for deferrable work. Sources: airflow-core/README.md
Around that core, the repository includes provider packages, SDK packages, deployment artifacts, and developer tooling. Provider distributions package integrations with external systems such as databases, cloud services, messaging systems, and execution backends. The official provider documentation uses provider-specific package pages, connection type references, Python API references, PyPI installation instructions, and source-installation pages. That documentation pattern is the reason this wiki separates provider overview, operators and hooks reference, core extension contracts, custom providers, and executor or runtime providers into focused pages instead of treating integrations as an appendix.
The root README’s visible badges point to PyPI and containers, which are two of the major ways users encounter Airflow before reading source code. PyPI packages are the normal Python installation surface, while container images support local development, quick starts, and production deployment patterns. The repository also includes chart and Docker-stack documentation areas, plus SDK packages such as the Task SDK and language-specific clients. Those areas serve different users: DAG authors mostly need public authoring APIs, platform operators need deployment and monitoring guidance, and integration maintainers need provider metadata and extension contracts. Sources: README.md
Official Documentation Mapping
The official Airflow documentation is organized by reader journey, and this OpenWiki outline mirrors that journey. New users usually start with overview, quick start, and installation material before moving into tutorials. DAG authors then need core concepts such as DAGs, tasks, operators, hooks, variables, XCom, parameters, connections, assets, scheduling, templates, and dynamic mapping. Operators move into administration topics such as components, scheduler behavior, DAG file processing, metadata database migrations, deployment, Kubernetes and Helm, Docker images, logging, health checks, callbacks, and security.
The public interface documentation for Airflow 3.0 and later is especially important for source readers because it draws a line between supported APIs and internal implementation details. In practice, that means a DAG author should prefer documented public imports and concepts over reaching into internal modules simply because they are visible in the repository. Provider authors and platform integrators should make the same distinction when using hooks, triggers, timetables, executors, secrets backends, auth managers, extra links, notifications, logging, and monitoring extension points. The source tree contains more than the public contract, so the documentation map helps readers avoid coupling to implementation details.
Provider documentation forms a second documentation family. A provider package page describes what external system the provider integrates with, how it is installed on top of an existing Airflow installation, what minimum Airflow version it supports, and which cross-provider dependencies unlock optional features. The provider installation-from-sources documentation emphasizes released source packages, checksums, signatures, and provenance. This matters for Airflow because integrations are both source modules in this repository and independently released distributions that users install, upgrade, and audit separately from the core package.
System-to-Code Mapping
At the system level, an Airflow deployment can be understood as a control plane for workflows. DAG authors define the desired workflow, the DAG processor reads those definitions, the scheduler decides when task instances are ready, runtime workers execute tasks through an executor, the triggerer handles asynchronous deferrals, and the API server exposes state and operations. The core README directly names four of those central components as functionality contained in the core package, which makes airflow-core the right starting point for runtime architecture pages. Sources: airflow-core/README.md
The repository also separates extension and operations concerns so that the core can remain stable while the ecosystem grows. Hooks are the connector-level abstraction used by operators and providers; providers package integration-specific hooks, operators, sensors, triggers, connection metadata, and sometimes executors or logging integrations. SDKs and clients provide programmatic surfaces for interacting with Airflow from task code, remote command-line tools, or generated OpenAPI clients. Deployment directories, container documentation, Helm chart files, and CI tooling support running Airflow in reproducible environments rather than changing the conceptual workflow model.
A useful way to navigate the source tree is to start from the actor you are trying to support. If you are a DAG author, begin with tutorials and core concepts, then read pages about scheduling, parameters, assets, templates, and dynamic task mapping. If you are an Airflow operator, begin with components, scheduler, DAG processing, database, deployment, logging, metrics, and security pages. If you are extending Airflow, begin with providers, operators and hooks, extension contracts, custom providers, plugins, and API references. This keeps source exploration aligned with supported product boundaries.
Execution and Reader Flow
The recommended learning flow is deliberately layered. First, understand that Airflow is a workflow orchestration platform with a core runtime and separately documented integrations. Second, install or start Airflow through the appropriate distribution path, such as Python packages or containers. Third, author a minimal DAG and run it through the scheduler-controlled lifecycle. Fourth, add integration points through connections, hooks, operators, assets, templates, and provider packages. Finally, harden the deployment with database migrations, logging, metrics, authentication, secrets, and operational runbooks.
That flow also explains why this wiki has many focused pages instead of one broad manual. Airflow’s concepts are connected, but each reader task has a different failure mode. A new user needs sequencing and commands; a DAG author needs stable authoring APIs; an operator needs runtime boundaries and deployment cautions; a provider maintainer needs packaging and extension contracts; an SDK user needs session, streaming, and channel semantics. The overview should help you choose the next page by role rather than by directory name alone.
Next Steps
If you are evaluating Airflow for the first time, continue to start, then installation, and then the tutorial pages for fundamentals, pipelines, and TaskFlow. If you already operate an Airflow environment, continue to airflow-components, scheduler, dag-file-processing, metadata-database-and-migrations, and the deployment pages. If you are integrating external services, continue to providers-overview-and-installation, operators-and-hooks-reference, and core-extension-contracts. If you are writing against programmatic interfaces, continue to rest-api-and-openapi-clients, airflowctl, task-sdk, task-sdk-sessions, and streaming-logs-and-channels. Sources: README.md, airflow-core/README.md