Dynamic DAG Generation
Purpose and Scope
Dynamic DAG generation means using Python code, metadata, configuration, or repeated patterns to create DAG definitions instead of writing every workflow by hand. The generated output is still an ordinary Airflow DAG: it has a stable identity, contains tasks with stable task identifiers, and is discovered when Airflow imports the DAG file. The reader problem is usually scale. A team may need dozens of nearly identical workflows for regions, tenants, datasets, or products, but it still wants the scheduler, UI, history, and operational ownership model to behave as if each DAG had been authored directly.
Sources: airflow-core/src/airflow/example_dags/example_dynamic_task_mapping.py, airflow-core/src/airflow/example_dags/example_dynamic_task_mapping_with_no_taskflow_operators.py
The official Airflow guidance separates dynamic DAG generation from dynamic task mapping. Dynamic DAG generation changes the shape or number of DAGs at parse time, while dynamic task mapping expands task instances during a DAG run from a list-like input. That distinction is the most important design choice on this page. Generate DAGs when workflow identity, schedule, ownership, access boundary, or operational lifecycle differs. Use mapping when the workflow remains the same but each run has a variable number of work items to process.
Sources: airflow-core/src/airflow/example_dags/example_dynamic_task_mapping.py
Relevant Source Files
airflow-core/src/airflow/example_dags/example_dynamic_task_mapping.py— Demonstrates TaskFlow-style dynamic task mapping, second-order mapping from an upstream task output, and mapped task groups inside statically declared example DAGs.airflow-core/src/airflow/example_dags/example_dynamic_task_mapping_with_no_taskflow_operators.py— Demonstrates the same map-and-reduce pattern with customBaseOperatorsubclasses usingpartial(...).expand(...)and downstream output consumption.docs/images/documentation_architecture.py— Shows a checked-in generator for documentation architecture diagrams, useful as repository context for deterministic generation from source-controlled definitions.airflow-core/src/airflow/ui/src/layouts/Nav/TokenGenerationModal.tsx— Shows a frontend OpenAPI-driven token generation modal, useful as a boundary example because it is unrelated to DAG authoring and parsing..apache-magpie-overrides/pr-management-config.md— Shows repository-level project automation configuration, useful as a contrast with runtime DAG configuration and generated DAG metadata.
Core Primitives
Generated DAGs should be built from the same public authoring primitives that appear in hand-written DAG files. The TaskFlow example imports DAG, task, and task_group from the Airflow SDK, then declares several workflow definitions with a DAG identifier, schedule, start date, catchup behavior, and tags. Inside those DAG contexts, decorated Python functions become Airflow tasks. Dependencies are established by calling tasks and passing their returned task outputs to downstream tasks or expansion calls. A generator should produce this same kind of clear object graph, just from a reusable builder function or metadata loop.
Sources: airflow-core/src/airflow/example_dags/example_dynamic_task_mapping.py
The non-TaskFlow example is important because many generated DAGs are not pure Python TaskFlow workflows. It defines two custom operators that inherit from BaseOperator, accept constructor arguments, and implement an execute method. The first operator adds one to a value; the second sums a collection of values and marks that field as templated. This pattern shows that dynamic authoring does not require decorated functions. Generated DAG code can instantiate provider operators, internal operators, or custom operators as long as the parse-time constructor arguments are deterministic and the run-time work stays inside task execution.
Sources: airflow-core/src/airflow/example_dags/example_dynamic_task_mapping_with_no_taskflow_operators.py
Use precise vocabulary when designing a generator. A DAG is the workflow definition that Airflow discovers and schedules. A task is a node in that workflow. An operator is a reusable implementation of task behavior. A TaskFlow-decorated function is a convenient way to declare a Python task. A mapped task is one task definition that Airflow expands into multiple task instances for a run. A task group is a grouping primitive for repeated or related subgraphs, and the example shows that an entire group can be mapped across input values.
Sources: airflow-core/src/airflow/example_dags/example_dynamic_task_mapping.py
Authoring Flow
Start a dynamic DAG implementation with a small builder function that accepts one configuration entry and returns or registers one DAG. Keep the configuration entry simple: identifiers, schedule, tags, owner-like metadata, and task parameters are good inputs. The loop that calls the builder should use trusted, deterministic metadata available during import. Environment variables, generated Python constants, or structured files deployed with the DAG bundle are usually better than live service calls. The reason is operational: Airflow imports DAG files repeatedly, so slow or unstable import work becomes scheduler and DAG processor overhead.
Sources: airflow-core/src/airflow/example_dags/example_dynamic_task_mapping.py, airflow-core/src/airflow/example_dags/example_dynamic_task_mapping_with_no_taskflow_operators.py
Then decide whether variability belongs in the generated graph or in mapped task instances. In the TaskFlow example, a single add-one task is expanded over three literal values and a downstream sum task receives the mapped output collection. Another DAG gets a list from an upstream task, maps a multiply-by-two task over that result, and then maps an add-ten task over the mapped output. These examples keep the DAG identity stable while allowing each run to fan out according to data. That is usually preferable when only the item count changes.
Sources: airflow-core/src/airflow/example_dags/example_dynamic_task_mapping.py
When the repeated unit is a multi-step operation, prefer a mapped task group over a code generator that manually creates many near-identical task identifiers. The mapped task group example defines a small group containing an add step and a multiply step, returns the final task output, and expands the group over three values. This keeps the graph understandable because the repeated structure is named once and expanded by Airflow. A parse-time generator is still useful for separate DAG identities, but a mapped group is often clearer for repeated work within one workflow run.
Sources: airflow-core/src/airflow/example_dags/example_dynamic_task_mapping.py
Compact Reference
| Concern | Source-backed pattern | When to use it |
|---|---|---|
| Stable workflow declaration | with DAG(dag_id=..., schedule=..., start_date=..., tags=...) | Every generated DAG should still look like a normal Airflow DAG definition. |
| TaskFlow fan-out | add_one.expand(x=[1, 2, 3]) | Use when a Python task should run once per input value. |
| Second-order mapping | Map over the output of an upstream task | Use when the input list is produced during the DAG run. |
| Operator fan-out | AddOneOperator.partial(...).expand(...) | Use with provider or custom operators that are not TaskFlow functions. |
| Reduce mapped outputs | Downstream task or operator consumes mapped output | Use when expanded work must be aggregated. |
| Repeated subgraph | @task_group followed by group expansion | Use when each input requires several related tasks. |
Caveats and Edge Cases
The largest caveat is parse-time cost. A DAG generator runs while Python modules are imported, not only when a task executes. If that code performs remote API calls, large database queries, expensive filesystem scans, or non-deterministic computations, Airflow can spend too much time discovering workflows and may produce changing graphs across parses. The supplied examples are intentionally lightweight at import time: they define functions, classes, DAG contexts, literal lists, and dependencies. Production generators should follow the same shape and move expensive discovery to a separate preparation step or cache.
Sources: airflow-core/src/airflow/example_dags/example_dynamic_task_mapping.py, airflow-core/src/airflow/example_dags/example_dynamic_task_mapping_with_no_taskflow_operators.py
The second caveat is stable observability. Generated workflows are only useful if operators can find and explain them in the UI. The examples use explicit DAG identifiers and the example tag, which makes them recognizable and filterable. A production generator should use predictable DAG IDs, task IDs, tags, and documentation strings derived from business metadata. Avoid names that include timestamps, random values, or unordered dictionary iteration results. If a configuration entry is invalid, fail with an error message that identifies the entry rather than hiding the source of the problem inside a generic loop.
Sources: airflow-core/src/airflow/example_dags/example_dynamic_task_mapping.py
A third caveat is serialization and default handling. Airflow documentation treats DAG serialization as an administration concern because serialized DAG data is consumed by the webserver and other runtime components. A generated graph that changes shape unexpectedly can make serialized state harder to reason about. Prefer deterministic builder functions, explicit default arguments, and small, typed configuration records. If the number of elements varies by run, mapped tasks are safer than regenerating the DAG with a new task list. If the number of DAGs varies by deployment, control that variation through versioned metadata.
Sources: airflow-core/src/airflow/example_dags/example_dynamic_task_mapping.py, airflow-core/src/airflow/example_dags/example_dynamic_task_mapping_with_no_taskflow_operators.py
Repository Boundaries and Source Mapping
The requested source set includes files outside DAG authoring, and those boundaries are useful for contributors. The documentation architecture script builds a diagram from checked-in Python definitions, image assets, and diagram nodes representing GitHub repositories, publication steps, S3, CloudFront, and the live Airflow website. It is not a DAG generator, but it models the same engineering preference that dynamic DAG generation should follow: generated artifacts should be reproducible from source-controlled definitions rather than ad hoc manual state.
Sources: docs/images/documentation_architecture.py
The token generation modal and PR-management configuration illustrate what not to treat as DAG generation evidence. The modal is a React component that calls an OpenAPI query hook, stores UI state, shows a generated token once, and resets state on close. The PR-management file defines labels, grace windows, and feedback delivery settings for repository automation. These files live in the same repository but serve frontend and contributor-workflow concerns. Dynamic DAG behavior should be reasoned about in DAG authoring, scheduler, serialization, and task runtime code, not UI token modals or triage configuration.
Sources: airflow-core/src/airflow/ui/src/layouts/Nav/TokenGenerationModal.tsx, .apache-magpie-overrides/pr-management-config.md
Next Steps
When you add a dynamic DAG generator, first implement one configuration entry and verify that repeated parses produce the same DAG ID, task IDs, dependencies, and tags. Next, add a second entry and confirm that the UI and logs make the generated workflows easy to distinguish. Finally, review whether any parse-time repetition should become dynamic task mapping instead. Read the related pages on DAGs, dynamic task mapping, DAG file processing, DAG bundles, and serialization before scaling a generator across many teams or datasets.