Dynamic Task Mapping

Purpose and Scope

Dynamic task mapping is Airflow’s authoring pattern for turning one logical task definition into many task instances at run time. Instead of writing a loop that creates a fixed number of tasks while the DAG file is parsed, the DAG declares which argument should expand, and Airflow creates the mapped instances when the run knows the input data. This page focuses on the semantics visible to DAG authors and operators: how TaskFlow functions, classic operators, and task groups express mapping, how downstream reduction consumes mapped results, and why each mapped child behaves like a normal task instance during execution.

Sources: airflow-core/src/airflow/example_dags/example_dynamic_task_mapping.py, airflow-core/src/airflow/example_dags/example_dynamic_task_mapping_with_no_taskflow_operators.py, task-sdk/src/airflow/sdk/execution_time/task_mapping.py

The official dynamic task mapping guide covers a broad set of authoring forms, including simple mapping, task-generated mapping, repeated mapping, multiple mapped parameters, named mapping, mapped task groups, filtering, transformation, zipping, concatenating upstreams, limits, and automatic skipping of empty maps. The checked-in examples demonstrate the core mechanics that those guide sections build on. Read the examples as executable documentation: they intentionally use small arithmetic tasks so that the important concept is the mapping boundary, not the business logic hidden inside a large operator.

Sources: airflow-core/src/airflow/example_dags/example_dynamic_task_mapping.py, airflow-core/src/airflow/example_dags/example_dynamic_task_mapping_with_no_taskflow_operators.py

Relevant Source Files

docs/images/documentation_architecture.py — documents how Airflow package documentation is generated and published from the repository to the live documentation site, which explains where the official dynamic task mapping guide fits in the docs supply chain.
airflow-core/src/airflow/example_dags/example_dynamic_task_mapping_with_no_taskflow_operators.py — shows dynamic mapping with custom non-TaskFlow operators using partial, expand, operator output, and a downstream aggregation operator.
airflow-core/src/airflow/example_dags/example_dynamic_task_mapping.py — contains TaskFlow dynamic mapping examples for a mapped task, second-order mapping from an upstream result, and mapping over a task group.
task-sdk/src/airflow/sdk/execution_time/task_mapping.py — contains Task SDK runtime helpers that compute relevant map indexes for XCom aggregation and mapped task-group relationships.
providers/apache/hdfs/src/airflow/providers/apache/hdfs/log/hdfs_task_handler.py — shows provider-side task log handling for task instances, including upload and read behavior against HDFS remote storage.

Core Authoring Primitives

The smallest TaskFlow example defines two decorated Python functions, then expands one of them over a literal list. The add-one task is declared once, but its input argument is expanded over three values. The downstream sum task receives the mapped task’s collected output and reduces the resulting values. This is the authoring distinction to keep in mind: mapping creates parallel task instances from one declaration, while reduction consumes the collection produced by those instances as a normal downstream dependency.

Sources: airflow-core/src/airflow/example_dags/example_dynamic_task_mapping.py

added_values = add_one.expand(x=[1, 2, 3])
sum_it(added_values)

Task-generated mapping appears when the expanded values come from another task rather than from a literal list in the DAG file. In the second example DAG, one task returns a list, a second task maps across that list, and a third task maps across the second task’s mapped output. This pattern is useful when the number of work items is discovered at run time, such as after listing files, partitions, records, or external objects. The DAG still describes a stable shape, but the run determines the cardinality of the mapped work.

Sources: airflow-core/src/airflow/example_dags/example_dynamic_task_mapping.py

Classic operators use the same model through an operator-oriented API. The non-TaskFlow example defines a custom operator that adds one to a single value and another custom operator that sums a collection of values. The mapped operator is created with a partial task definition and an expanded value argument. The aggregation operator receives the mapped operator’s output as its input values. This matters for teams that already have custom operators or provider operators: dynamic mapping is not limited to Python functions decorated with the TaskFlow API.

Sources: airflow-core/src/airflow/example_dags/example_dynamic_task_mapping_with_no_taskflow_operators.py

add_one_task = AddOneOperator.partial(task_id="add_one").expand(value=[1, 2, 3])
sum_it_task = SumItOperator(task_id="sum_it", values=add_one_task.output)

Runtime Semantics and Map Index Resolution

At execution time, Airflow must know which mapped upstream instances are relevant to a downstream task. The Task SDK file describes this in terms of task instance counts, map indexes, and XCom aggregation. A helper queries the runtime supervisor for the count of task instances for a task, run, and DAG, then verifies that the response has the expected count message type. Another helper determines the relevant map indexes when resolving XCom values, especially for tasks inside mapped task groups where a downstream task should not always read every upstream child indiscriminately.

Sources: task-sdk/src/airflow/sdk/execution_time/task_mapping.py

The map-index logic is careful about task-group context. One helper searches for the innermost common mapped task group shared by two operators, returning no group when operators are not assigned to a DAG, belong to different DAGs, or have no shared mapped group. Another helper checks whether an operator is further mapped inside a container group by inspecting the operator and walking parent task groups. These details protect nested mappings from accidentally aggregating the wrong level of results and explain why mapped task groups need runtime-aware resolution rather than simple list passing.

Sources: task-sdk/src/airflow/sdk/execution_time/task_mapping.py

Task Groups, Reduction, and UI Behavior

Mapped task groups let a repeated unit contain more than one task. The example task group accepts a number, runs an add step, passes that result to a multiply step, and then expands the group over three inputs. Conceptually, each input creates a group instance with its own internal task instances. This is different from mapping each internal task independently without a group boundary, because the group boundary gives Airflow a container for dependency structure, display grouping, and map-index relationships between the tasks inside that repeated unit.

Sources: airflow-core/src/airflow/example_dags/example_dynamic_task_mapping.py, task-sdk/src/airflow/sdk/execution_time/task_mapping.py

In the runtime and UI, a mapped task is still made of task instances. Operators and log handlers therefore need to treat mapped children as executable units with states, attempts, and logs. The HDFS task handler is not specific to mapping, but it illustrates the provider contract that task logs are uploaded and read for task instances through a handler that receives runtime task-instance context. If a remote HDFS path exists, it reads the file; otherwise it emits a message explaining that no logs were found for that task instance.

Sources: providers/apache/hdfs/src/airflow/providers/apache/hdfs/log/hdfs_task_handler.py

Documentation-to-Code Mapping

The repository also contains documentation infrastructure that explains why the official guide and the example DAGs should be read together. The documentation architecture script creates an image describing Airflow GitHub repositories, package-doc publishing, live documentation storage, a CloudFront cache, and the public Airflow documentation site. For dynamic task mapping, the practical effect is that small source examples in the repository can support user-facing documentation that appears on the published site. The example file even marks the TaskFlow mapping section with start and end comments for documentation extraction.

Sources: docs/images/documentation_architecture.py, airflow-core/src/airflow/example_dags/example_dynamic_task_mapping.py

When you map a task, choose the authoring form based on where the variable work list comes from. Use a literal list for examples, small static fan-out, or simple demonstrations. Use task-generated mapping when a previous task discovers the items. Use classic operator mapping when the work is packaged as an operator rather than a TaskFlow function. Use mapped task groups when each item requires a repeated subgraph. Those choices all lead to the same runtime concern: Airflow has to know how many task instances exist and which map indexes feed each downstream dependency.

Practical Guidance and Next Steps

For a first implementation, keep the mapped unit small and make the reducer explicit. The examples intentionally separate add-one work from sum-it aggregation, which makes it easy to reason about where fan-out ends and where collection begins. If you move from literal lists to task-generated lists, confirm that the upstream task returns an expandable collection. If you map inside a task group, remember that nested group structure affects which XCom values are relevant to each mapped child. For operations, inspect mapped children as individual task instances and follow normal log-reading paths for failed attempts.

Next, read the DAGs, TaskFlow, and Task SDK pages if you need the broader authoring and runtime model. Read task logs and logging architecture when diagnosing failures in mapped children. Read dynamic DAG generation separately: dynamic task mapping changes the number of task instances in a DAG run, while dynamic DAG generation changes what DAG structure is produced when files are parsed. Keeping that distinction clear helps avoid parse-time loops where a run-time expansion is the better Airflow primitive.

Ask a question...