TaskFlow Tutorial

Purpose and Scope

The TaskFlow tutorial teaches the most Pythonic authoring path in Airflow: define a DAG as a Python function, decorate normal Python callables as tasks, and express dependencies by calling those functions and passing their returned values. In the official tutorial sequence, this follows the introductory workflow material and focuses on the “big picture” pipeline, defining the DAG, writing tasks with @task, building the flow, running it, and understanding what Airflow does behind the scenes. The repository example tutorial_taskflow_api_virtualenv.py is the clearest code artifact for this page because it shows the Extract, Transform, Load shape using airflow.sdk.dag and airflow.sdk.task.

Sources: airflow-core/src/airflow/example_dags/tutorial_taskflow_api_virtualenv.py, docs/images/documentation_architecture.py

TaskFlow does not remove Airflow’s core model. A DAG still defines scheduling metadata such as schedule, start_date, catchup, and tags; tasks still execute independently; and task results still move through Airflow-managed runtime state rather than through a single in-memory Python process. What changes is the authoring interface. Instead of manually creating an operator for each callable and manually wiring dependency objects, the decorated function call returns a task-like object whose output can be passed to downstream decorated tasks. This lets a pipeline read like ordinary Python while still producing an Airflow DAG for the scheduler and UI.

Relevant Source Files

airflow-core/src/airflow/example_dags/tutorial_taskflow_api_virtualenv.py - Demonstrates the tutorial-style TaskFlow DAG with @dag, @task.virtualenv, @task(multiple_outputs=True), task return values, and downstream parameter passing.
airflow-core/src/airflow/example_dags/example_setup_teardown_taskflow.py - Shows advanced TaskFlow patterns for setup tasks, teardown tasks, task groups, dependency chaining, and context-managed teardown sections.
airflow-core/src/airflow/example_dags/example_dynamic_task_mapping_with_no_taskflow_operators.py - Provides a non-TaskFlow dynamic mapping example that helps distinguish the TaskFlow authoring style from the lower-level operator contract.
airflow-core/src/airflow/example_dags/example_asset_alias_with_no_taskflow.py - Demonstrates asset and asset-alias scheduling with PythonOperator, giving a contrast point for features that can be used without TaskFlow decorators.
docs/images/documentation_architecture.py - Shows that Airflow’s package documentation is published from repository artifacts to the live documentation site, which explains why tutorial examples in source are part of the reader-facing documentation system.

Core Primitives

A TaskFlow DAG normally begins with the @dag decorator. In the virtualenv tutorial example, tutorial_taskflow_api_virtualenv is a Python function decorated with @dag(schedule=None, start_date=datetime(2021, 1, 1), catchup=False, tags=["example"]). The function body is the authoring scope for the pipeline. Airflow evaluates this function at parse time to create task objects and relationships, then the final tutorial_dag = tutorial_taskflow_api_virtualenv() assignment exposes the DAG object to the Dag processor. This pattern is important: calling the decorated DAG function does not run the business workload immediately; it builds the Airflow graph that later runs under the scheduler and executor.

Sources: airflow-core/src/airflow/example_dags/tutorial_taskflow_api_virtualenv.py

The second primitive is @task, which wraps a Python function as an Airflow task. The tutorial example defines extract, transform, and load as nested functions. extract is decorated with @task.virtualenv, specifying serializer="dill", system_site_packages=False, and requirements=["funcsigs"], so it demonstrates a task that runs in an isolated virtual environment with explicit Python requirements. transform uses @task(multiple_outputs=True), returns a dictionary containing total_order_value, and allows downstream code to reference order_summary["total_order_value"]. load receives the value as a typed Python parameter and prints the formatted result.

The third primitive is the task invocation result. In the example, order_data = extract() creates the upstream task and produces a value reference, order_summary = transform(order_data) creates a downstream task consuming that reference, and load(order_summary["total_order_value"]) consumes one field from the transform output. This is the heart of TaskFlow: dependencies are inferred from dataflow-style calls, while Airflow still records task instances, retries, logs, and results through its runtime machinery. The source example is small, but it captures the practical mental model readers need before exploring retries, parameterization, sensors, templates, or mixed operator graphs.

Tutorial Flow

Start by writing the DAG boundary before writing the task bodies. The tutorial example uses schedule=None, which means it is intended to be triggered manually or used as a simple example rather than run on a recurring timetable. It also sets catchup=False, avoiding historical backfill behavior for the fixed start_date. That choice keeps the tutorial focused on how data moves between tasks instead of schedule catchup semantics. For a first TaskFlow DAG, this is a useful pattern: define a minimal schedule, choose explicit tags, and keep the pipeline small enough that the generated graph can be inspected in the UI.

Next, write each task as a side-effect-limited Python function with clear inputs and outputs. In the tutorial, extract parses a hardcoded JSON string into a dictionary, transform computes the total order value, and load prints the result. Although this is intentionally simple, it illustrates an important design constraint: a TaskFlow function should communicate with downstream tasks through its return value or declared side effects, not by relying on local variables from another task’s Python frame. Airflow tasks can run in separate processes, containers, or virtual environments, so the graph must be serializable and schedulable.

Finally, build the flow by calling the decorated functions in dependency order. This is where TaskFlow differs most visibly from classic operator authoring. You do not need to call set_upstream, set_downstream, or write bitshift dependencies for the common case where a downstream task consumes an upstream output. The function-call expression is enough. For values with multiple_outputs=True, dictionary keys can become addressable output references. This creates a readable pipeline definition while preserving Airflow concepts such as task IDs, rendered logs, task instance state, and DAG visualization.

How TaskFlow Compares with Classic Operators

The repository also includes examples that intentionally avoid TaskFlow, and those are useful for understanding what TaskFlow abstracts. In example_dynamic_task_mapping_with_no_taskflow_operators.py, custom classes inherit from BaseOperator, implement execute, and are instantiated directly. AddOneOperator.partial(task_id="add_one").expand(value=[1, 2, 3]) maps a classic operator over a list, and SumItOperator(task_id="sum_it", values=add_one_task.output) reduces the mapped outputs. This demonstrates that dynamic task mapping is not exclusive to decorated functions; TaskFlow is an authoring convenience layered over the same DAG and task concepts.

Sources: airflow-core/src/airflow/example_dags/example_dynamic_task_mapping_with_no_taskflow_operators.py

The asset-alias example makes the same point for data-aware scheduling. It uses DAG, Asset, AssetAlias, and PythonOperator to produce asset events, resolve an alias such as example-alias-no-taskflow, and schedule consumer DAGs from assets or aliases. The task callable receives outlet_events or inlet_events, but the tasks themselves are classic PythonOperator instances. A TaskFlow user should take this as a boundary marker: Airflow features such as assets, dynamic mapping, and provider operators can be mixed with or used without TaskFlow. Choose TaskFlow when the pipeline is naturally expressed as Python functions and returned values.

Sources: airflow-core/src/airflow/example_dags/example_asset_alias_with_no_taskflow.py

Advanced Patterns in the Examples

After the basic tutorial, the next repository-backed step is setup and teardown. example_setup_teardown_taskflow.py imports DAG, setup, task, task_group, and teardown from airflow.sdk. It first shows ordinary @task functions chained with task_1 >> task_2 >> task_3.as_teardown(setups=task_1), where as_teardown marks the third task as teardown, marks the first as setup, and establishes the direct setup-to-teardown relationship. The comments explain an operational consequence: clearing the middle task can also clear its setup and teardown tasks, which helps keep resource lifecycle work consistent.

Sources: airflow-core/src/airflow/example_dags/example_setup_teardown_taskflow.py

The same file then demonstrates decorator-based lifecycle tasks. outer_setup returns a cluster identifier, outer_teardown receives that identifier, and outer_work runs between them. A nested @task_group called section_1 defines its own inner_setup, inner_work, and inner_teardown, passing the setup return value into both work and teardown. The context-managed form with outer_teardown(outer_setup()): makes the lifecycle relationship readable in TaskFlow style. This example is a good bridge from the beginner tutorial to production DAGs where temporary clusters, scratch storage, credentials, or external sessions must be created and reliably cleaned up.

Practical Reference

Construct	Source example	What it does
`@dag(...)`	`tutorial_taskflow_api_virtualenv`	Declares DAG metadata around a Python function and builds a DAG when called.
`@task.virtualenv(...)`	`extract`	Runs a Python task in a virtualenv with serializer, site-package, and requirements options.
`@task(multiple_outputs=True)`	`transform`	Treats a returned dictionary as multiple addressable outputs.
`task_output["key"]`	`order_summary["total_order_value"]`	Passes one returned field into a downstream task.
`@setup` and `@teardown`	`outer_setup`, `outer_teardown`	Marks lifecycle tasks that prepare and clean up resources.
`@task_group`	`section_1`	Groups related TaskFlow tasks inside a larger DAG section.
`BaseOperator.partial().expand(...)`	`AddOneOperator`	Shows comparable dynamic mapping using non-TaskFlow operators.

Use the TaskFlow tutorial when the reader’s immediate goal is to write a Python data pipeline with minimal Airflow boilerplate. Move next to dynamic task mapping when the number of task instances depends on runtime data, to assets when DAGs should be scheduled from data events, and to setup/teardown when tasks allocate external resources that need deterministic cleanup. The key habit is to keep the Python code readable while remembering that decorated calls define an Airflow graph, not a single local function call chain.

Next Steps

Run or inspect tutorial_taskflow_api_virtualenv.py first, then compare it with the non-TaskFlow operator and asset examples to understand what the decorators are hiding. If you are authoring real DAGs, add retries, task IDs, provider operators, and asset schedules incrementally rather than converting every concept at once. For deeper follow-up, read the pages on DAGs, dynamic task mapping, assets, deferrable operators, and templates, because those concepts compose with TaskFlow rather than replacing it.

Ask a question...