Build a Pipeline

Purpose and Scope

This page explains how to think about an Airflow pipeline as a schedulable DAG made of task nodes, data handoffs, and integration-specific execution steps. The official pipeline tutorial is organized around a practical flow: initial setup, creating a connection, preparing staging and final tables, loading data, cleaning or merging it, defining the DAG, and then triggering and exploring the result. In repository terms, the same pattern appears in example DAGs and provider integrations: Python code declares the workflow shape, operators or decorators declare units of work, hooks isolate external systems, and the Airflow runtime schedules the graph.

Sources: providers/common/ai/src/airflow/providers/common/ai/example_dags/example_llm_analysis_pipeline.py, providers/apache/spark/src/airflow/providers/apache/spark/operators/spark_pipelines.py, providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_pipelines.py

A pipeline is not just a script that runs top to bottom. In Airflow, the DAG definition is parsed separately from task execution, and each task should be understandable as an independent runtime step. The example LLM analysis pipeline demonstrates this separation clearly: one task fetches work items, a mapped task analyzes each item, and a final task stores the collected results. Provider code shows the same design at integration scale: a Spark pipeline operator represents one task in the DAG, while a Spark hook turns that task request into a command-line invocation against the configured Spark environment.

Sources: providers/common/ai/src/airflow/providers/common/ai/example_dags/example_llm_analysis_pipeline.py, providers/apache/spark/src/airflow/providers/apache/spark/operators/spark_pipelines.py

Relevant Source Files

• providers/common/ai/src/airflow/providers/common/ai/example_dags/example_llm_analysis_pipeline.py - Defines a complete example DAG using @dag, @task, @task.llm, structured Pydantic output, dynamic task mapping with expand, and a downstream aggregation task.
• providers/apache/spark/src/airflow/providers/apache/spark/operators/spark_pipelines.py - Provides SparkPipelinesOperator, an operator that can run or dry-run Spark Declarative Pipelines from an Airflow task and exposes templated fields and Spark execution options.
• providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_pipelines.py - Provides SparkPipelinesHook, which implements the Spark Declarative Pipelines command execution model and connection-resolution behavior, including Spark Connect handling.
• providers/google/src/airflow/providers/google/cloud/hooks/vertex_ai/pipeline_job.py - Provides PipelineJobHook for Google Cloud Vertex AI Pipeline Job APIs, showing how cloud pipeline systems are accessed through hooks rather than embedded directly in DAG code.
• docs/images/documentation_architecture.py - Generates a documentation architecture diagram and is useful context for how Airflow publishes package documentation, including tutorial material, from repository-managed sources.

Core Pipeline Primitives

The first primitive is the DAG, created in the example with the @dag decorator. A DAG is the named workflow boundary: it groups tasks, provides metadata such as tags, and gives the scheduler a graph to parse. Inside the DAG function, normal Python assignment captures dependencies because the output of one decorated task is passed into the next step. This keeps the authoring model readable while still producing a runtime graph that Airflow can schedule, render, and execute independently of the original Python call stack.

Sources: providers/common/ai/src/airflow/providers/common/ai/example_dags/example_llm_analysis_pipeline.py

The second primitive is the task. In the example pipeline, get_support_tickets returns a list of ticket texts, analyze_ticket turns each text into a structured analysis, and store_results receives the collection. The TicketAnalysis Pydantic model is deliberately defined at module scope because downstream tasks need to deserialize the XCom payload by importable name. That is an important authoring rule for pipelines that pass typed objects between steps: the object contract must be stable enough for another task process to reconstruct it later.

Sources: providers/common/ai/src/airflow/providers/common/ai/example_dags/example_llm_analysis_pipeline.py

The third primitive is the external-system boundary. Airflow pipelines often call databases, Spark clusters, cloud ML services, object stores, or APIs. The Spark provider splits this boundary into an operator and a hook. SparkPipelinesOperator is the DAG-facing task class; SparkPipelinesHook is responsible for assembling and executing the Spark Declarative Pipelines command. The Google provider follows the same general idea with PipelineJobHook, which returns Vertex AI clients and PipelineJob objects using Airflow-managed Google credentials and impersonation settings.

Sources: providers/apache/spark/src/airflow/providers/apache/spark/operators/spark_pipelines.py, providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_pipelines.py, providers/google/src/airflow/providers/google/cloud/hooks/vertex_ai/pipeline_job.py

Execution Flow

A practical pipeline starts by choosing the state that crosses task boundaries. The official tutorial frames this as creating a connection, creating staging and final tables, loading raw data, and then merging or cleaning it. The repository-backed LLM example uses a smaller but equivalent shape: fetch unprocessed inputs, transform them into structured results, and store the outputs. In both cases, the key design decision is to keep each step narrow. Fetching, transformation, and persistence are separate tasks so retries, logs, mapping, and UI inspection have meaningful task-level boundaries.

Sources: providers/common/ai/src/airflow/providers/common/ai/example_dags/example_llm_analysis_pipeline.py

A minimal TaskFlow-style pipeline has the same structure as the example DAG:

from airflow.providers.common.compat.sdk import dag, task
 
@dag(tags=['example'])
def my_pipeline():
    @task
    def extract():
        return ['one', 'two']
 
    @task
    def transform(value: str):
        return value.upper()
 
    @task
    def load(values: list[str]):
        for value in values:
            print(value)
 
    items = extract()
    results = transform.expand(value=items)
    load(results)
 
my_pipeline()

This example mirrors the source pipeline’s dependency style without copying its LLM-specific behavior. The call to extract creates an upstream task output, transform.expand creates mapped task instances for each item, and load consumes the collection. The important mental model is that these function calls define the graph; they are not meant to perform the production work while the DAG file is being parsed. Work belongs inside task bodies, and any external system call should normally be isolated in a task, operator, or hook.

Sources: providers/common/ai/src/airflow/providers/common/ai/example_dags/example_llm_analysis_pipeline.py

Provider Pipeline Integrations

When a pipeline step belongs to a specialized execution engine, use the provider abstraction instead of embedding all execution details in a Python task. SparkPipelinesOperator executes Spark Declarative Pipelines through the spark-pipelines CLI and supports both run and dry-run commands. Its templated fields include pipeline_spec, conf, env_vars, keytab, and principal, which means those values can vary per DAG run using Airflow templating. The operator also exposes resource and deployment settings such as executor counts, memory, deploy mode, YARN queue, and OpenLineage injection flags.

Sources: providers/apache/spark/src/airflow/providers/apache/spark/operators/spark_pipelines.py

The hook behind that operator is where connection-specific behavior lives. SparkPipelinesHook extends SparkSubmitHook, validates that the command is either run or dry-run, and resolves the configured connection before command execution. Its docstring describes two supported modes. Legacy Spark-style connections use cluster-manager flags assembled by SparkSubmitHook. Spark Connect connections set SPARK_REMOTE from an sc:// URI and avoid cluster-manager flags because the Connect-native CLI rejects remote mode when master or deploy mode are also specified.

Sources: providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_pipelines.py

Vertex AI pipeline jobs show the cloud-service version of the same pattern. PipelineJobHook derives from GoogleBaseHook and OperationHelper, accepts gcp_conn_id and impersonation_chain, constructs regional API endpoints when appropriate, and returns PipelineServiceClient instances with Airflow’s Google client information and credential handling. It can also build a PipelineJob object from fields such as display name, template path, job id, pipeline root, parameter values, input artifacts, caching, encryption key, labels, project, location, and failure policy.

Sources: providers/google/src/airflow/providers/google/cloud/hooks/vertex_ai/pipeline_job.py

Compact Reference

Triggering, Exploring, and Iterating

After a pipeline is defined, the next development loop is to trigger it and inspect it in the Airflow UI. The official tutorial calls out triggering and exploring the DAG as the final hands-on step, and that is where task boundaries pay off. You can see whether extraction produced inputs, whether mapped transformations expanded as expected, and whether the final persistence step received the intended structured records. For provider-backed tasks, inspect rendered templated fields and task logs to confirm the exact pipeline specification, connection id, and runtime options used by the operator.

Sources: providers/common/ai/src/airflow/providers/common/ai/example_dags/example_llm_analysis_pipeline.py, providers/apache/spark/src/airflow/providers/apache/spark/operators/spark_pipelines.py

Documentation and examples also have a publishing lifecycle. The documentation architecture script models Airflow repositories, release managers, committers, S3-hosted package docs, CloudFront cache, and the public airflow.apache.org webserver. For a tutorial reader, the main takeaway is that examples and provider references are part of a maintained documentation system, not incidental comments. When you move from the simple tutorial to Spark, Vertex AI, or AI provider pipelines, the source-backed provider docs and example DAGs are the right next layer to read.

Sources: docs/images/documentation_architecture.py

Next Steps

To build your own pipeline, start with a narrow DAG that separates extraction, transformation, and loading into task-sized units. Add typed boundaries only when downstream tasks need structured objects, and keep those types importable at module scope. Use dynamic task mapping when one upstream collection should fan out into repeated work. When the task is really a Spark, Vertex AI, database, or cloud operation, prefer the provider’s operator and hook contract so Airflow manages connections, templating, credentials, logging, and retries consistently.

Related pages: tutorial-taskflow, dags, tasks-operators-and-hooks, connections, dynamic-task-mapping, operators-and-hooks-reference