Task SDK

Purpose and Scope

The Task SDK is the Airflow distribution aimed at people and tools that define and execute tasks without coupling their code directly to the full set of Airflow internals. In the repository, the package README describes it as containing interfaces for DAG authors and task execution logic for Python, and it is published separately as the apache-airflow-task-sdk package. That separation is important because task code should be easy to author, test, package, and evolve even when the scheduler, API server, DAG processor, and triggerer continue to change independently. Sources: task-sdk/README.md

Official Task SDK documentation frames the SDK as a python-native interface for defining DAGs, executing tasks in isolated subprocesses, and interacting at runtime with Airflow resources such as Connections, Variables, XComs, Metrics, Logs, and OpenLineage events. In practice, that means a DAG author reaches first for stable authoring primitives such as dag and task, while the runtime uses SDK components to communicate with Airflow services. The SDK is therefore both an authoring surface and an execution boundary: it helps keep user code concise while reducing assumptions about metadata database access or scheduler implementation details.

The second source file for this page shows that the Task SDK is not limited to handwritten Python files. The airflow-metadata.schema.json document defines build-time metadata for an Airflow native-executable SDK bundle. A bundle manifest declares which DAG and task identifiers an executable SDK bundle exposes, which SDK produced it, and what supervisor wire-schema version the bundle was compiled against. This schema gives Airflow a static description of executable task definitions before runtime, so non-Python or compiled SDK scenarios can still participate in Airflow orchestration in a predictable way. Sources: task-sdk/docs/airflow-metadata.schema.json

Relevant Source Files

• task-sdk/README.md - Package-level README for the Task SDK distribution, including the package purpose and the pip install apache-airflow-task-sdk installation command.
• task-sdk/docs/airflow-metadata.schema.json - JSON Schema for Airflow executable SDK bundle metadata, including required manifest fields for SDK identity, source display name, DAG identifiers, and task identifiers.

Core Primitives

For a new DAG author, the most visible Task SDK primitives are the Python authoring interfaces imported from airflow.sdk. The official getting-started example uses dag to define the workflow boundary and task to turn a Python function into an Airflow task. A DAG, or Directed Acyclic Graph, describes the workflow and dependency structure; a task is an executable unit inside that graph. The TaskFlow style keeps ordinary Python functions central, so developers can express the flow in normal Python while Airflow records the task graph and runtime metadata needed for scheduling and execution.

from airflow.sdk import dag, task
 
@dag
def example_simplest_dag():
    @task
    def my_task():
        pass
 
    my_task()

Runtime primitives are broader than decorators. The official Task SDK overview names Connections, Variables, XComs, Metrics, Logs, and OpenLineage events as resources task code can interact with through SDK-supported interfaces. Connections represent configured access to external systems; Variables provide deployment-scoped values; XComs carry small pieces of data between task instances; Metrics and Logs expose observability signals; and OpenLineage events describe data lineage. The key design point is that these interactions are mediated by a task execution interface rather than by arbitrary direct access to Airflow internals.

Executable bundle metadata introduces another primitive: the manifest. The schema title, description, and required fields define a build-time contract for SDK bundles that expose DAGs and tasks. The manifest must include airflow_bundle_metadata_version, sdk, source, and dags. Under sdk, the producer must declare language, version, and supervisor_schema_version; under dags, every exposed dag_id maps to an entry with a static list of task IDs. This lets Airflow inspect bundle contents and understand compatibility before coordinating execution. Sources: task-sdk/docs/airflow-metadata.schema.json

Installation and First DAG

Install the Python Task SDK package with pip when you want the standalone SDK distribution rather than relying on the monolithic Airflow installation surface. The README gives the direct command, which is intentionally minimal and mirrors the official documentation installation guidance. In a development environment, run it inside the virtual environment used to author DAGs or to build a task bundle. In a deployment environment, align the SDK version with the Airflow environment that will execute or coordinate the tasks, especially when using bundle metadata with an explicit supervisor schema version. Sources: task-sdk/README.md

pip install apache-airflow-task-sdk

After installation, start with the smallest useful DAG: define a @dag function, define one or more @task functions inside or near it, and call the task functions to build dependencies. The call does not execute the Python function immediately as ordinary application code; it participates in DAG construction so Airflow can schedule task instances later. As workflows grow, keep business logic in normal Python functions and use Airflow constructs to describe scheduling, dependencies, parameters, and runtime resource access. This keeps the DAG readable and leaves execution coordination to Airflow.

System-to-Code Mapping

The Task SDK sits between DAG author code and Airflow runtime services. The README establishes the package boundary: this distribution contains author-facing interfaces and Python task execution logic. The official docs then describe why that boundary exists: task code should interact with Airflow resources through a dedicated execution API for state transitions, heartbeats, XComs, and resource fetching instead of reaching into the metadata database directly. That model supports isolation, makes remote or subprocess task execution easier, and gives Airflow a controlled place to evolve runtime behavior across versions. Sources: task-sdk/README.md

The executable bundle schema maps the same idea into a build artifact. source is the display name of the original primary DAG source file, while dags is a mapping from DAG ID to static task IDs. The sdk.language field identifies the source language that produced the bundle, and the schema examples explicitly allow lower-case language identifiers such as Go, Rust, C++, or Zig. The supervisor_schema_version field is a dated compatibility marker used by the coordinator and supervisor to exchange messages in a shape the bundle understands. Sources: task-sdk/docs/airflow-metadata.schema.json

Executable Bundle Metadata Reference

The bundle metadata schema is permissive in extension points but strict about the minimum identity Airflow needs. The root object allows additional properties, but it still requires the bundle spec version, SDK identity, source display name, and DAG map. Version strings are validated: airflow_bundle_metadata_version follows a numeric semantic-looking pattern, sdk.language must be a lower-case identifier, and sdk.supervisor_schema_version must use a dated form such as 2026-06-16. Those constraints are useful for build tools because validation catches malformed manifests before the artifact reaches a scheduler or coordinator. Sources: task-sdk/docs/airflow-metadata.schema.json

{
  "airflow_bundle_metadata_version": "1.0",
  "sdk": {
    "language": "go",
    "version": "1.2.3",
    "supervisor_schema_version": "2026-06-16"
  },
  "source": "example.go",
  "dags": {
    "example_dag": {
      "tasks": ["extract", "load"]
    }
  }
}

Next Steps

Use this page as the entry point for deciding which Task SDK surface you need. If you are writing Python DAGs, begin with the airflow.sdk authoring primitives and the TaskFlow tutorial path, then add runtime resource access only when the task needs Connections, Variables, XComs, metrics, logs, or lineage events. If you are building language SDKs or executable bundles, validate the bundle manifest against task-sdk/docs/airflow-metadata.schema.json and pay close attention to the supervisor schema version because it is the explicit compatibility contract for runtime messaging.

Related pages: tutorial-taskflow, task-sdk-sessions, streaming-logs-and-channels, rest-api-and-openapi-clients