OpenWikiapache/airflow

Task SDK Sessions

Purpose and Scope

Task SDK sessions are the runtime conversation between an executing task handler and the Airflow side that supervises it. The Python Task SDK README positions the package as both a Dag-author interface and task execution logic for Python, while the Task SDK official documentation describes runtime access to Airflow resources through a dedicated Task Execution API rather than direct metadata database access. The TypeScript SDK extends that same idea to Node handlers: Airflow starts a coordinator runtime, the handler receives task context and a client, and resource operations flow through the session instead of through local Airflow internals.

Sources: task-sdk/README.md, ts-sdk/README.md

A session has two sides. On the user side, a task handler is ordinary application code registered against a Dag identity and task identity. On the coordinator side, Airflow supplies startup details, establishes a communication channel, and expects task completion, failure, retry, abort handling, and parse-mode behavior to follow a defined wire protocol. This matters because non-Python handlers run outside the traditional Python task runner path. They still need Variables, Connections, XCom, logs, cancellation signals, and return-value handling, but those operations must be mediated by the runtime.

Sources: ts-sdk/README.md, ts-sdk/tests/coordinator/integration.test.ts

Relevant Source Files

  • task-sdk/README.md - Identifies the Apache Airflow Task SDK package, its installation command, and its role as interfaces for Dag authors plus task execution logic for Python.
  • ts-sdk/README.md - Documents the TypeScript SDK status, task handler contract, coordinator usage, Python stub Dag pattern, coordinator configuration, bundle requirements, TaskClient usage, and return-value XCom behavior.
  • ts-sdk/tests/coordinator/client.test.ts - Tests the TaskClient contract around Variable and XCom lookup, exact not-found error handling, context-bound defaults, and override behavior.
  • ts-sdk/tests/coordinator/integration.test.ts - Exercises the coordinator runtime with an in-process supervisor, length-prefixed MessagePack frames, startup details, task success and failure, retry, abort signaling, task-time RPCs, missing handlers, and parse-mode responses.
  • kubernetes-tests/lang_sdk/README.md - Describes a KubernetesExecutor end-to-end scenario where language SDK coordinators route work by queue and stage Go and Java artifacts into worker pods through configured coordinator settings.

Core Session Primitives

The TypeScript SDK README shows the core public shape for a session. A handler receives TaskHandlerArgs, including ctx for task identity and client for task-time Airflow data access. The handler may call client methods such as getVariable, getConnection, and getXCom, then return a value. Non-undefined return values are pushed to XCom under the return_value key by the active runtime, matching Python task behavior. This keeps handler code focused on business logic while the coordinator owns the Airflow-specific exchange needed to read resources and publish results.

Sources: ts-sdk/README.md

The context bound to the client contains the runtime identity of the task instance: dag id, task id, run id, try number, map index, and an abort signal. The client tests demonstrate that requests default to this context and can be overridden when a lookup needs another task, map index, or related identity. This is especially important for mapped tasks and upstream XCom reads, because the handler should not have to reconstruct its own runtime identity. The session gives each request enough metadata to be interpreted by the Airflow backend consistently.

Sources: ts-sdk/tests/coordinator/client.test.ts

Authoring and Configuration Flow

The TypeScript coordinator mode deliberately separates scheduling shape from implementation. The README explains that declaring Airflow Dags in TypeScript is not supported yet; a Python Dag still declares stub tasks, dependencies, queues, and scheduling. The TypeScript bundle registers handlers with matching Dag and task identifiers, then calls startCoordinator. Airflow maps a queue to a configured coordinator entry, such as a NodeCoordinator classpath with a bundles root. That means deployment has two artifacts to keep aligned: the Python Dag that Airflow parses, and the language bundle that provides executable handlers.

Sources: ts-sdk/README.md

A typical authoring flow begins with a Python stub Dag that assigns language-specific work to a queue such as typescript. Next, the Airflow configuration declares a coordinator and maps that queue to it. Each bundle directory must contain bundle.mjs and airflow-metadata.yaml, giving the coordinator enough structure to discover and run the registered handlers. Finally, the TypeScript entrypoint imports every module that calls registerTask and starts the coordinator. If the Dag id or task id differs between the Python stub and the registered handler, the runtime cannot bind the session to the intended implementation.

Sources: ts-sdk/README.md

Runtime Protocol and Client Behavior

The integration test describes a pure Node supervisor that mirrors the real BaseCoordinator subprocess entrypoint closely enough to validate the wire format without Python or an Airflow install. It uses a TCP server, length-prefixed MessagePack frames, and request or response tuples. StartupDetails include the task instance identity, Dag relative path, bundle information, start date, task instance context, hostname, queue, try number, and map index. After startup, the runtime handles supervisor requests and runtime-initiated RPCs, producing task success, failure, retry, abort, missing-handler, and parse-mode responses under test.

Sources: ts-sdk/tests/coordinator/integration.test.ts

The client tests define important edge cases for resource access. getVariable returns null only for the exact VARIABLE_NOT_FOUND error code, and getXCom returns null only for the exact XCOM_NOT_FOUND code. Other ErrorResponse values throw, including strings that merely contain a not-found-looking substring. getVariableOrThrow strengthens the contract by throwing a VariableNotFoundError for a missing key or a null-valued result. These distinctions prevent silent data loss: absence is a deliberate API result, while backend failures and unexpected errors remain visible to the handler.

Sources: ts-sdk/tests/coordinator/client.test.ts

Kubernetes and Multi-Language Sessions

The Kubernetes language SDK system test shows why sessions are broader than a local Node subprocess detail. In coordinator mode on KubernetesExecutor, a Dag can mix Python, Go, and Java tasks, route work by queue, and use per-queue coordinator configuration with extra pod_template_file settings. The worker pod does not run the normal Python task runner for those language tasks, so artifacts are staged by an init container. That init container reuses the DagBundle interface to download language artifacts from object storage into the paths scanned by the coordinator.

Sources: kubernetes-tests/lang_sdk/README.md

This deployment model reinforces the session boundary. The scheduler and Dag processor still understand the Python stub Dag, queues, and dependencies, but the actual language runtime may be a Go binary, a Java jar, or a Node bundle inside a specialized pod. The shared contract is the coordinator session: startup metadata, task identity, task-time RPCs, signal handling, and result propagation. Operators should therefore treat queue routing, bundle metadata, artifact staging, and coordinator configuration as part of the same operational unit, not as optional packaging details.

Sources: kubernetes-tests/lang_sdk/README.md

Practical Checklist

  • Install the Python Task SDK when authoring Python Task SDK code: pip install apache-airflow-task-sdk.
  • Install the TypeScript SDK for Node handlers with pnpm add @apache-airflow/ts-sdk.
  • Declare the Dag and dependency graph in Python when using the current TypeScript SDK coordinator mode.
  • Register each non-Python handler with the exact Dag id and task id used by the Python stub task.
  • Configure the sdk coordinators mapping and queue_to_coordinator mapping so queued stub tasks reach the intended language runtime.
  • Ensure each TypeScript bundle directory contains bundle.mjs and airflow-metadata.yaml.
  • Handle null returns from getVariable and getXCom as intentional absence, and let other error responses fail loudly.

Read the Task SDK page for the broader package purpose and public Python authoring model, then use Streaming Logs and Channels for the lower-level communication and log-flow surfaces around coordinator sessions. For deployment, pair this page with Executor and Runtime Providers and Kubernetes and Helm, because queue routing, pod templates, bundles, and worker isolation determine whether the session can actually start. For authoring patterns, compare TaskFlow Tutorial and Dynamic Task Mapping when handlers consume upstream XCom values or execute across mapped task instances.

Ask a question...