Executor and Runtime Providers
Purpose and Scope
Executor and runtime providers are the pieces of Airflow that decide where task instances actually run and how operators integrate with the environment around them. An executor is the scheduling back end used by Airflow to dispatch task work, such as Celery workers or Kubernetes pods. A runtime provider can also include supporting user-interface components, worker status views, language SDK coordination, pod templates, or deployment-specific behavior. This page orients operators and extension authors around the provider-backed execution model rather than a single deployment recipe.
The official provider documentation treats Celery and Kubernetes executors as installable provider capabilities. Celery scales workers through a broker-backed worker pool, while Kubernetes starts task instances in pods. The Edge provider source evidence in this repository shows the same provider pattern applied to runtime visibility: it contains a React UI plugin intended to provide details of Edge workers. Together, these examples show that Airflow runtime integration is not only scheduler configuration; it is a set of packaged components, documentation, commands, and UI surfaces that must fit the chosen deployment topology.
Relevant Source Files
providers/edge3/src/airflow/providers/edge3/plugins/www/README.md- Documents the Edge Provider UI plugin, its React library build, development commands, library output, consumption pattern, and plugin best practices for displaying Edge worker details.
Sources: providers/edge3/src/airflow/providers/edge3/plugins/www/README.md
Runtime Provider Model
Airflow separates the logical workflow from the runtime that executes it. DAG authors define tasks, dependencies, queues, and scheduling rules; platform operators decide whether those task instances run in Celery workers, Kubernetes pods, local processes, Edge workers, or another executor-backed environment. This separation matters because task code often has dependencies on Python packages, system libraries, credentials, network access, and compute shape. Runtime provider documentation should therefore answer two questions: how to enable the executor, and what must be present in the task execution environment.
The Celery provider documentation describes CeleryExecutor as a scale-out option that requires a Celery backend such as RabbitMQ, Redis, or Redis Sentinel, plus homogeneous Airflow configuration across the cluster. In that model, workers are long-running Airflow-capable hosts or containers, and the deployment responsibility is to keep the worker environment consistent with the scheduler and web/API components. The Kubernetes provider documentation describes KubernetesExecutor as running each task instance in its own pod, shifting more of the isolation and dependency boundary into pod templates, images, Kubernetes configuration, and provider-specific settings.
Runtime isolation is the common design pressure behind these integrations. A shared worker fleet is efficient, but it can become difficult to operate when every task requires a different dependency set. Per-task pods improve isolation, but require Kubernetes-native packaging and pod configuration. Edge-style runtimes introduce another shape: workers may be distributed closer to workload locations and need UI visibility into their state. The Edge Provider UI plugin README supports this by describing a plugin component built for Airflow UI consumption to provide details of Edge workers. Sources: providers/edge3/src/airflow/providers/edge3/plugins/www/README.md
System-to-Code Mapping
The Edge3 source path included for this page is not the executor itself; it is a provider UI plugin template. That distinction is important. Runtime providers frequently include more than scheduler-facing Python classes. They may package frontend components, configuration references, CLI commands, examples, test fixtures, and installation metadata. In this case, the README describes a React component library that can be consumed by other applications, with TypeScript declarations, source maps, CSS injection, and externalized dependencies. Those choices are implementation details of a UI extension, but they directly support the operator-facing need to inspect Edge worker information.
The plugin development flow uses pnpm dev for hot reload, pnpm build for production output, pnpm build:types for declaration files, pnpm build:lib for the JavaScript library, and separate pnpm test, pnpm lint, and pnpm format commands. The documented library output includes dist/main.js as the ES module JavaScript library and dist/main.d.ts as the TypeScript declaration file. The README also notes that usable UI development should run Airflow in parallel with breeze start-airflow, which connects the provider plugin workflow to the repository’s local development environment. Sources: providers/edge3/src/airflow/providers/edge3/plugins/www/README.md
Execution Flow by Provider Type
For Celery-based deployments, the flow starts with installing the Celery provider or the celery extra, configuring executor = CeleryExecutor, and providing the broker and result-backend related settings. The scheduler queues task instances, Celery transports them through the broker, and workers execute them in an Airflow environment that has the required operator dependencies. The operational risk is usually environmental drift: if a task imports a dependency available on one worker but not another, behavior depends on where the task lands.
For Kubernetes-based deployments, the flow starts with installing the CNCF Kubernetes provider or the cncf.kubernetes extra, then configuring the Kubernetes executor and its pod-related settings. The scheduler asks Kubernetes to launch a pod for each task instance. This model aligns well with image-based dependency isolation and cluster-native resource controls. It also means executor documentation must be read together with deployment documentation: service accounts, namespaces, pod templates, images, secrets, volumes, and logs all become part of the task runtime contract.
For Edge-style runtimes, the flow includes a runtime control plane plus visibility into distributed workers. The provided source path documents the UI plugin side: a React component library built with Vite and TypeScript, exported for consumption by Airflow UI integration points. The README’s best practices are runtime-provider relevant: keep React external to avoid dependency conflicts, use a standardized global name such as AirflowPlugin, implement error boundaries and fallbacks, type plugin props and exports, and monitor bundle size by externalizing large dependencies. Sources: providers/edge3/src/airflow/providers/edge3/plugins/www/README.md
Compact Reference
| Area | Concrete names and responsibilities |
|---|---|
| Celery provider | CeleryExecutor scales task execution through Celery workers and requires a broker/backend plus worker environments with Airflow and task dependencies installed. |
| Celery hybrid mode | CeleryKubernetesExecutor appears in provider documentation as a combined executor option for deployments that route work between Celery and Kubernetes styles. |
| Kubernetes provider | KubernetesExecutor runs each task instance in a Kubernetes pod; LocalKubernetesExecutor is documented by the same provider family for mixed local and pod-based execution. |
| Edge runtime UI | providers/edge3/src/airflow/providers/edge3/plugins/www/README.md documents the Edge Provider UI plugin for displaying Edge worker details. |
| Edge plugin commands | pnpm dev, pnpm build, pnpm build:types, pnpm build:lib, pnpm test, pnpm lint, and pnpm format. |
| Edge plugin outputs | dist/main.js, dist/main.d.ts, and source maps generated by the documented library build. |
The Edge plugin reference is useful even for readers focused on executor selection because it shows how runtime providers expose operational state to users. A production runtime integration is not complete merely because the scheduler can dispatch tasks; operators need a way to understand where work is running, whether workers are healthy, and how provider-specific status appears in the UI. The documented plugin library pattern gives maintainers a route for adding that visibility without bundling conflicting React dependencies into Airflow’s frontend. Sources: providers/edge3/src/airflow/providers/edge3/plugins/www/README.md
Operational Guidance and Next Steps
Choose an executor by starting from the workload boundary. If tasks can share a homogeneous Python and system environment, Celery is often a straightforward scale-out model. If tasks need stronger isolation, Kubernetes pod-per-task execution can reduce dependency coupling at the cost of Kubernetes configuration complexity. If tasks run in remote or specialized worker environments, look for provider documentation that explains both execution semantics and observability surfaces. In all cases, verify installation extras, provider package versions, worker dependency availability, queue routing, logging behavior, and how failures appear in the Airflow UI.
When extending runtime providers, treat UI, packaging, and testing as part of the public operator experience. The Edge Provider UI plugin README gives concrete next steps for local frontend development: run the development server with hot reload, run Airflow in parallel through Breeze, build the library for production, and keep generated types available for consumers. From here, read the scheduler, Kubernetes and Helm, Docker stack, task logs, and provider extension contract pages to connect executor selection with deployment, observability, and extension maintenance. Sources: providers/edge3/src/airflow/providers/edge3/plugins/www/README.md