Pools and Priority Weights
Purpose and Scope
Pools are Airflow’s administrative control for limiting how much work of a particular kind can run at once. A pool has a fixed number of slots, and tasks assigned to that pool consume slots while they are eligible to occupy capacity. This gives operators a way to protect scarce external systems, such as databases, APIs, GPU clusters, or shared warehouses, without changing every DAG’s schedule. Priority weights complement pools by helping the scheduler decide which runnable task should be favored when there is more eligible work than immediate capacity.
Airflow’s official administration documentation treats pools and priority weights as deployment-time tuning tools rather than DAG modeling concepts. The important distinction is that a schedule determines when work becomes eligible, while pools and priority weights influence how eligible work competes for constrained execution resources. A DAG author can still express dependencies, assets, retries, and mapping as usual; pools then apply a cluster-level concurrency budget, and priority weights provide ordering pressure when multiple task instances are waiting for workers or pool slots.
Sources: airflow-core/src/airflow/ui/src/components/PoolBar.test.tsx
Relevant Source Files
airflow-core/src/airflow/ui/src/components/PoolBar.test.tsx- Tests the UI-facingPoolBarcomponent using the generatedPoolResponsetype and validates how occupied, open, deferred, queued, scheduled, and running slot fields are represented for pool usage.
Core Terms
A pool is identified by a name, such as the default default_pool, and has a configured slot count. A slot is the accounting unit for pool capacity. A task can use one or more pool slots, so an expensive operation can be made proportionally more costly than a lightweight one. The source-backed PoolResponse shape used by the UI includes slots, open_slots, occupied_slots, queued_slots, running_slots, scheduled_slots, and deferred_slots, which are the names a reader will see reflected in pool status surfaces and API-derived UI state.
The UI test also exposes a subtle but operationally important setting: include_deferred. Deferred tasks are tasks waiting asynchronously, commonly through deferrable operators and triggers, rather than occupying a worker process in the traditional running state. When include_deferred is false, deferred slots are presented as secondary information and are not treated as consuming pool capacity in the pool bar. When it is true, deferred tasks are included in the usage bar and reduce open slot capacity. This distinction matters when a deployment has many long waits and wants either strict external-system budgeting or maximum worker efficiency.
Sources: airflow-core/src/airflow/ui/src/components/PoolBar.test.tsx
System-to-Code Mapping
The tested PoolBar receives a pool object and a totalSlots value, then renders pool capacity as a usage visualization. The test factory creates a PoolResponse with slots: 128, open_slots: 128, occupied_slots: 1, deferred_slots: 1, and state-specific counters for queued, running, and scheduled tasks. Those field names are useful because they describe the boundary between backend scheduling state and frontend observability: the scheduler and API provide pool accounting, while the UI turns that accounting into links, labels, and progress-bar segments for operators.
In the first tested case, deferred work does not consume pool slots. The test renders PoolBar with include_deferred: false, then asserts that there is no link containing task_state=deferred while the deferred count is still visible as secondary information. In the second case, the test renders with include_deferred: true and open_slots: 127, then expects a deferred-state link to be present. This tells operators how to interpret the UI: a deferred count may either be informational or part of the visible slot usage, depending on pool configuration.
Sources: airflow-core/src/airflow/ui/src/components/PoolBar.test.tsx
Execution Flow for Capacity Control
A practical capacity-control workflow starts by identifying a shared bottleneck. For example, if many DAGs call the same partner API, create a pool representing that API’s safe concurrency limit. Assign tasks to the pool and set pool slots according to cost: ordinary calls may consume one slot, while bulk exports may consume several. Once the scheduler has more runnable task instances than available slots, pool accounting prevents all of them from executing at the same time. Operators then use the pool UI to see whether slots are open, occupied, queued, scheduled, running, or deferred.
Priority weights enter after tasks are eligible but capacity is contested. When many task instances are waiting, priority weights give Airflow a way to prefer more important work. Official documentation separates this from pools because the two controls answer different questions: pools ask whether capacity is available for a task’s resource class, while priority weights help choose among tasks that could otherwise run. Custom weight rules can further adapt this behavior for organizations that need queueing policies based on downstream impact, business priority, or DAG-specific conventions.
Deferred Tasks and Pool Slots
Deferred tasks deserve special attention because they can wait outside a worker while still representing pressure on an external system. If a deferred sensor is merely waiting for an event and does not burden the target system, excluding deferred tasks from pool consumption can improve throughput. If the deferred state represents a reservation, lease, or outstanding request against a limited service, including deferred tasks in pool usage keeps the pool aligned with real-world capacity. The tested include_deferred behavior gives the UI a way to make that policy visible rather than hiding it behind scheduler internals.
The visible result is intentionally different in each mode. When deferred slots are excluded, the operator still sees the deferred count, but the bar does not treat it as a pool-usage segment and does not provide the same deferred-state filtering link. When deferred slots are included, the deferred state becomes part of usage navigation. That matters during incident response: a pool that appears full because of deferred work should lead the operator to different next steps than a pool saturated by actively running tasks.
Sources: airflow-core/src/airflow/ui/src/components/PoolBar.test.tsx
Compact Reference
| Concept | Practical meaning | Source-backed UI/API field |
|---|---|---|
| Pool name | Identifier used to group capacity | name |
| Total capacity | Configured number of slots in the pool | slots, totalSlots |
| Available capacity | Slots not currently consumed | open_slots |
| Used capacity | Slots currently accounted as occupied | occupied_slots |
| Deferred accounting | Whether deferred work consumes pool capacity | deferred_slots, include_deferred |
| State breakdown | Operator-facing counts by task state | queued_slots, running_slots, scheduled_slots |
Use pools when the same external resource is shared by many DAGs or task groups. Use multiple pool slots when one task is heavier than another. Use priority weights when the problem is not total capacity, but selection among competing work. Review deferred-slot policy whenever deferrable operators are introduced at scale, because the correct setting depends on whether a deferred task is still consuming the protected resource. For deeper operational context, continue with the scheduler, deferrable operators and triggers, and metrics and health-checks pages.