Deferrable Operators and Triggers
Purpose and Scope
Deferrable operators are Airflow operators that can pause while they wait for an external condition, instead of occupying a worker slot during a long polling loop. A trigger is the asynchronous waiting component that performs that polling until an event or completion condition is reached. This page explains the pattern through the Amazon provider guide, because Amazon operators commonly start remote AWS operations that return quickly but complete later. The same mental model applies to many provider integrations: do synchronous setup, start the remote operation, then defer the wait to a trigger when possible.
Sources: providers/amazon/src/airflow/providers/amazon/aws/triggers/README.md
The Amazon provider documentation is intentionally framed as a conversion guide rather than a single universal recipe. It warns that Amazon Provider Package operators vary in complexity, so maintainers should study the operator before applying a deferrable pattern. That distinction matters for users too: enabling deferrable mode is not simply a different timeout value. It changes where the waiting work runs, how task state is resumed, and which part of the operator must be safe to execute before and after deferral.
Sources: providers/amazon/src/airflow/providers/amazon/aws/triggers/README.md
Relevant Source Files
providers/amazon/src/airflow/providers/amazon/aws/triggers/README.md- Provides the provider-maintainer guide for writing deferrable Amazon operators, including the three-stage AWS operator model, use ofdeferrable,self.defer,wait_for_completion, botocore waiters, and custom waiters.
Core Primitives
The first primitive is the ordinary operator execution path. In the Amazon provider guide, a typical operator begins with pre-processing, such as looking up information through boto3 API calls or formatting parameters. It then performs the main boto3 API call that starts the desired AWS operation. This can be resource provisioning, changing a resource state, or starting a job. The boto3 call returns a response immediately, ignoring network latency, even when the remote operation itself may take a long time to finish.
Sources: providers/amazon/src/airflow/providers/amazon/aws/triggers/README.md
The second primitive is the polling stage. Many AWS operations require repeated checks until a completion criterion is met. In a non-deferrable operator, that polling commonly happens inside the worker process. In a deferrable operator, the guide identifies this final stage as the best candidate for deferral. The operator should complete the setup and start the external work first, then hand off the waiting portion to a trigger that can poll asynchronously and wake the task when the condition has been satisfied or failed.
Sources: providers/amazon/src/airflow/providers/amazon/aws/triggers/README.md
The third primitive is the waiter. Botocore includes built-in waiters for some AWS services; a waiter encapsulates polling a service until a named condition is met. The Amazon provider guide also describes custom waiters that follow the same logic for services not covered by botocore. For deferrable Amazon operators, these built-in and custom waiters become the reusable waiting mechanism used by triggers, which keeps the operator-specific implementation focused on the business operation rather than on reinventing polling loops.
Sources: providers/amazon/src/airflow/providers/amazon/aws/triggers/README.md
Execution Flow
A deferrable Amazon operator should usually follow a staged execution flow. First, it runs any necessary pre-processing while the task is active. Second, it makes the main boto3 request that initiates work in AWS. Third, if the operator is configured to wait for completion and deferrable mode is enabled, it defers before entering a synchronous polling loop. This preserves the important side effect of starting the remote operation while avoiding worker occupancy during the potentially long period where Airflow is only waiting for AWS state to change.
Sources: providers/amazon/src/airflow/providers/amazon/aws/triggers/README.md
The guide gives a concrete placement rule for operators that already expose wait_for_completion. In that case, the self.defer call should be placed immediately before the check for wait_for_completion. This recommendation is subtle but useful: it keeps the old synchronous behavior available when deferral is not requested, while allowing deferrable mode to replace the blocking wait path. The code therefore has a clear branch point between continuing in-process and transferring the wait to a trigger.
Sources: providers/amazon/src/airflow/providers/amazon/aws/triggers/README.md
For operators without a simple wait_for_completion switch, the right deferral point depends on the operator implementation. The Amazon guide explicitly says there is no one method that works for every operator. Maintainers should identify the exact moment after the external AWS operation has been started and before local polling begins. Deferring too early can skip required setup or fail to create the remote resource; deferring too late leaves unnecessary blocking work in the worker and undermines the benefit of deferrable mode.
Sources: providers/amazon/src/airflow/providers/amazon/aws/triggers/README.md
System-to-Code Mapping
| Concept | Source-backed contract | Practical effect |
|---|---|---|
deferrable parameter | Added to the operator and initialized in the constructor | Lets callers opt into asynchronous waiting behavior |
| Pre-processing stage | boto3 lookups, parameter formatting, and other preparation | Still runs in the operator before deferral |
| Main boto3 call | Starts provisioning, state change, job execution, or similar AWS work | Must happen before the wait is deferred |
| Polling stage | Repeated checks until a completion criterion is met | Primary candidate for trigger-based deferral |
| Botocore waiter | Built-in polling abstraction for supported AWS services | Can provide trigger wait logic without custom polling code |
| Custom waiter | Provider extension of waiter logic for unsupported services | Allows deferrable behavior for services beyond botocore coverage |
The mapping is important because deferrable mode is not an all-or-nothing rewrite of an operator. Most of the operator remains ordinary synchronous Python: it validates inputs, calls boto3, and handles the initial response. The trigger only replaces the long-running observation phase. This separation keeps provider code understandable and preserves compatibility with existing operator semantics. It also helps reviewers reason about idempotency: the main AWS operation should not be accidentally repeated when the deferred task resumes.
Sources: providers/amazon/src/airflow/providers/amazon/aws/triggers/README.md
Implementation Details
When converting an Amazon operator, start by adding deferrable to the operator’s public parameters and storing it during construction. Then inspect the existing execute path and identify whether it already contains wait_for_completion. If it does, place self.defer immediately before that waiting branch. The trigger should then perform the waiter-driven polling and return control once completion is detected. The older synchronous waiting path remains useful for compatibility, testing, or deployments that do not want to use deferrable mode for that operator.
Sources: providers/amazon/src/airflow/providers/amazon/aws/triggers/README.md
The recommended trigger implementation should reuse waiter semantics where possible. Built-in botocore waiters are appropriate when AWS already defines the expected polling behavior. Custom waiters are appropriate when the service or condition is not included in botocore but can still be expressed as repeated API calls plus an acceptor condition. This gives Amazon provider maintainers a shared vocabulary for asynchronous waiting and reduces the risk that every deferrable operator develops slightly different retry, delay, and completion rules.
Sources: providers/amazon/src/airflow/providers/amazon/aws/triggers/README.md
Next Steps
If you are using Airflow, look for a provider operator’s deferrable option and its wait_for_completion behavior before assuming a task will release worker capacity while waiting. If you maintain a provider operator, model the code around the three stages from the Amazon guide and defer only the polling stage. Related topics to read next are tasks-operators-and-hooks for the base operator model, operators-and-hooks-reference for provider surfaces, and executor-and-runtime-providers for understanding how waiting behavior affects runtime capacity.