Asset and Event Scheduling

Purpose and Scope

Asset and event scheduling is the part of Airflow authoring where a DAG run can be created because data changed, not only because a clock reached a cron boundary. The official documentation presents this as asset-aware scheduling, with assets, asset events, asset aliases, conditional expressions, and event-driven DAG use cases. In practical terms, an asset represents a named data dependency, and an emitted asset event records that something new is available. A consuming DAG can then be scheduled from that event, letting workflow authors model data freshness directly instead of encoding all coordination as time-based polling.

Event-driven scheduling extends the same idea to external systems. The official event scheduling guide separates push-based scheduling through the REST API from pull-based scheduling through asset watchers. The Google provider evidence in this repository shows the pull-based shape: a common messaging trigger watches a Google Pub/Sub subscription and is attached to an Airflow asset through an asset watcher. That placement is important because the event source is provider-specific, but the consuming DAG still uses Airflow-level scheduling concepts such as assets and triggering asset events.

Sources: providers/google/src/airflow/providers/google/event_scheduling/events/pubsub.py

Relevant Source Files

• docs/images/documentation_architecture.py — Generates a documentation architecture diagram and shows that the repository contains code used to publish and maintain Airflow documentation artifacts, including package documentation that explains scheduling features.
• providers/google/src/airflow/providers/google/event_scheduling/init.py — Marks the Google provider event-scheduling package namespace for provider-level event scheduling integrations.
• providers/google/src/airflow/providers/google/event_scheduling/events/init.py — Marks the provider events namespace where event-source-specific scheduling integrations live.
• providers/google/src/airflow/providers/google/event_scheduling/events/pubsub.py — Defines the Google Pub/Sub message queue provider container used by common-messaging event scheduling.
• airflow-core/src/airflow/migrations/versions/0016_2_9_2_remove_idx_last_scheduling_decision_.py — Records a metadata database migration touching the DAG run scheduling-decision index, which is relevant when reasoning about scheduler state and upgrades.

Core Primitives

The first primitive is the asset. In Airflow documentation, assets are named data objects that tasks can emit events for and DAGs can depend on. The second primitive is the asset event, which is the occurrence that a producer has updated or observed an asset. A scheduled DAG may inspect the triggering event in templates or Python to understand what caused the run. The third primitive is the watcher, which connects an asset to an event trigger. Together, these let authors describe a data dependency, an event source, and a consumer without tying every DAG run to a fixed timetable.

The provider implementation makes the external-event boundary explicit. PubSubMessageQueueEventTriggerContainer inherits from BaseMessageQueueProvider, declares the provider scheme as google+pubsub, and returns PubsubPullTrigger from its trigger_class method. That means an Airflow common-messaging trigger can resolve a URL-like scheme to the Google Pub/Sub trigger implementation. The docstring example shows a MessageQueueTrigger configured with a project, subscription, acknowledgement behavior, maximum messages, connection id, and polling interval, then attached to an Asset through an AssetWatcher.

Sources: providers/google/src/airflow/providers/google/event_scheduling/events/pubsub.py

Execution Flow

A typical pull-based event scheduling flow starts with a DAG author defining an asset that represents a queue, topic-derived data feed, file family, or other external signal. The author attaches an asset watcher whose trigger knows how to listen to the external system. For Pub/Sub, the trigger is configured through common-messaging with the Google-specific scheme and Pub/Sub parameters. The trigger waits outside the task body, detects a message, and produces an event that Airflow can treat as evidence that the watched asset changed or became available.

After the asset event is observed, the scheduler can create DAG runs for DAGs whose schedules depend on that asset. This keeps the DAG definition focused on the dependency graph while provider packages own the integration details. It also allows multiple event sources to share a common contract. A Pub/Sub-backed asset watcher and another provider-backed watcher can both feed asset events into the Airflow scheduling model, even though their cloud APIs and authentication settings differ. The Airflow-facing abstraction is the asset event, not the provider’s native message object.

A simple authoring shape, based on the provider docstring, is:

from airflow.providers.common.messaging.triggers.msg_queue import MessageQueueTrigger
from airflow.sdk import Asset, AssetWatcher
 
trigger = MessageQueueTrigger(
    scheme="google+pubsub",
    project_id="my_project",
    subscription="my_subscription",
    ack_messages=True,
    max_messages=1,
    gcp_conn_id="google_cloud_default",
    poke_interval=60.0,
 )
 
asset = Asset(
    "pubsub_queue_asset",
    watchers=[AssetWatcher(name="pubsub_watcher", trigger=trigger)],
)

The official guidance also warns about event-driven DAGs that can schedule themselves indefinitely. In data-aware workflows, this usually means authors should be careful when the same DAG both consumes and emits a closely related event, or when a watcher observes a signal that the DAG itself continuously creates. A good design makes the triggering condition narrow, idempotent, and tied to externally meaningful data availability. For Pub/Sub, settings such as acknowledging messages and limiting the number of pulled messages are part of controlling how each external event maps to scheduling decisions.

Sources: providers/google/src/airflow/providers/google/event_scheduling/events/pubsub.py

System-to-Code Mapping

The Google provider package layout separates the namespace from the implementation. The package files under event scheduling and events establish import locations for provider event scheduling, while the Pub/Sub file supplies the concrete message queue provider. This mirrors Airflow’s broader extension model: core concepts such as assets, triggers, and scheduling are stable Airflow surfaces, and provider packages add cloud-specific connections to those surfaces. When adding a new event source, follow this separation by keeping the public provider container small and delegating the actual asynchronous waiting to a trigger class.

Documentation support is also visible in the repository. The documentation architecture script builds an image that connects the Airflow repository, the website repository, release managers, S3-backed package documentation, CloudFront, and the live documentation site. That matters for this page because asset-aware and event-driven scheduling are not only implementation details; they are documented authoring features. The source tree therefore contains both provider code that enables one integration and documentation tooling that helps publish the official conceptual and how-to material around it.

Sources: docs/images/documentation_architecture.py, providers/google/src/airflow/providers/google/event_scheduling/init.py, providers/google/src/airflow/providers/google/event_scheduling/events/init.py, providers/google/src/airflow/providers/google/event_scheduling/events/pubsub.py

Operational and Migration Signals

Event scheduling ultimately creates normal Airflow scheduling records, so it depends on the same metadata database discipline as time-based scheduling. The migration evidence here removes the idx_last_scheduling_decision index from the dag_run table for Airflow 2.9.2 and provides a downgrade that recreates it. The page should not be read as saying this migration implements asset scheduling itself. Instead, it is a reminder that scheduler behavior is coupled to database schema evolution, and operators should apply migrations as part of upgrades before judging event-scheduling performance or correctness.

From an operational perspective, treat watchers and triggers as long-lived integration points. They need provider dependencies installed, connection ids configured, and external permissions that allow reading the event source. For the Pub/Sub path, the example names google_cloud_default, a project, and a subscription, so deployments must ensure the Airflow environment can authenticate to that subscription. Authors should also decide whether message acknowledgement belongs in the trigger configuration and how many messages should map to one scheduling event, because those settings affect duplicate runs and missed events.

Sources: airflow-core/src/airflow/migrations/versions/0016_2_9_2_remove_idx_last_scheduling_decision_.py, providers/google/src/airflow/providers/google/event_scheduling/events/pubsub.py

Compact Reference

Next Steps

Read the Assets page first if you need the vocabulary for assets, emitted events, aliases, partitions, and fetching event metadata. Then read the Scheduling, Cron, Timetables, and Timezones page to compare asset schedules with time-based schedules. For implementation work, continue to Deferrable Operators and Triggers because event-driven scheduling depends on trigger behavior and asynchronous waiting. If you are installing integrations, use the Providers Overview and Installation page before relying on provider-specific schemes such as the Google Pub/Sub common-messaging provider.