Assets

Purpose and Scope

Assets are Airflow’s data-aware scheduling primitive: they describe externally meaningful data objects, such as files, tables, topics, or other integration-specific resources, that tasks can produce and downstream DAGs can depend on. In DAG authoring, an asset is not the bytes or rows themselves; it is a stable identifier and optional metadata that lets Airflow record that something happened to a data object. The official documentation frames this through asset definitions, asset events, asset-aware schedules, conditional asset expressions, aliases, partitions, and event-driven scheduling.

This page orients maintainers and DAG authors to the source-backed parts visible in this repository slice. The concrete code examples here come from the Apache HDFS and Apache Hive providers, where each provider exposes an asset URI scheme and helper functions for constructing, validating, and translating assets. Those provider modules show the public shape Airflow expects from integration-specific asset schemes: a URI format, a factory that returns an Asset, validation rules for the URI, and optional conversion into an OpenLineage dataset.

Sources: providers/apache/hdfs/src/airflow/providers/apache/hdfs/assets/hdfs.py, providers/apache/hive/src/airflow/providers/apache/hive/assets/hive.py

Relevant Source Files

docs/images/documentation_architecture.py - Generates the architecture image for the Airflow documentation publication flow, showing how repository docs become the published documentation site that contains the asset authoring and scheduling pages.
providers/apache/hdfs/src/airflow/providers/apache/hdfs/assets/__init__.py - Marks the HDFS provider asset package; the implementation lives in the sibling hdfs.py module.
providers/apache/hdfs/src/airflow/providers/apache/hdfs/assets/hdfs.py - Implements the HDFS asset URI scheme, including URI validation, an Asset factory, and OpenLineage conversion.
providers/apache/hive/src/airflow/providers/apache/hive/assets/__init__.py - Marks the Hive provider asset package; the implementation lives in the sibling hive.py module.
providers/apache/hive/src/airflow/providers/apache/hive/assets/hive.py - Implements the Hive asset URI scheme, including URI validation, an Asset factory, and OpenLineage conversion.

Core Primitives

An Asset is identified by a URI. The provider examples import Asset from airflow.providers.common.compat.assets, then return Asset(uri=..., extra=extra) from provider-specific factory functions. This is important because it keeps DAG code integration-aware without making every author hand-build URI strings. For HDFS, the helper accepts a host, path, optional port defaulting to 8020, and optional extra metadata. For Hive, the helper accepts a host, database, table, optional port defaulting to 10000, and optional extra metadata.

Sources: providers/apache/hdfs/src/airflow/providers/apache/hdfs/assets/hdfs.py, providers/apache/hive/src/airflow/providers/apache/hive/assets/hive.py

An asset event is the runtime record that a task emitted or updated an asset. In authoring terms, producers create events by declaring task outlets or using asset-aware APIs, and consumers schedule DAGs from those events. Asset state is the accumulated view Airflow uses to decide whether a data-aware schedule is satisfied. When you see official docs discuss fetching information from previously emitted asset events, triggering asset events in Python, or Jinja access to events, they are describing how the scheduling layer exposes that state back to DAG code.

Asset expressions extend single-asset scheduling into conditional scheduling. Instead of saying “run when this one asset updates,” authors can define schedules over multiple assets using logical conditions. The official documentation describes logical operators for assets and advanced asset scheduling with conditional expressions. The provider code does not implement the expression evaluator; it supplies the integration-specific asset identities that can participate in expressions. In practice, the identity must be valid and stable, because expression state is only meaningful when the same logical data object resolves to the same asset URI over time.

Provider Asset Schemes

Provider asset schemes are how community-managed integrations attach Airflow’s generic asset model to concrete systems. The HDFS provider declares the hdfs scheme through a module that can build URIs like hdfs://host:8020/path and validate that the parsed URI contains both a namenode host and a path. These validation checks are small, but they encode a key contract: asset URIs are not arbitrary labels. They must contain enough structure for Airflow, integrations, and lineage tools to agree on the resource being referenced.

Sources: providers/apache/hdfs/src/airflow/providers/apache/hdfs/assets/hdfs.py

The Hive provider follows the same pattern with Hive-specific shape. Its factory builds hive://host:10000/database/table, and its sanitizer requires a host plus a path that splits into the leading slash, database name, and table name. The error message says the URI must contain database, schema, and table names, while the implementation maps the two path components to database and table. For users, the practical rule is to create Hive assets with the helper rather than improvising strings, because the helper preserves the provider’s expected URI structure and default port.

Sources: providers/apache/hive/src/airflow/providers/apache/hive/assets/hive.py

Both provider modules also expose convert_asset_to_openlineage. That function translates an Airflow asset into an OpenLineage dataset object after parsing the asset URI. HDFS uses the URI network location as the OpenLineage namespace and the path, stripped of its leading slash, as the dataset name, defaulting to / when empty. Hive uses the Hive network location as the namespace and combines the database and table path components into a database.table dataset name. This demonstrates how asset identity can serve scheduling and lineage at the same time.

Sources: providers/apache/hdfs/src/airflow/providers/apache/hdfs/assets/hdfs.py, providers/apache/hive/src/airflow/providers/apache/hive/assets/hive.py

Compact Reference

Component	Public entry point	Required inputs	Defaults	Behavior
HDFS asset factory	`create_asset(*, host: str, path: str, port: int = 8020, extra: dict	None = None) -> Asset`	`host`, `path`	`port=8020`, `extra=None`
HDFS URI sanitizer	`sanitize_uri(uri: SplitResult) -> SplitResult`	Parsed `hdfs://` URI	None	Requires `netloc` and `path`; raises `ValueError` when either is absent.
HDFS OpenLineage conversion	`convert_asset_to_openlineage(asset: Asset, lineage_context) -> OpenLineageDataset`	Airflow `Asset`	None	Converts namespace to `hdfs://{parsed.netloc}` and name to the URI path without the leading slash.
Hive asset factory	`create_asset(*, host: str, database: str, table: str, port: int = 10000, extra: dict	None = None) -> Asset`	`host`, `database`, `table`	`port=10000`, `extra=None`
Hive URI sanitizer	`sanitize_uri(uri: SplitResult) -> SplitResult`	Parsed `hive://` URI	None	Requires `netloc` and a two-component path after the leading slash; raises `ValueError` otherwise.
Hive OpenLineage conversion	`convert_asset_to_openlineage(asset: Asset, lineage_context) -> OpenLineageDataset`	Airflow `Asset`	None	Converts namespace to `hive://{parsed.netloc}` and name to `database.table`.

The reference table also highlights a convention that applies beyond these two providers: asset factories should make common DAG-authoring cases safer than raw URI construction. A DAG author can still understand and inspect the URI, but the provider owns the canonical format for its service. The sanitizer functions are equally important for integrations that receive URIs from configuration, serialized DAGs, REST payloads, or provider metadata. They make invalid state fail early with actionable error messages rather than letting malformed identifiers propagate into scheduling, lineage, or UI surfaces.

Sources: providers/apache/hdfs/src/airflow/providers/apache/hdfs/assets/hdfs.py, providers/apache/hive/src/airflow/providers/apache/hive/assets/hive.py

System-to-Code Mapping

At the system level, asset-aware scheduling begins in DAG authoring: a task emits an asset event, and a DAG can be scheduled based on one or more assets. The source files in this page sit at the integration boundary, not the scheduler core. They answer the question “what does this external object look like as an Airflow asset?” HDFS answers with a namenode and path; Hive answers with a metastore host, database, and table. Once a stable asset identity exists, Airflow’s scheduling and event surfaces can reason about it consistently.

The extra argument in both factories is a useful design signal. Asset identity belongs in the URI, while supplemental information belongs in metadata. That separation matters because scheduling should be driven by a durable resource identifier, not by incidental run-specific context. Extra information can enrich event records or downstream processing, but changing extra should not be used as a substitute for choosing the correct URI. When modeling assets, put the resource’s identity in the scheme, host, and path components, and reserve metadata for annotations.

The documentation architecture script is not part of runtime asset scheduling, but it explains why asset documentation is maintained as a first-class published documentation area. It builds a diagram showing Airflow GitHub repositories, the package docs publication workflow, S3-backed live docs, the Apache webserver, and CloudFront caching. For this page, that matters because Airflow’s user-facing asset documentation is generated and published from repository-controlled sources, while provider modules supply the concrete implementation hooks that make documented URI schemes real.

Sources: docs/images/documentation_architecture.py

Execution Flow

A typical asset workflow starts when a DAG author chooses the resource boundary that should trigger downstream work. For HDFS, that might be a directory or file path produced by an ingestion task. For Hive, it might be a table produced by a transformation job. The author constructs the corresponding Asset using the provider helper, attaches it to the producing task according to the authoring API, and then declares a consuming DAG schedule that references the same asset or an expression over multiple assets.

When the producing task succeeds and emits an asset event, Airflow records that event as scheduling state. A downstream schedule can then become eligible when its asset condition is satisfied. With multiple assets, expressions allow authors to describe whether all inputs, any input, or another logical combination should trigger a run. The provider code’s role remains intentionally narrow during this flow: it keeps the resource identifier valid, recognizable, and convertible. The scheduler can then treat HDFS and Hive assets as instances of the same Airflow asset abstraction.

Lineage conversion is a second flow that can happen alongside scheduling. If an integration needs to report OpenLineage datasets, the provider conversion functions parse the same asset URI and map it into OpenLineage namespace/name pairs. That avoids inventing a separate naming system for lineage. It also means mistakes in asset URI construction can affect both scheduling clarity and lineage quality. Provider factories and sanitizers are therefore not just conveniences; they are part of the reliability boundary between DAG code, provider integrations, and external observability systems.

Sources: providers/apache/hdfs/src/airflow/providers/apache/hdfs/assets/hdfs.py, providers/apache/hive/src/airflow/providers/apache/hive/assets/hive.py

Authoring Guidance and Next Steps

Prefer provider helpers when one exists for the system you are modeling. For the HDFS and Hive examples, helper functions encode the default ports and URI layout expected by the provider. Keep URIs stable across DAG parses and deployments, because asset state and expressions depend on stable identity. Use extra for descriptive or event-specific details, not for the identifying parts of the external resource. If a URI cannot be validated by the provider’s sanitizer, fix the model before relying on it in a schedule.

After defining individual assets, move to asset-aware scheduling concepts: multiple assets, triggering event inspection, Jinja access to triggering events, Python access to event metadata, aliases, partitions, and event-driven scheduling. If you are extending Airflow with a new provider, use the HDFS and Hive modules as compact examples of the provider-side contract: expose an asset package, validate a scheme-specific URI, provide a safe factory, and translate into OpenLineage when the integration can describe a dataset consistently.

Ask a question...