Operators and Hooks Reference
Purpose and Scope
The operators and hooks reference is the discovery surface for Airflow’s executable integration APIs. An operator is the task implementation a DAG author places in a workflow, while a hook is the reusable connection and client layer that operators use to talk to external systems. Airflow keeps a small core set of shared hook foundations in the main distribution and publishes most integration-specific operators, sensors, transfers, hooks, and related classes through provider packages. This split lets DAG authors install only the integrations they need while still using a consistent Airflow task model.
Sources: airflow-core/src/airflow/hooks/README.md, providers/apache/beam/src/airflow/providers/apache/beam/README.md
The official reference pages reflect that packaging model. The Airflow core documentation points readers to core modules and notes that many commonly used operators and sensors come from provider distributions such as the standard provider. The providers documentation then lists independently released integrations by ecosystem, such as ASF projects, cloud platforms, protocols, services, and software integrations. When you are looking for an operator or hook, the practical question is not only “what class should I import?” but also “which distribution installs that class and what optional extras unlock its full feature set?”
Sources: airflow-core/src/airflow/hooks/README.md, providers/apache/beam/src/airflow/providers/apache/beam/README.md
Relevant Source Files
airflow-core/src/airflow/hooks/README.md— Defines Airflow hooks as interfaces to external platforms and databases, explains that hooks implement a common interface, and states that core hooks are included by default and can be inherited by provider-distributed hooks.providers/apache/beam/src/airflow/providers/apache/beam/README.md— Documents the Apache Beam provider package, including its package name, installation command, cross-provider extras, and the provider class summary for Beam operators and hooks.
Core and Provider Responsibilities
Core Airflow provides the hook concept and the base integration surface. The core hooks README describes hooks as interfaces to external platforms and databases and explicitly calls them building blocks for operators. That definition is important for reading provider references: a hook is not usually the DAG-facing object by itself, but it is often the part that owns credentials, clients, connection resolution, and service-specific API calls. Operators then compose that lower-level behavior into schedulable Airflow tasks that participate in retries, task instances, templating, deferral, logging, and the UI.
Sources: airflow-core/src/airflow/hooks/README.md
Provider packages carry the integration-specific public classes. The Apache Beam provider README states that all classes for the provider package live under the airflow.providers.apache.beam Python package, and its class summary separates operators from hooks. That separation is the same mental model to use across provider references: locate the distribution, confirm the import package, inspect the operator classes intended for DAG authoring, and then inspect hooks when you need lower-level behavior or when you are writing custom operators that reuse provider logic.
Sources: providers/apache/beam/src/airflow/providers/apache/beam/README.md
Discovery Workflow
Start with the stable Airflow operators and hooks reference when you are unsure whether a class is part of core Airflow or the default provider set. The official page identifies base modules such as hook, operator, and sensor foundations, then points to providers for the much larger integration catalog. If the integration is independent of Airflow core, move to the provider operators and hooks reference. That provider index is grouped by integration family, making it easier to scan by technology area rather than by Python module name.
Sources: airflow-core/src/airflow/hooks/README.md
After choosing an integration family, read the provider package page before copying imports into a DAG. The Beam provider README shows the information that a provider reference should answer: the PyPI distribution name is apache-airflow-providers-apache-beam, the provider namespace is airflow.providers.apache.beam, and installation is performed on top of an existing Airflow installation. It also documents optional cross-provider dependencies, which matter when an operator delegates part of its behavior to another provider, such as enabling Google support for Beam integrations.
Sources: providers/apache/beam/src/airflow/providers/apache/beam/README.md
A useful discovery pattern is: identify the provider distribution, install it into the same environment as Airflow, then import only the public operator or hook classes documented by that provider. For Beam, the README lists operators.beam.BeamRunJavaPipelineOperator and operators.beam.BeamRunPythonPipelineOperator as new Airflow 2.0 operators, and hooks.beam.BeamHook as the new hook. Those entries tell you both the class names and the source module family, which is enough to move from reference documentation to implementation details or DAG authoring.
Sources: providers/apache/beam/src/airflow/providers/apache/beam/README.md
Compact Reference
| Surface | Source-backed names | How to use it |
|---|---|---|
| Core hooks concept | Airflow Hooks | Interfaces to external platforms and databases; common building blocks for operators. |
| Core hook location | airflow-core/src/airflow/hooks/README.md | Included by default in Airflow and available for provider hooks to inherit from. |
| Beam provider distribution | apache-airflow-providers-apache-beam | Install on top of an existing Airflow installation. |
| Beam provider namespace | airflow.providers.apache.beam | Python package where the provider classes are located. |
| Beam installation command | pip install apache-airflow-providers-apache-beam | Installs the Beam provider distribution. |
| Beam optional extra | pip install apache-airflow-providers-apache-beam[google] | Installs the provider with the documented Google cross-provider dependency. |
| Beam operators | operators.beam.BeamRunJavaPipelineOperator, operators.beam.BeamRunPythonPipelineOperator | DAG-facing task classes for running Beam pipelines. |
| Beam hook | hooks.beam.BeamHook | Provider hook for reusable Beam integration behavior. |
System-to-Code Mapping
The repository layout mirrors the documentation split between core and providers. Core hook documentation lives under airflow-core, which signals that the hook abstraction is part of the Airflow implementation included by default. Provider documentation lives under providers/apache/beam/src/airflow/providers/apache/beam, matching the import namespace described by the Beam README. That mapping is deliberate: a provider’s documentation, Python package, and distribution metadata are kept close together so maintainers can publish integration-specific documentation independently from the Airflow core release cadence.
Sources: airflow-core/src/airflow/hooks/README.md, providers/apache/beam/src/airflow/providers/apache/beam/README.md
This organization also explains why the reference is broader than a single API page. For a DAG author, an operator reference answers “which task class should I use?” For a platform maintainer, the provider page answers “which package and extras must be installed in the Airflow environment?” For an extension author, the hook reference answers “which reusable integration layer should my custom operator call?” The Beam README demonstrates all three concerns by listing the package, the install command, the optional dependency extra, and the operator and hook class summary in one provider-focused document.
Sources: providers/apache/beam/src/airflow/providers/apache/beam/README.md
Implementation Details and Usage Guidance
When using an operator from a provider, treat the provider package as part of your Airflow deployment contract. A DAG that imports BeamRunPythonPipelineOperator or BeamRunJavaPipelineOperator will not be portable to an environment that only installed core Airflow unless the Beam provider is also present. If the DAG uses functionality that depends on another provider, install the documented extra as well. The Beam provider README gives the concrete form of that pattern with the [google] extra, which pulls in the Google provider dependency needed for supported cross-provider features.
Sources: providers/apache/beam/src/airflow/providers/apache/beam/README.md
When writing custom integrations, prefer reusing hooks instead of embedding service-client logic directly inside operators. The core hooks README defines hooks as common interfaces and operator building blocks, which is the design contract that keeps provider implementations composable. A custom operator can focus on Airflow task semantics while delegating connection handling and external API calls to a hook. This is also why provider references list hooks beside operators: hooks are public integration surfaces for maintainers and advanced DAG authors, not just private implementation details.
Sources: airflow-core/src/airflow/hooks/README.md
Next Steps
Use the operators and hooks reference as a navigation index, then open the provider package page for the integration you intend to use. Confirm the PyPI distribution, optional extras, import namespace, and class names before committing a DAG. If you are evaluating Beam specifically, install apache-airflow-providers-apache-beam, add [google] when those cross-provider features are required, and choose between the Java and Python Beam pipeline operators based on the pipeline artifact you need to run. For deeper extension work, read the core extension and custom provider pages next.