Secrets Backends and Masking

Purpose and Scope

Airflow uses sensitive values in three recurring places: connections, variables, and configuration. A secrets backend is the extension point that lets an Airflow deployment read those values from an external secret manager instead of storing every value in the Airflow metadata database. The official documentation frames this as a read-oriented integration: Airflow can tap into existing enterprise secret stores through provider implementations, while writing secrets back to those stores remains outside the backend contract. That separation is important operationally because write permissions usually require a different trust boundary from read permissions.

Secret masking solves a different but related problem. Even when credentials are read securely, they can be exposed accidentally through rendered templates, task logs, UI fields, exception messages, or operator debug output. Airflow’s user-facing security documentation therefore treats masking as a runtime safety feature that complements secret storage. A well-designed deployment uses both: a backend to reduce where secrets are stored, and masking rules to reduce where secret values can be observed after they enter an Airflow process.

Sources: providers/akeyless/src/airflow/providers/akeyless/secrets/akeyless.py, providers/amazon/src/airflow/providers/amazon/aws/hooks/secrets_manager.py

Relevant Source Files

• docs/images/documentation_architecture.py - Generates the documentation architecture diagram that shows how package documentation is published from the apache-airflow repository into the live documentation site.
• devel-common/src/sphinx_exts/providers_extensions.py - Provides custom Sphinx extension support used by provider documentation, including the provider extension reference pages where community secret backend implementations are surfaced.
• providers/akeyless/src/airflow/providers/akeyless/secrets/__init__.py - Defines the Akeyless provider secrets package namespace.
• providers/akeyless/src/airflow/providers/akeyless/secrets/akeyless.py - Implements AkeylessBackend, a provider secrets backend for Airflow connections, variables, and configuration.
• providers/amazon/src/airflow/providers/amazon/aws/hooks/secrets_manager.py - Implements SecretsManagerHook, the AWS hook wrapper around the Secrets Manager client used to retrieve string, binary, or JSON secrets.

Core Concepts

A secret backend in Airflow is a Python class that implements the BaseSecretsBackend contract and is configured under the [secrets] section of airflow.cfg. The Akeyless provider file states its purpose directly: it is a “Secrets Backend for sourcing Connections, Variables, and Config from Akeyless,” and the class docstring says it retrieves connections, variables, and configuration. It also documents the canonical configuration shape, with backend pointing to a fully qualified backend class and backend_kwargs carrying backend-specific connection and path settings.

Sources: providers/akeyless/src/airflow/providers/akeyless/secrets/akeyless.py

The search model is path-oriented. In the Akeyless backend, secrets are looked up by joining a configured base path with the requested key. Separate base paths can be configured for connections, variables, and configuration, and setting one of those paths to None disables that category for the backend. This is a practical deployment pattern: teams can route Airflow connection URIs to one external tree, variable values to another, and configuration secrets to a third, while leaving categories disabled when another system remains authoritative.

Sources: providers/akeyless/src/airflow/providers/akeyless/secrets/akeyless.py

Airflow’s official secrets documentation also describes a search path and configuration flow. In normal operation, code that asks for a connection, variable, or config value does not need to know whether the value came from the metadata database, a local filesystem backend, Akeyless, AWS Secrets Manager, or another provider. The deployment chooses that resolution behavior centrally. This lets DAG authors use Airflow primitives while platform operators decide where credentials live, how they are rotated, and which workers are allowed to retrieve them.

Provider Backend Example: Akeyless

AkeylessBackend is a concrete example of a provider-supplied backend. It subclasses BaseSecretsBackend and LoggingMixin, imports Airflow configuration through the compatibility SDK module, and depends on the external akeyless client library. Its constructor exposes category paths, path separator behavior, multi-team lookup behavior, Akeyless API endpoint configuration, credential fields, authentication type, and token caching. The supported backend authentication types are explicitly constrained to api_key and uid, while the docstring directs users who need cloud-based authentication such as AWS IAM, GCP, or Azure AD to use AkeylessHook directly.

Sources: providers/akeyless/src/airflow/providers/akeyless/secrets/akeyless.py

The multi-team behavior is especially important for larger Airflow deployments. When core.multi_team = True, the backend first searches for a secret under a team-specific path, then falls back to a global path. The global fallback can include global_secrets_path, or it can use the base path directly when no global segment is configured. This allows platform teams to give a team its own namespace without forcing every shared credential to be duplicated. It also means secret naming conventions should be designed deliberately before teams begin depending on them.

Sources: providers/akeyless/src/airflow/providers/akeyless/secrets/akeyless.py

A minimal Akeyless backend configuration follows the pattern documented in the provider source. In production, prefer injecting the backend_kwargs through your deployment system and avoid committing access keys to source control.

[secrets]
backend = airflow.providers.akeyless.secrets.akeyless.AkeylessBackend
backend_kwargs = {
    "connections_path": "/airflow/connections",
    "variables_path": "/airflow/variables",
    "api_url": "https://api.akeyless.io",
    "access_id": "p-xxxx",
    "access_key": "xxxx"
}

AWS Secrets Manager Hook Surface

The supplied AWS source path documents SecretsManagerHook, which is a hook rather than a backend. It subclasses AwsBaseHook and initializes the underlying AWS client with client_type="secretsmanager". The class docstring describes it as a thin wrapper around boto3.client("secretsmanager"), with additional arguments such as aws_conn_id passed through the AWS base hook. This is the low-level integration shape Airflow providers use when interacting with AWS Secrets Manager from hooks, operators, sensors, or higher-level backend code.

Sources: providers/amazon/src/airflow/providers/amazon/aws/hooks/secrets_manager.py

The hook exposes two retrieval methods. get_secret(secret_name: str) -> str | bytes calls get_secret_value and returns either SecretString or decoded SecretBinary, preserving whether the stored secret is textual or binary. get_secret_as_dict(secret_name: str) -> dict parses the result of get_secret with json.loads, making it convenient for JSON-shaped secrets such as dictionaries of connection fields. The distinction matters: binary secrets should not be assumed to be UTF-8, and JSON parsing should only be used when the stored value is intentionally serialized as JSON.

Sources: providers/amazon/src/airflow/providers/amazon/aws/hooks/secrets_manager.py

Masking Sensitive Values

Secret masking is not a storage backend; it is a display and logging protection. The official Airflow security documentation organizes masking around sensitive field names, adding custom masks, and a warning that masking does not apply in the same way when secrets are supplied through environment variables. The practical implication is that deployment security cannot rely on masking alone. A value may be hidden in Airflow-controlled rendering paths but still leak through a shell command, third-party library logging, environment inspection, or a task that prints raw process state.

When reviewing a DAG or provider integration, distinguish between the identifier of a secret and the secret value itself. A connection ID such as my_postgres can safely appear in logs because it is a lookup key. A password, token, private key, access key, or rendered connection URI must be treated as sensitive. Prefer passing identifiers to operators and hooks, let Airflow resolve the value at runtime, and avoid interpolating full credentials into command strings or templates. If a custom operator handles sensitive fields, document those fields and register appropriate masks where Airflow’s public masking hooks support it.

Masking also depends on timing. A backend lookup introduces the real value into a worker, scheduler, triggerer, API server, or other Airflow component process only when code requests it. Once the value has been materialized, ordinary Python code can accidentally propagate it. For example, serializing a full connection object for debug output, raising an exception containing request headers, or logging a provider client configuration can bypass the intention of centralized secret storage. Treat backend usage as a way to minimize persistence, not as a guarantee that runtime code cannot disclose secrets.

System-to-Code Mapping

Sources: docs/images/documentation_architecture.py, devel-common/src/sphinx_exts/providers_extensions.py, providers/akeyless/src/airflow/providers/akeyless/secrets/akeyless.py, providers/amazon/src/airflow/providers/amazon/aws/hooks/secrets_manager.py

Compact Reference

AkeylessBackend constructor options

• connections_path: str | None = "/airflow/connections" - Base path for connection secrets, or None to disable connection lookup.
• variables_path: str | None = "/airflow/variables" - Base path for variable secrets, or None to disable variable lookup.
• config_path: str | None = "/airflow/config" - Base path for configuration secrets, or None to disable config lookup.
• sep: str = "/" - Separator used when joining the base path and key.
• use_team_secrets_path: bool = True - Enables team-scoped lookup before global fallback in multi-team mode.
• global_secrets_path: str | None = None - Optional global fallback segment.
• api_url: str = "https://api.akeyless.io" - Akeyless API endpoint.
• access_id: str | None = None - Akeyless access ID.
• access_key: str | None = None - Access key used with api_key authentication.
• access_type: str = "api_key" - Backend-supported authentication type; supported values are api_key and uid.
• token_ttl - Token cache duration, documented with a default of 600 seconds.

SecretsManagerHook methods

• SecretsManagerHook(*args, **kwargs) - Initializes AwsBaseHook with client_type="secretsmanager".
• get_secret(secret_name: str) -> str | bytes - Returns SecretString when present, otherwise base64-decodes SecretBinary.
• get_secret_as_dict(secret_name: str) -> dict - Parses the retrieved secret value as JSON and returns a dictionary.

Sources: providers/akeyless/src/airflow/providers/akeyless/secrets/akeyless.py, providers/amazon/src/airflow/providers/amazon/aws/hooks/secrets_manager.py

Operational Guidance and Next Steps

For a new deployment, start by deciding which system owns each class of secret. If external systems own connections, variables, or configuration values, configure the Airflow [secrets] backend and limit database-stored fallbacks. Then define naming conventions for connection IDs, variable keys, team namespaces, and global fallbacks. Finally, test from the same Airflow component type that will perform the lookup, because scheduler, worker, triggerer, and API server environments may not share identical network access or credentials.

For existing deployments, review logs and DAG code before migrating secrets. Moving a password from the metadata database to Akeyless or AWS Secrets Manager improves storage posture, but it does not automatically remove hard-coded credentials from DAG files or stop tasks from printing values. Pair backend migration with masking review, operator logging review, and connection usage review. Related pages to read next are connections, security-model, providers-overview-and-installation, core-extension-contracts, and operators-and-hooks-reference.