Metadata Database and Migrations

Purpose and Scope

Airflow stores scheduler state, DAG metadata, task instance history, connection definitions, variables, permissions, and many other runtime records in a metadata database. This page explains how to think about that database as an operational dependency rather than as an application API. The official documentation presents database setup, database migrations, and the ERD reference as administrative material: they help operators configure and upgrade Airflow safely, but they are not an invitation to read or write internal tables directly. In day-to-day use, DAG authors and automation should use Airflow’s public interfaces instead of coupling to table layouts.

The repository itself reflects that split between user-facing Airflow distributions and database-specific implementation details. The top-level README.md identifies Apache Airflow as the main distribution published through PyPI and container images, while provider packages can carry their own integration-specific resources. Two supplied provider snippets show migration directories: the Edge3 provider states that it has Alembic-based database migrations, and the FAB provider uses a generic single-database migration configuration. Those details matter because Airflow deployments often combine core Airflow with providers, and database change management must account for installed components, not just a single Python package.

Sources: README.md, providers/edge3/src/airflow/providers/edge3/migrations/README.md, providers/fab/src/airflow/providers/fab/migrations/README

Relevant Source Files

• README.md — establishes this repository as the Apache Airflow project distribution, including PyPI and container distribution signals that shape how operators install and upgrade running environments.
• providers/edge3/src/airflow/providers/edge3/migrations/README.md — documents that the Edge3 provider includes database migrations implemented with Alembic, showing that migration concerns can exist in provider packages as well as in core Airflow.
• providers/fab/src/airflow/providers/fab/migrations/README — identifies a provider migration area using a generic single-database configuration, which is relevant when reasoning about migration tooling and database backend assumptions.

System-to-Code Mapping

The metadata database sits below the scheduler, API server, web UI, workers, triggerer, and provider integrations. It is the shared persistence layer that lets Airflow coordinate work across processes and restarts. The source evidence available for this page is intentionally narrow, but it still shows an important architectural pattern: Airflow is distributed as a core project, and some providers include database migration assets of their own. That means operational documentation should treat migrations as part of the deployed Airflow environment, not only as a one-time initialization step for the core package.

The official database ERD reference describes the schema as an internal detail and warns users not to directly access the database to retrieve or modify information. That warning is central to this page. The ERD is useful when troubleshooting locks, row growth, retention issues, or upgrade planning, but application integrations should prefer the stable REST API, CLI, SDK-facing surfaces, or documented configuration mechanisms. Direct SQL against internal tables can break across Airflow versions because schema shape, relationships, and table meaning may change as migrations evolve.

Sources: README.md, providers/edge3/src/airflow/providers/edge3/migrations/README.md, providers/fab/src/airflow/providers/fab/migrations/README

Database Setup Responsibilities

Setting up the metadata database starts with choosing a supported backend and configuring Airflow to reach it through its database URI and related configuration. The official setup guide separates backend choice, database URI configuration, SQLite setup, PostgreSQL setup, MySQL setup, and operational monitoring. That sequence is practical: first decide which database technology is appropriate for the deployment, then configure connectivity, then initialize or migrate the schema, and finally monitor the database as a production dependency. Lightweight local environments can use simpler defaults, while production systems need a database that matches workload and availability requirements.

Operators should treat database setup as part of Airflow deployment design. The scheduler and API server rely on timely database access, and workers depend on persisted task state to coordinate execution. A slow, undersized, or poorly maintained metadata database can look like an Airflow scheduling problem even when the root cause is storage, indexing, connection management, or cleanup. The official setup material therefore includes monitoring and maintenance topics alongside initialization. In practice, database readiness should be checked before rolling out new schedulers, applying migrations, or increasing task concurrency.

A minimal operational flow usually looks like this:

# Configure Airflow to use the intended metadata database.
# Then initialize or migrate the schema for the installed Airflow version.
airflow db migrate

The exact surrounding commands and configuration depend on the deployment method, but airflow db migrate is the key documented operation for applying database migrations. Before running it in shared environments, confirm the Airflow version, installed providers, database backup state, and expected downtime or locking behavior for the target backend.

Migration Workflow and Upgrade Planning

Database migrations are versioned changes that transform the metadata schema and sometimes its data so the installed Airflow code can run correctly. The official migrations reference is designed for users who want to understand what migration steps will execute between versions and assess their impact. That is especially useful for DB-conscious operators who need to evaluate table rewrites, new indexes, constraints, or data updates before upgrading a large production deployment. The point is not to hand-edit migrations; it is to plan upgrades with enough context to avoid surprises.

Because provider packages can include migration directories, upgrade planning should include the full Python environment or container image contents. The Edge3 provider snippet explicitly says its database migrations use Alembic, while the FAB provider snippet identifies a generic single-database configuration. Those are small but meaningful signals: migration behavior can be packaged near the component that owns a feature, and migration execution must target the configured metadata database consistently. When validating an upgrade, compare the installed Airflow and provider versions in staging against production, not just the core package version.

Sources: providers/edge3/src/airflow/providers/edge3/migrations/README.md, providers/fab/src/airflow/providers/fab/migrations/README

A safe migration routine should be boring and repeatable. Take a database backup, stop or quiesce components as required by your deployment procedure, apply migrations once from a controlled environment, and then bring Airflow components back with the matching application version. For high-availability installations, avoid letting multiple containers or automation jobs race to run migrations. After the migration, verify scheduler startup, API availability, DAG parsing, and representative task state transitions. If the deployment uses custom providers or extensions, include them in staging because their database assumptions may not be visible from core Airflow documentation alone.

ERD Reference and Internal Database Boundaries

The ERD reference is a troubleshooting and learning tool. It gives operators a snapshot view of relationships in the metadata database for a specific Airflow release, which can help explain why cleanup jobs, migrations, or failed task state transitions touch certain tables. The official warning is important: the diagram is version-specific and internal. Treat it like an implementation map for incident response, not a contract for application development. If a workflow needs DAG run information, task status, variables, connections, or audit-style data, prefer documented public APIs and administrative commands.

This boundary protects both operators and developers. Airflow’s schema can evolve to support new scheduling concepts, UI features, security models, provider integrations, or performance improvements. If external systems depend on direct SQL queries, every upgrade becomes harder because the integration has to understand internal migration history. By contrast, code that uses public APIs can benefit from Airflow’s compatibility guarantees and clearer error behavior. The ERD remains valuable when diagnosing database bloat, unexpected foreign-key relationships, migration failures, or cleanup decisions, but it should not become a hidden integration layer.

Practical Next Steps

For a new deployment, start with the database backend setup guide before tuning schedulers or workers. Confirm the database backend, connection URI, credentials handling, and operational backup policy, then run the documented initialization or migration command for the installed version. For an upgrade, read the migration reference for the versions being crossed, rehearse the migration in staging with the same providers installed, and review the ERD only when you need to understand internal relationships or troubleshoot a database-specific issue.

Readers maintaining production Airflow should connect this page with deployment, security, and monitoring guidance. Database credentials belong in the same security model as other Airflow secrets, migration execution belongs in release automation, and database health belongs in operational monitoring. Continue with pages on production deployment, metrics and health checks, secrets backends and masking, and the REST API when you need stable ways to interact with Airflow metadata without depending on internal tables.