Apache Airflow

Apache Airflow schedules batch work, which makes it the same shape as every other integration runtime this product cares about: one scheduler running many DAGs, where the DAG is what somebody wants alerted on and the scheduler is merely where it happens to run.

Key: airflow · applied automatically to services emitting airflow.dagrun., airflow.executor., airflow.pool. or airflow.scheduler-prefixed metrics.

It detects the scheduler, not “Airflow”

Airflow is four processes, and they emit almost disjoint sets of metrics:

Process	Emits
scheduler	airflow.dagrun.*, airflow.executor.*, airflow.pool.*, airflow.scheduler*, airflow.ti*
dag-processor	airflow.dag_processing.*, airflow.dagbag_size, airflow.dag_processor_heartbeat
triggerer	airflow.triggers.*, airflow.triggerer*
apiserver	almost nothing

So detection names the scheduler’s own families rather than the bare airflow. prefix. Matching that would offer this type to the triggerer and the dag-processor too, whose metrics none of these checks mention, and every applied check would sit in the health list looking armed while it could never fire.

The other two processes worth watching have their own types, for the same reason: the Airflow DAG processor - where an import error stops a DAG from running at all, with no failure anywhere to show for it - and the Airflow triggerer, where a dead process leaves every deferred task waiting in a way that looks exactly like waiting correctly. Apply each to its own service.

Getting the telemetry

Airflow emits OTLP metrics natively. In airflow.cfg, or the matching environment variables:

airflow.cfg

[metrics]
otel_on = True
otel_host = your-collector
otel_port = 4318
otel_interval_milliseconds = 30000

Point it at a collector that exports to your cell, and give each Airflow process its own service.name (airflow-scheduler, airflow-triggerer, …) so the scheduler is a service in its own right. Traces come separately, from Airflow’s OpenTelemetry traces support, and are not required by any check here.

Alerting per DAG

The checks below split by dag_id, so a firing alert names the DAG rather than the scheduler. That split does more than read better:

A check split by an attribute is understood to describe one flow, not the process. So a failing DAG no longer drags every other DAG on the scheduler down with it - which is how it used to read, because the check is bound to the scheduler and the scheduler is shared. The two process-level checks here are deliberately left unsplit for the same reason, so that they do reach every DAG: a dead scheduler and a starved pool break all of them at once.

For alerting on one DAG alone, make it an integration - match service.name = airflow-scheduler and dag_id = <your dag> - and bind checks to that. Its health, messages and errors are then that DAG’s, and nothing about a sibling reaches it.

A note on metric names

Airflow still emits its StatsD-era names alongside the attributed ones, with the DAG baked into the name:

airflow.dagrun.orders_export.first_task_scheduling_delay   legacy, one metric per DAG
airflow.dagrun.first_task_scheduling_delay  {dag_id=…}      attributed

No template can enumerate your DAG names, so the checks use the attributed form. If you are on a configuration that only emits the legacy names, the per-DAG checks will not fire and the metric names need replacing per DAG by hand.

Starter checks

Check	Condition	Severity	Scope
DAG task failures	airflow.ti_failures rising, split by dag_id	Warning	per DAG
Operator failures	airflow.operator_failures rising, split by dag_id	Warning	per DAG
Scheduler not running	airflow.scheduler_heartbeat stops arriving	Critical	whole scheduler
Tasks starving for a pool slot	airflow.pool.starving_tasks > 0	Warning	whole scheduler
Executor out of slots	airflow.executor.open_slots < 1	Warning	whole scheduler
Tasks queued and not starting	airflow.executor.queued_tasks > 50	Warning	whole scheduler

The heartbeat fires on absence

“Scheduler not running” is the one check here that cannot be written as an ordinary threshold. A scheduler that dies stops emitting, so there is no value left to breach and every other check goes quiet at exactly the moment something is most wrong. This check fires on the absence of airflow.scheduler_heartbeat instead, guarded so that it only ever fires for a metric that has reported before - a cell that never saw a heartbeat does not get paged for one.

Tuning notes

• Task failures at “any failure” is deliberately loud. A DAG with a retrying sensor will fire it. Raise the threshold per DAG once you know which of yours fail routinely, or scope it to an integration and tune there.
• Executor out of slots is a floor, not a warning. At < 1 it fires when saturation has already happened. Set it to a fraction of your parallelism to hear about it on the way.
• Queued tasks depends entirely on your executor. 50 is a stuck LocalExecutor and an ordinary minute on a large Celery pool.
• Pool starvation is the check worth keeping even when it is quiet. Tasks waiting for a slot neither fail nor run, so nothing else in Airflow reports them: a DAG simply appears slow, for hours.
• No check on DAG run duration. airflow.dagrun.duration.failed and .success are histograms, which arrive here with the value being the bucket sum - the total seconds of failed runs, which is neither a duration nor a count. For “did this DAG take too long”, use a multi-stage completion check on the DAG’s integration, which measures the run itself.