Free Hive

Released:

Prometheus Exporter for Airflow Metrics

Project description

The Airflow Prometheus Exporter exposes various metrics about the Scheduler, DAGs and Tasks which helps improve the observability of an Airflow cluster.

The exporter is based on this prometheus exporter for Airflow.

Requirements

I'm currently running an airflow (1.9.0) instance on python 3.6.5. I have a manual workflow that I'd like to move to a DAG. This manual workflow now requires code written in python 2 and 3. This video shows the usage of the F&P Airvo™ 2 Humidified High Flow System & Optiflow Nasal High Flow therapy in different departments of the Royal Berkshire Hospital in Reading, UK. It shows the benefits they have found both to the patients and hospital since its introduction. (3 minutes) Find out more about Optiflow therapy.

The plugin has been tested with:

• Airflow >= 1.10.4
• Python 3.6+

The scheduler metrics assume that there is a DAG named canary_dag. In our setup, the canary_dag is a DAG which has a tasks which perform very simple actions such as establishing database connections. This DAG is used to test the uptime of the Airflow scheduler itself.

Installation

The exporter can be installed as an Airflow Plugin using:

pip install airflow-prometheus-exporter

This should ideally be installed in your Airflow virtualenv.

Metrics

Metrics will be available at

http://<your_airflow_host_and_port>/admin/metrics/

Task Specific Metrics

airflow_task_status

Number of tasks with a specific status.

All the possible states are listed here.

airflow_task_duration

Duration of successful tasks in seconds.

airflow_task_fail_count

Number of times a particular task has failed.

airflow_xcom_param

value of configurable parameter in xcom table

xcom fields is deserialized as a dictionary and if key is found for a paticular task-id, the value is reported as a guage

Add task / key combinations in config.yaml:

a task_id of 'all' will match against all airflow tasks:

Dag Specific Metrics

airflow_dag_status

Number of DAGs with a specific status.

All the possible states are listed here

airflow_dag_run_duration

Duration of successful DagRun in seconds.

Scheduler Metrics

airflow_dag_scheduler_delay

Scheduling delay for a DAG Run in seconds. This metric assumes there is a canary_dag.

The scheduling delay is measured as the delay between when a DAG is marked as SCHEDULED and when it actually starts RUNNING.

airflow_task_scheduler_delay

Scheduling delay for a Task in seconds. This metric assumes there is a canary_dag.

airflow_num_queued_tasks

Number of tasks in the QUEUED state at any given instance.

Release historyRelease notifications RSS feed

1.0.8

1.0.7

1.0.6

1.0.5

1.0.4

1.0.3

1.0.2

1.0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Hashes for airflow_prometheus_exporter-1.0.8-py3-none-any.whl

Airflow 2 3 79

Hashes for airflow_prometheus_exporter-1.0.8.tar.gz

Airflow 2 3 7 Vacuum Hose