feat(monitoring): add Airflow monitoring via statsd-exporter
- Add statsd-exporter service to docker-compose.yml (prom/statsd-exporter:v0.27.1) - Add StatsD env vars to airflow-default-env for metrics export - Add airflow job to prometheus.yml scrape configs - Add Airflow Overview dashboard (Grafana provisioning) - Add Airflow alert rules: scheduler down, queue backlog, failures, parse time - Add configs/statsd_mapping.yml for StatsD → Prometheus conversion - Use Prometheus naming convention (_total for counters, _seconds for timers) - Add monitoring plan at plans/monitoring_airflow_plan.md - Update OPERATIONS.md and Makefile for airflow monitoring Tested: all 3 jobs (airflow, clickhouse, kafka) showing UP in Prometheus, metrics flowing (dagbag_size=3, executor slots, heartbeats with _total suffix), all 4 alert rules loaded in Grafana
This commit is contained in:
@@ -18,4 +18,13 @@ scrape_configs:
|
||||
- targets: ["kafka-exporter:9308"]
|
||||
labels:
|
||||
instance: Kafka-1
|
||||
honor_labels: true
|
||||
|
||||
# Airflow metrics via statsd-exporter (StatsD → Prometheus)
|
||||
- job_name: "airflow"
|
||||
metrics_path: "/metrics"
|
||||
static_configs:
|
||||
- targets: ["statsd-exporter:9102"]
|
||||
labels:
|
||||
instance: Airflow-1
|
||||
honor_labels: true
|
||||
Reference in New Issue
Block a user