ddmitry/clickstream-ch-kafka-superset-demo
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(monitoring): add Airflow monitoring via statsd-exporter

Browse Source 
- Add statsd-exporter service to docker-compose.yml (prom/statsd-exporter:v0.27.1)
- Add StatsD env vars to airflow-default-env for metrics export
- Add airflow job to prometheus.yml scrape configs
- Add Airflow Overview dashboard (Grafana provisioning)
- Add Airflow alert rules: scheduler down, queue backlog, failures, parse time
- Add configs/statsd_mapping.yml for StatsD → Prometheus conversion
- Use Prometheus naming convention (_total for counters, _seconds for timers)
- Add monitoring plan at plans/monitoring_airflow_plan.md
- Update OPERATIONS.md and Makefile for airflow monitoring

Tested: all 3 jobs (airflow, clickhouse, kafka) showing UP in Prometheus,
metrics flowing (dagbag_size=3, executor slots, heartbeats with _total suffix),
all 4 alert rules loaded in Grafana
This commit is contained in:
ddadmin
2026-02-08 22:27:32 +03:00
parent 8e31b06241
commit 4917a9a6ad
8 changed files with 1701 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Makefile
+1 -1
View File
@@ -17,7 +17,7 @@ transform:
# Перезагрузка конфигурации мониторинга (после изменений в provisioning)
reload-monitoring:
@echo "=== Перезагрузка сервисов мониторинга ==="
$(COMPOSE) up -d prometheus grafana kafka-exporter
$(COMPOSE) up -d prometheus grafana kafka-exporter statsd-exporter
$(COMPOSE) restart prometheus
@echo "=== Перезагрузка provisioning Grafana ==="
@sleep 2