ddmitry/clickstream-ch-kafka-superset-demo
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(monitoring): add recover flow for stuck monitoring stack

Browse Source 
- Why:
  - during intensive development monitoring can get stuck (No data, out of bounds)
  - regular reload is not always enough to recover Prometheus + StatsD pipeline
- What:
  - add make target recover-monitoring for hard recovery path
  - recreate prometheus and statsd-exporter, restart airflow scheduler/webserver
  - keep Grafana provisioning reload and target checks in one command
  - document when to use recover-monitoring in OPERATIONS runbook
- Check:
  - run make recover-monitoring
  - verify Prometheus targets for airflow/clickhouse/kafka are up
This commit is contained in:
ddadmin
2026-02-09 09:28:13 +03:00
parent df173c00f7
commit e851ef9788
2 changed files with 24 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Makefile
+16 -1
View File
@@ -1,4 +1,4 @@
.PHONY: up ddl data transform reload-monitoring
.PHONY: up ddl data transform reload-monitoring recover-monitoring
COMPOSE ?= docker compose
@@ -26,3 +26,18 @@ reload-monitoring:
@curl -s -u admin:admin -X POST http://localhost:3000/api/admin/provisioning/alerting/reload && echo " [alerting]"
@echo "=== Проверка ==="
@curl -s http://localhost:9090/api/v1/targets | grep -o '"job":"[^"]*"' | sort | uniq
# Восстановление мониторинга после сбоев (например, out of bounds / пустые дашборды)
recover-monitoring:
@echo "=== Восстановление мониторинга (жесткий режим) ==="
$(COMPOSE) rm -sf prometheus statsd-exporter
$(COMPOSE) up -d prometheus grafana kafka-exporter statsd-exporter
$(COMPOSE) restart airflow-scheduler airflow-webserver
@echo "=== Перезагрузка provisioning Grafana ==="
@sleep 2
@curl -s -u admin:admin -X POST http://localhost:3000/api/admin/provisioning/datasources/reload && echo " [datasources]"
@curl -s -u admin:admin -X POST http://localhost:3000/api/admin/provisioning/dashboards/reload && echo " [dashboards]"
@curl -s -u admin:admin -X POST http://localhost:3000/api/admin/provisioning/alerting/reload && echo " [alerting]"
@echo "=== Проверка targets ==="
@curl -s http://localhost:9090/api/v1/targets | grep -o '"job":"[^"]*"' | sort | uniq
@curl -s http://localhost:9090/api/v1/targets | grep -o '"health":"[^"]*"' | sort | uniq -c