ddmitry/clickstream-ch-kafka-superset-demo
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(monitoring): add recover flow for stuck monitoring stack

Browse Source 
- Why:
  - during intensive development monitoring can get stuck (No data, out of bounds)
  - regular reload is not always enough to recover Prometheus + StatsD pipeline
- What:
  - add make target recover-monitoring for hard recovery path
  - recreate prometheus and statsd-exporter, restart airflow scheduler/webserver
  - keep Grafana provisioning reload and target checks in one command
  - document when to use recover-monitoring in OPERATIONS runbook
- Check:
  - run make recover-monitoring
  - verify Prometheus targets for airflow/clickhouse/kafka are up
This commit is contained in:
ddadmin
2026-02-09 09:28:13 +03:00
parent df173c00f7
commit e851ef9788
2 changed files with 24 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/OPERATIONS.md
+8 -2
View File
@@ -95,6 +95,8 @@ docker compose exec -T clickhouse clickhouse-client --user=default --password=12
```bash
# Быстрый вариант (make)
make reload-monitoring
# Если мониторинг "залип" (No data/out of bounds) — жесткое восстановление:
make recover-monitoring
# Или вручную:
docker compose up -d prometheus grafana kafka-exporter statsd-exporter
@@ -136,10 +138,13 @@ curl -s http://localhost:9090/api/v1/targets | grep -o '"health":"[^"]*"'
```bash
# Рекомендуемый способ (через make)
make reload-monitoring
# Если метрики пропали/залипли:
make recover-monitoring
# Или вручную:
docker compose up -d prometheus grafana kafka-exporter
docker compose restart prometheus
docker compose rm -sf prometheus statsd-exporter
docker compose up -d prometheus grafana kafka-exporter statsd-exporter
docker compose restart airflow-scheduler airflow-webserver
curl -s -u admin:admin -X POST http://localhost:3000/api/admin/provisioning/datasources/reload
curl -s -u admin:admin -X POST http://localhost:3000/api/admin/provisioning/dashboards/reload
curl -s -u admin:admin -X POST http://localhost:3000/api/admin/provisioning/alerting/reload
@@ -248,6 +253,7 @@ curl -s -X POST -u admin:admin http://localhost:3000/api/admin/provisioning/aler
**Общие проблемы:**
- **"No data" в Grafana**: проверить, что Prometheus видит target (`Status -> Targets` в UI)
- **Dashboard не загрузился**: проверить логи Grafana — provisioning работает при первом старте контейнера
- **Prometheus spam `out of bounds` и дашборды пустые**: выполнить `make recover-monitoring`
**ClickHouse:**
- **Метрики не обновляются**: ClickHouse экспортирует метрики на `0.0.0.0:9126` внутри сети Docker