fix(monitoring): correct Kafka metrics and alert rules

- Why:
  - dashboard showed offset as throughput and produced misleading values
  - kafka-exporter metric/label naming was inconsistent across alerts/docs
  - consumer-group-missing alert was noisy for demo runs
- What:
  - switch throughput panel to rate(kafka_topic_partition_current_offset[5m]) aggregated by topic and exclude __* topics
  - align lag metric/labels to kafka_consumergroup_lag + consumergroup
  - remove Kafka Consumer Group Missing alert from provisioning
  - pin kafka-exporter image to v1.9.0 and update OPERATIONS.md checks
- Check:
  - airflow dags list-import-errors -> No data found
  - Prometheus targets: clickhouse up, kafka up
  - PromQL kafka_consumergroup_lag returns series
  - Grafana dashboards provisioning reload returns success
This commit is contained in:
2026-02-08 21:52:21 +03:00
parent afdfc98eef
commit 8e31b06241
4 changed files with 11 additions and 42 deletions
+3 -3
View File
@@ -200,7 +200,7 @@ curl -s "http://localhost:9090/api/v1/query?query=ClickHouseProfileEvents_Query"
# Prometheus собирает метрики Kafka
curl -s "http://localhost:9090/api/v1/query?query=kafka_brokers"
curl -s "http://localhost:9090/api/v1/query?query=kafka_consumer_group_lag"
curl -s "http://localhost:9090/api/v1/query?query=kafka_consumergroup_lag"
# Прямая проверка kafka-exporter
curl -s http://localhost:9308/metrics | grep "^kafka_"
@@ -227,9 +227,9 @@ curl -s "http://localhost:9090/api/v1/query?query=ClickHouseProfileEvents_Query"
Настроены правила:
- `Kafka Broker Down``kafka_brokers < 1` в течение `1m`
- `Kafka Consumer Lag High``kafka_consumer_group_lag > 10000` в течение `5m`
- `Kafka Consumer Lag High``kafka_consumergroup_lag > 10000` в течение `5m`
- `Kafka No Messages Produced``rate(kafka_topic_partition_current_offset[5m]) < 0.1` в течение `10m`
- `Kafka Consumer Group Missing` — пропала консьюмер-группа в течение `5m`
- `Kafka Consumer Group Missing` не включён: для демо-стенда даёт шум на стартовых прогонах и не повышает диагностику по сравнению с lag/throughput.
Проверка и reload без рестарта контейнера: