feat(monitoring): add Grafana alert rules and fix datasource binding
- Why: - dashboard panels could resolve to stale datasource uid and show No data - monitoring required proactive alerts for ClickHouse health signals - What: - pin dashboard panels to prometheus_uid and remove datasource templating variable - fix PromQL metrics for CPU, inserted rows, and parts panels - add provisioning alert rules for failed queries, memory resident, and active parts - pin Prometheus datasource uid and update monitoring documentation - Check: - POST /api/admin/provisioning/datasources/reload - POST /api/admin/provisioning/dashboards/reload - POST /api/admin/provisioning/alerting/reload - GET /api/v1/provisioning/alert-rules
This commit is contained in:
+23
-1
@@ -109,6 +109,7 @@ curl -s http://localhost:9090/api/v1/targets | grep -o '"health":"[^"]*"'
|
||||
- **Grafana provisioning** (`configs/grafana/provisioning/`):
|
||||
- Datasource Prometheus автоматически настроен
|
||||
- Dashboard "ClickHouse Overview" загружается при старте
|
||||
- Alert rules для ClickHouse загружаются при старте
|
||||
|
||||
### Дашборд ClickHouse Overview
|
||||
|
||||
@@ -118,7 +119,9 @@ URL: `http://localhost:3000/d/clickhouse-overview/clickhouse-overview`
|
||||
|--------|---------|
|
||||
| System Health | CPU Usage, Memory Resident, Memory Code |
|
||||
| Query Performance | Queries/sec, Active Queries, Failed Queries, Total Queries, Inserted Rows/sec |
|
||||
| MergeTree Storage | Total Parts, Parts by Table, Total Merges, Merges/sec |
|
||||
| MergeTree Storage | Total Parts, Parts by State, Total Merges, Merges/sec |
|
||||
|
||||
Принятое решение по метрикам: сверили naming через Context7 (`/clickhouse/clickhouse-docs`, раздел Prometheus interface) и заменили недоступные в `25.1` серии на фактически экспортируемые (`ClickHouseProfileEvents_InsertedRows`, `ClickHouseAsyncMetrics_TotalPartsOfMergeTreeTables`, `ClickHouseMetrics_Parts*`).
|
||||
|
||||
### Проверка метрик
|
||||
|
||||
@@ -130,6 +133,25 @@ curl -s "http://localhost:9090/api/v1/query?query=ClickHouseAsyncMetrics_MemoryR
|
||||
curl -s "http://localhost:9090/api/v1/query?query=ClickHouseProfileEvents_Query"
|
||||
```
|
||||
|
||||
### Алерты Grafana
|
||||
|
||||
Provisioning-файл: `configs/grafana/provisioning/alerting/clickhouse-alert-rules.yml`
|
||||
|
||||
Настроены правила:
|
||||
- `ClickHouse Failed Queries Rate` — `rate(ClickHouseProfileEvents_FailedQuery[5m]) > 0` в течение `2m`
|
||||
- `ClickHouse Memory Resident High` — `MemoryResident / OSMemoryTotal * 100 > 85` в течение `5m`
|
||||
- `ClickHouse Parts Active High` — `ClickHouseMetrics_PartsActive > 500` в течение `10m`
|
||||
|
||||
Проверка и reload без рестарта контейнера:
|
||||
|
||||
```bash
|
||||
# Список правил unified alerting
|
||||
curl -s -u admin:admin http://localhost:3000/api/v1/provisioning/alert-rules
|
||||
|
||||
# Принудительно перечитать provisioning alerting
|
||||
curl -s -X POST -u admin:admin http://localhost:3000/api/admin/provisioning/alerting/reload
|
||||
```
|
||||
|
||||
### Troubleshooting мониторинга
|
||||
|
||||
- **"No data" в Grafana**: проверить, что Prometheus видит target (`Status -> Targets` в UI)
|
||||
|
||||
@@ -32,6 +32,7 @@
|
||||
|
||||
- `data/*.jsonl` — исходные данные (могут быть грязными)
|
||||
- `configs/` — конфиги ClickHouse, Prometheus, Grafana
|
||||
- `configs/grafana/provisioning/alerting/clickhouse-alert-rules.yml` — правила алертинга Grafana для ClickHouse
|
||||
|
||||
## Документация
|
||||
|
||||
|
||||
Reference in New Issue
Block a user