feat(monitoring): add Grafana alert rules and fix datasource binding

- Why:
  - dashboard panels could resolve to stale datasource uid and show No data
  - monitoring required proactive alerts for ClickHouse health signals
- What:
  - pin dashboard panels to prometheus_uid and remove datasource templating variable
  - fix PromQL metrics for CPU, inserted rows, and parts panels
  - add provisioning alert rules for failed queries, memory resident, and active parts
  - pin Prometheus datasource uid and update monitoring documentation
- Check:
  - POST /api/admin/provisioning/datasources/reload
  - POST /api/admin/provisioning/dashboards/reload
  - POST /api/admin/provisioning/alerting/reload
  - GET /api/v1/provisioning/alert-rules
This commit is contained in:
2026-02-08 20:49:31 +03:00
parent 140b711233
commit 5270273d8a
6 changed files with 326 additions and 43 deletions
+23 -1
View File
@@ -109,6 +109,7 @@ curl -s http://localhost:9090/api/v1/targets | grep -o '"health":"[^"]*"'
- **Grafana provisioning** (`configs/grafana/provisioning/`):
- Datasource Prometheus автоматически настроен
- Dashboard "ClickHouse Overview" загружается при старте
- Alert rules для ClickHouse загружаются при старте
### Дашборд ClickHouse Overview
@@ -118,7 +119,9 @@ URL: `http://localhost:3000/d/clickhouse-overview/clickhouse-overview`
|--------|---------|
| System Health | CPU Usage, Memory Resident, Memory Code |
| Query Performance | Queries/sec, Active Queries, Failed Queries, Total Queries, Inserted Rows/sec |
| MergeTree Storage | Total Parts, Parts by Table, Total Merges, Merges/sec |
| MergeTree Storage | Total Parts, Parts by State, Total Merges, Merges/sec |
Принятое решение по метрикам: сверили naming через Context7 (`/clickhouse/clickhouse-docs`, раздел Prometheus interface) и заменили недоступные в `25.1` серии на фактически экспортируемые (`ClickHouseProfileEvents_InsertedRows`, `ClickHouseAsyncMetrics_TotalPartsOfMergeTreeTables`, `ClickHouseMetrics_Parts*`).
### Проверка метрик
@@ -130,6 +133,25 @@ curl -s "http://localhost:9090/api/v1/query?query=ClickHouseAsyncMetrics_MemoryR
curl -s "http://localhost:9090/api/v1/query?query=ClickHouseProfileEvents_Query"
```
### Алерты Grafana
Provisioning-файл: `configs/grafana/provisioning/alerting/clickhouse-alert-rules.yml`
Настроены правила:
- `ClickHouse Failed Queries Rate``rate(ClickHouseProfileEvents_FailedQuery[5m]) > 0` в течение `2m`
- `ClickHouse Memory Resident High``MemoryResident / OSMemoryTotal * 100 > 85` в течение `5m`
- `ClickHouse Parts Active High``ClickHouseMetrics_PartsActive > 500` в течение `10m`
Проверка и reload без рестарта контейнера:
```bash
# Список правил unified alerting
curl -s -u admin:admin http://localhost:3000/api/v1/provisioning/alert-rules
# Принудительно перечитать provisioning alerting
curl -s -X POST -u admin:admin http://localhost:3000/api/admin/provisioning/alerting/reload
```
### Troubleshooting мониторинга
- **"No data" в Grafana**: проверить, что Prometheus видит target (`Status -> Targets` в UI)
+1
View File
@@ -32,6 +32,7 @@
- `data/*.jsonl` — исходные данные (могут быть грязными)
- `configs/` — конфиги ClickHouse, Prometheus, Grafana
- `configs/grafana/provisioning/alerting/clickhouse-alert-rules.yml` — правила алертинга Grafana для ClickHouse
## Документация