- Add port 9126 mapping for ClickHouse Prometheus metrics endpoint
(was configured in prometheus_ch.xml but not exposed in docker-compose.yml)
- Fix CPU Usage panel: use delta() instead of rate() for gauge metric
ClickHouseProfileEvents_OSCPUVirtualTimeMicroseconds is a gauge, not counter
- Add explicit datasource blocks to dashboard queries for consistency
ClickHouse ProfileEvents metrics correctly use rate() — they are counters.
Warning about missing _total suffix is expected (ClickHouse naming convention).
Fix Grafana warning about using rate() on gauge metric:
- ClickHouseProfileEvents_OSCPUVirtualTimeMicroseconds is a gauge, not counter
- rate() should only be used with counters; using delta() instead
- Add explicit datasource block for consistency
API verified via Context7:
- /prometheus/docs: rate() should never be used on gauges
- Why:
- dashboard panels could resolve to stale datasource uid and show No data
- monitoring required proactive alerts for ClickHouse health signals
- What:
- pin dashboard panels to prometheus_uid and remove datasource templating variable
- fix PromQL metrics for CPU, inserted rows, and parts panels
- add provisioning alert rules for failed queries, memory resident, and active parts
- pin Prometheus datasource uid and update monitoring documentation
- Check:
- POST /api/admin/provisioning/datasources/reload
- POST /api/admin/provisioning/dashboards/reload
- POST /api/admin/provisioning/alerting/reload
- GET /api/v1/provisioning/alert-rules
ddadmin