- Why:
- dashboard showed offset as throughput and produced misleading values
- kafka-exporter metric/label naming was inconsistent across alerts/docs
- consumer-group-missing alert was noisy for demo runs
- What:
- switch throughput panel to rate(kafka_topic_partition_current_offset[5m]) aggregated by topic and exclude __* topics
- align lag metric/labels to kafka_consumergroup_lag + consumergroup
- remove Kafka Consumer Group Missing alert from provisioning
- pin kafka-exporter image to v1.9.0 and update OPERATIONS.md checks
- Check:
- airflow dags list-import-errors -> No data found
- Prometheus targets: clickhouse up, kafka up
- PromQL kafka_consumergroup_lag returns series
- Grafana dashboards provisioning reload returns success
- Why:
- students hit permission denied after pull and grafana restart-loop with readonly db
- What:
- run grafana as default non-root user
- mount provisioning directory as read-only
- add troubleshooting for git permission issues and grafana volume reset
- normalize file modes for data jsonl and docs/DE-task.md to 100644
- Check:
- docker compose config
- docker compose up -d grafana
- curl -u admin:admin http://localhost:3000/api/health
- Add kafka-exporter service to docker-compose.yml
- Add kafka job to prometheus.yml scrape configs
- Add Kafka Overview dashboard (Grafana provisioning)
- Add Kafka alert rules (broker down, consumer lag, etc.)
- Add make reload-monitoring command for easy updates
- Update OPERATIONS.md with TL;DR and troubleshooting
API verified via Context7:
- /danielqsj/kafka_exporter for exporter config
- /prometheus/docs for scrape_configs format
- Why:
- student needs a simple way to apply Grafana/monitoring config updates after git pull
- What:
- add TL;DR block with minimal commands in monitoring section
- add detailed post-pull runbook for datasource/dashboard/alerting reload
- include clickhouse restart note for prometheus_ch.xml changes
- Check:
- reviewed commands and paths in docs/OPERATIONS.md
- Why:
- dashboard panels could resolve to stale datasource uid and show No data
- monitoring required proactive alerts for ClickHouse health signals
- What:
- pin dashboard panels to prometheus_uid and remove datasource templating variable
- fix PromQL metrics for CPU, inserted rows, and parts panels
- add provisioning alert rules for failed queries, memory resident, and active parts
- pin Prometheus datasource uid and update monitoring documentation
- Check:
- POST /api/admin/provisioning/datasources/reload
- POST /api/admin/provisioning/dashboards/reload
- POST /api/admin/provisioning/alerting/reload
- GET /api/v1/provisioning/alert-rules
- Why:\n - AGENTS.md became too large and mixed policy with operational details\n - context7 requirement was easy to miss in long text\n- What:\n - reduce AGENTS.md to a compact contributor contract\n - add explicit mandatory MCP Context7 workflow block\n - move runbook details to docs/OPERATIONS.md\n - move artifact map to docs/REPO_MAP.md\n- Check:\n - reviewed links and content after split\n - ensured only documentation files are included in commit
ddadmin