- Add port 9126 mapping for ClickHouse Prometheus metrics endpoint
(was configured in prometheus_ch.xml but not exposed in docker-compose.yml)
- Fix CPU Usage panel: use delta() instead of rate() for gauge metric
ClickHouseProfileEvents_OSCPUVirtualTimeMicroseconds is a gauge, not counter
- Add explicit datasource blocks to dashboard queries for consistency
ClickHouse ProfileEvents metrics correctly use rate() — they are counters.
Warning about missing _total suffix is expected (ClickHouse naming convention).
- Add statsd-exporter service to docker-compose.yml (prom/statsd-exporter:v0.27.1)
- Add StatsD env vars to airflow-default-env for metrics export
- Add airflow job to prometheus.yml scrape configs
- Add Airflow Overview dashboard (Grafana provisioning)
- Add Airflow alert rules: scheduler down, queue backlog, failures, parse time
- Add configs/statsd_mapping.yml for StatsD → Prometheus conversion
- Use Prometheus naming convention (_total for counters, _seconds for timers)
- Add monitoring plan at plans/monitoring_airflow_plan.md
- Update OPERATIONS.md and Makefile for airflow monitoring
Tested: all 3 jobs (airflow, clickhouse, kafka) showing UP in Prometheus,
metrics flowing (dagbag_size=3, executor slots, heartbeats with _total suffix),
all 4 alert rules loaded in Grafana
- Why:
- dashboard showed offset as throughput and produced misleading values
- kafka-exporter metric/label naming was inconsistent across alerts/docs
- consumer-group-missing alert was noisy for demo runs
- What:
- switch throughput panel to rate(kafka_topic_partition_current_offset[5m]) aggregated by topic and exclude __* topics
- align lag metric/labels to kafka_consumergroup_lag + consumergroup
- remove Kafka Consumer Group Missing alert from provisioning
- pin kafka-exporter image to v1.9.0 and update OPERATIONS.md checks
- Check:
- airflow dags list-import-errors -> No data found
- Prometheus targets: clickhouse up, kafka up
- PromQL kafka_consumergroup_lag returns series
- Grafana dashboards provisioning reload returns success
- Why:
- students hit permission denied after pull and grafana restart-loop with readonly db
- What:
- run grafana as default non-root user
- mount provisioning directory as read-only
- add troubleshooting for git permission issues and grafana volume reset
- normalize file modes for data jsonl and docs/DE-task.md to 100644
- Check:
- docker compose config
- docker compose up -d grafana
- curl -u admin:admin http://localhost:3000/api/health
- Add kafka-exporter service to docker-compose.yml
- Add kafka job to prometheus.yml scrape configs
- Add Kafka Overview dashboard (Grafana provisioning)
- Add Kafka alert rules (broker down, consumer lag, etc.)
- Add make reload-monitoring command for easy updates
- Update OPERATIONS.md with TL;DR and troubleshooting
API verified via Context7:
- /danielqsj/kafka_exporter for exporter config
- /prometheus/docs for scrape_configs format
- Why:
- keep Airflow artifacts under a single airflow/ directory
- align repository layout with intended project structure
- What:
- move dags/ to airflow/dags/ and update compose mounts
- make SQL root resolution work in container and local runs
- update DAG path references in README, AGENTS, ARCHITECTURE, and plans
- remove tracked Python cache artifacts from old DAG location
- Check:
- airflow dags list
- airflow dags list-import-errors
- e2e success: ddl_init, kafka_load(limit=50), etl_pipeline
- Изменен путь volume с /tmp/kraft-combined-logs на /var/lib/kafka/data
- Решена проблема с правами доступа при старте Kafka в KRaft mode
- Kafka теперь корректно инициализирует метаданные при первом запуске
Add persistent volume for ClickHouse to preserve data across container
restarts. The volume `clickhouse-data` is mounted to `/var/lib/clickhouse`,
ensuring data remains when containers are recreated.
Add comprehensive DAG implementation for ClickHouse schema initialization
and ETL pipeline orchestration. The ddl_init_dag manages database schema
creation across stg/ods/dds/dm layers with verification capabilities. The
etl_pipeline_dag implements full ODS to DDS to DM transformation flow with
data quality checks, branching logic for full/incremental loads, and
timeout handling for data availability.
Additional changes:
- Upgrade Airflow from 2.9.3 to 2.10.5
- Fix ClickHouse connection to use native protocol port 9000
- Mount SQL directory in docker-compose for DAG execution
- Update project requirements and documentation comments
- Remove unused pandas dependency
Update Airflow configuration to integrate with ClickHouse DWH instead of
PostgreSQL training database. Changes include:
- Switch Airflow dependencies from PostgreSQL to ClickHouse connector
- Update docker-compose to use ClickHouse connection and correct Dockerfile
- Refactor airflow/requirements.txt to include only essential packages
- Add DAGs directory for ETL pipeline orchestration
- Update documentation to reflect Airflow integration and access credentials
- Adjust service dependencies to wait for ClickHouse startup
Add Apache Airflow infrastructure with webserver, scheduler, and metadata
database to enable DAG-based pipeline orchestration. Includes optimized
requirements file and Docker configuration for Airflow 2.9.3.
Add Apache Superset for data visualization to the docker-compose setup.
The custom Dockerfile installs additional tools and the clickhouse-connect
driver. The service is configured with health check, persistent volumes,
and environment variables.
Fix network definition name from 'ch_replicated' to 'cs_dwh' to match
service references. Comment out hardcoded container names to allow
Docker to generate unique names automatically and avoid conflicts.
ddadmin