- Why:
- formalize complete end-to-end verification for the demo DWH stack
- provide fast regression checks and full validation before demo/release
- What:
- add new TEST_PLAN.md with two execution contours: Smoke and Full
- include checks for infra bootstrap, Airflow DAG flow, STG/ODS/DDS/DM data quality, monitoring and alert provisioning
- add dedicated scenario proving dirty records are captured in ods.*_errors without breaking ETL
- Check:
- aligned steps with current DAG parameters/tasks and SQL transformation flow
- validated expected alert names against Grafana provisioning files
- Why:
- intensive development needs quick cluster stop/cleanup commands
- current Makefile had only up and pipeline/monitoring targets
- What:
- add make target down for standard docker compose shutdown
- add make target clean for full cleanup with volumes and orphans
- update OPERATIONS runbook with new make commands
- Check:
- make -n down clean
- Why:
- during intensive development monitoring can get stuck (No data, out of bounds)
- regular reload is not always enough to recover Prometheus + StatsD pipeline
- What:
- add make target recover-monitoring for hard recovery path
- recreate prometheus and statsd-exporter, restart airflow scheduler/webserver
- keep Grafana provisioning reload and target checks in one command
- document when to use recover-monitoring in OPERATIONS runbook
- Check:
- run make recover-monitoring
- verify Prometheus targets for airflow/clickhouse/kafka are up
- Add port 9126 mapping for ClickHouse Prometheus metrics endpoint
(was configured in prometheus_ch.xml but not exposed in docker-compose.yml)
- Fix CPU Usage panel: use delta() instead of rate() for gauge metric
ClickHouseProfileEvents_OSCPUVirtualTimeMicroseconds is a gauge, not counter
- Add explicit datasource blocks to dashboard queries for consistency
ClickHouse ProfileEvents metrics correctly use rate() — they are counters.
Warning about missing _total suffix is expected (ClickHouse naming convention).
- Why:
- Airflow task metrics were mapped to non-emitted StatsD keys
- reload-monitoring did not restart statsd-exporter after mapping changes
- What:
- update StatsD mapping for Airflow 2.10.5 metric names
- remove problematic catch-all mapping that produced inconsistent series
- restart statsd-exporter in reload-monitoring flow
- sync operations runbook and airflow monitoring plan with actual metrics
- Check:
- make reload-monitoring
- Prometheus targets: airflow/clickhouse/kafka are UP
- trigger ddl_init and verify airflow_task_duration_seconds_count
- verify airflow_task_success_total and airflow_task_failures_total in Prometheus
Fix Grafana warning about using rate() on gauge metric:
- ClickHouseProfileEvents_OSCPUVirtualTimeMicroseconds is a gauge, not counter
- rate() should only be used with counters; using delta() instead
- Add explicit datasource block for consistency
API verified via Context7:
- /prometheus/docs: rate() should never be used on gauges
Add missing entries for monitoring infrastructure:
- prometheus.yml, statsd_mapping.yml configs
- ClickHouse user configs (default_user.xml, prometheus_ch.xml)
- Grafana alerting rules for Kafka and Airflow
- Grafana dashboards for all services
- Monitoring plans (airflow, kafka)
This completes the documentation for the monitoring stack added
in the previous commits.
- Add statsd-exporter service to docker-compose.yml (prom/statsd-exporter:v0.27.1)
- Add StatsD env vars to airflow-default-env for metrics export
- Add airflow job to prometheus.yml scrape configs
- Add Airflow Overview dashboard (Grafana provisioning)
- Add Airflow alert rules: scheduler down, queue backlog, failures, parse time
- Add configs/statsd_mapping.yml for StatsD → Prometheus conversion
- Use Prometheus naming convention (_total for counters, _seconds for timers)
- Add monitoring plan at plans/monitoring_airflow_plan.md
- Update OPERATIONS.md and Makefile for airflow monitoring
Tested: all 3 jobs (airflow, clickhouse, kafka) showing UP in Prometheus,
metrics flowing (dagbag_size=3, executor slots, heartbeats with _total suffix),
all 4 alert rules loaded in Grafana
- Why:
- dashboard showed offset as throughput and produced misleading values
- kafka-exporter metric/label naming was inconsistent across alerts/docs
- consumer-group-missing alert was noisy for demo runs
- What:
- switch throughput panel to rate(kafka_topic_partition_current_offset[5m]) aggregated by topic and exclude __* topics
- align lag metric/labels to kafka_consumergroup_lag + consumergroup
- remove Kafka Consumer Group Missing alert from provisioning
- pin kafka-exporter image to v1.9.0 and update OPERATIONS.md checks
- Check:
- airflow dags list-import-errors -> No data found
- Prometheus targets: clickhouse up, kafka up
- PromQL kafka_consumergroup_lag returns series
- Grafana dashboards provisioning reload returns success
- Why:
- students hit permission denied after pull and grafana restart-loop with readonly db
- What:
- run grafana as default non-root user
- mount provisioning directory as read-only
- add troubleshooting for git permission issues and grafana volume reset
- normalize file modes for data jsonl and docs/DE-task.md to 100644
- Check:
- docker compose config
- docker compose up -d grafana
- curl -u admin:admin http://localhost:3000/api/health
- Add kafka-exporter service to docker-compose.yml
- Add kafka job to prometheus.yml scrape configs
- Add Kafka Overview dashboard (Grafana provisioning)
- Add Kafka alert rules (broker down, consumer lag, etc.)
- Add make reload-monitoring command for easy updates
- Update OPERATIONS.md with TL;DR and troubleshooting
API verified via Context7:
- /danielqsj/kafka_exporter for exporter config
- /prometheus/docs for scrape_configs format
- Why:
- student needs a simple way to apply Grafana/monitoring config updates after git pull
- What:
- add TL;DR block with minimal commands in monitoring section
- add detailed post-pull runbook for datasource/dashboard/alerting reload
- include clickhouse restart note for prometheus_ch.xml changes
- Check:
- reviewed commands and paths in docs/OPERATIONS.md
- Why:
- dashboard panels could resolve to stale datasource uid and show No data
- monitoring required proactive alerts for ClickHouse health signals
- What:
- pin dashboard panels to prometheus_uid and remove datasource templating variable
- fix PromQL metrics for CPU, inserted rows, and parts panels
- add provisioning alert rules for failed queries, memory resident, and active parts
- pin Prometheus datasource uid and update monitoring documentation
- Check:
- POST /api/admin/provisioning/datasources/reload
- POST /api/admin/provisioning/dashboards/reload
- POST /api/admin/provisioning/alerting/reload
- GET /api/v1/provisioning/alert-rules
- Why:
- commit messages with literal \n are hard to read in UI
- What:
- add explicit rule for multiline body formatting in CLI
- add correct examples with git commit -m and -F heredoc
- Check:
- reviewed new section in docs/COMMIT_RULES.md
- Why:\n - AGENTS.md became too large and mixed policy with operational details\n - context7 requirement was easy to miss in long text\n- What:\n - reduce AGENTS.md to a compact contributor contract\n - add explicit mandatory MCP Context7 workflow block\n - move runbook details to docs/OPERATIONS.md\n - move artifact map to docs/REPO_MAP.md\n- Check:\n - reviewed links and content after split\n - ensured only documentation files are included in commit
- Why:
- keep Airflow artifacts under a single airflow/ directory
- align repository layout with intended project structure
- What:
- move dags/ to airflow/dags/ and update compose mounts
- make SQL root resolution work in container and local runs
- update DAG path references in README, AGENTS, ARCHITECTURE, and plans
- remove tracked Python cache artifacts from old DAG location
- Check:
- airflow dags list
- airflow dags list-import-errors
- e2e success: ddl_init, kafka_load(limit=50), etl_pipeline
Слияние ветки с реализацией автоматизированной загрузки JSONL-файлов в Kafka
через Airflow DAG с валидацией, мониторингом и документацией.
- Что добавлено:
- dags/kafka_load_dag.py: TaskGroup-пайплайн загрузки 4 потоков данных
- dags/utils/kafka_helpers.py: хелперы для работы с Kafka (проверка,
создание топиков, загрузка с лимитом)
- airflow/requirements.txt: зависимость kafka-python==2.0.6
- .gitignore: полноценный шаблон для ETL-проекта
- Параметры DAG:
- limit: ограничение строк (0 = все)
- reset_topics: пересоздание топиков перед загрузкой
- load_browser/device/geo/location_events: выбор потоков
- Обновлена документация:
- README.md, AGENTS.md, docs/ARCHITECTURE.md
- plans/runbook.md, plans/airflow_dags_plan.md
- Why:\n - User-facing docs mixed Airflow and legacy CLI ingest paths and caused confusion\n- What:\n - Rework README quick start and status to use DAG chain ddl_init -> kafka_load -> etl_pipeline\n - Rewrite runbook as canonical Airflow-first execution flow\n - Sync architecture diagrams/sequence and DQ wording with current SQL and DAG behavior\n- Check:\n - Verified updated sections and removed stale markers with rg in README.md, docs/ARCHITECTURE.md, plans/runbook.md
- Why:
- For DE task we only need full ingest or limit-based sample.
- load_* and full_load params were redundant and unclear in current flow.
- What:
- Remove full_load and load_* params from kafka_load DAG contract.
- Simplify kafka helpers (validate/check files) to fixed 4-stream ingest.
- Sync AGENTS, README, runbook, architecture and airflow plan docs.
- Check:
- python3 -m py_compile dags/kafka_load_dag.py dags/utils/kafka_helpers.py
- Airflow smoke/full runs: ddl_init -> kafka_load -> etl_pipeline (all success).
- Legacy path: make data && make transform (success).
- Why:
- Align with Conventional Commits specification for consistency
- English is standard for open-source and team collaboration
- What:
- Change primary language to English (Russian still allowed)
- Add type and scope reference tables
- Add both English and Russian body templates
- Add good/bad examples section
- Add quick reference for common commit types
- Check:
- File renders correctly in markdown viewer
- Examples follow the new format rules
- Зачем:
- унифицировать стиль коммитов для всех участников проекта
- Что сделано:
- добавлен документ docs/COMMIT_RULES.md с форматом и примерами
- добавлена ссылка на правила в AGENTS.md
- Проверка:
- проверен staged diff перед коммитом
- Изменен путь volume с /tmp/kraft-combined-logs на /var/lib/kafka/data
- Решена проблема с правами доступа при старте Kafka в KRaft mode
- Kafka теперь корректно инициализирует метаданные при первом запуске
Add persistent volume for ClickHouse to preserve data across container
restarts. The volume `clickhouse-data` is mounted to `/var/lib/clickhouse`,
ensuring data remains when containers are recreated.
Add comprehensive DAG implementation for ClickHouse schema initialization
and ETL pipeline orchestration. The ddl_init_dag manages database schema
creation across stg/ods/dds/dm layers with verification capabilities. The
etl_pipeline_dag implements full ODS to DDS to DM transformation flow with
data quality checks, branching logic for full/incremental loads, and
timeout handling for data availability.
Additional changes:
- Upgrade Airflow from 2.9.3 to 2.10.5
- Fix ClickHouse connection to use native protocol port 9000
- Mount SQL directory in docker-compose for DAG execution
- Update project requirements and documentation comments
- Remove unused pandas dependency
Move DDL files from flat ddl/ directory to sql/ddl/ with layer-based
subdirectories (stg, ods, dds, dm). Move batch transformation SQL from
jobs/ to sql/ layer directories. Update scripts and documentation to
reflect new paths for improved organization and Airflow integration.
Split Kafka ingestion into a dedicated `kafka_load` DAG to enable independent
experimentation with data loading without triggering the full ETL pipeline.
Restructure implementation phases: Stage 1 uses `make data` for MVP, Stage 2
adds the standalone Kafka DAG, Stage 3 adds monitoring. Update DAG numbering,
parameters, task groups, and acceptance criteria to reflect the new
architecture.
Consolidate the orchestration strategy by merging `kafka_load` and
`etl_batch_transform` into a unified `etl_pipeline`. Replace BashOperator
dependencies on Kafka CLI with PythonOperators utilizing `kafka-python`.
Add detailed technical specifications for helper functions, MVP stages,
and validation checks to align with current infrastructure constraints.
Detail the architecture for migrating ETL orchestration from make to
Airflow. Define DAG structures for database initialization, Kafka data
ingestion, batch transformation, and quality monitoring. Include
technical specifications, operator details, and implementation phases.
Update Airflow configuration to integrate with ClickHouse DWH instead of
PostgreSQL training database. Changes include:
- Switch Airflow dependencies from PostgreSQL to ClickHouse connector
- Update docker-compose to use ClickHouse connection and correct Dockerfile
- Refactor airflow/requirements.txt to include only essential packages
- Add DAGs directory for ETL pipeline orchestration
- Update documentation to reflect Airflow integration and access credentials
- Adjust service dependencies to wait for ClickHouse startup
Add Apache Airflow infrastructure with webserver, scheduler, and metadata
database to enable DAG-based pipeline orchestration. Includes optimized
requirements file and Docker configuration for Airflow 2.9.3.
Refine data flow diagrams and documentation to clarify error handling
in the ODS layer and partial data processing in the DDS layer. Add
detailed explanations for materialized views, batch SQL transformations,
and data quality metrics. Split DDS entity assembly diagrams for better
readability of event and click processing pipelines.
Add materialized views to capture parsing errors from browser, location,
device, and geo raw staging tables and route them to dedicated error
tables in the ODS layer. Refactor DDS refresh logic to handle partial
data arrivals where device and geo events may arrive independently by
using a unified click_id source with LEFT JOINs. Add TRUNCATE command
to prevent duplicate data accumulation in DQ summary table.
ddadmin