- Add kafka-exporter service to docker-compose.yml
- Add kafka job to prometheus.yml scrape configs
- Add Kafka Overview dashboard (Grafana provisioning)
- Add Kafka alert rules (broker down, consumer lag, etc.)
- Add make reload-monitoring command for easy updates
- Update OPERATIONS.md with TL;DR and troubleshooting
API verified via Context7:
- /danielqsj/kafka_exporter for exporter config
- /prometheus/docs for scrape_configs format
- Why:
- keep Airflow artifacts under a single airflow/ directory
- align repository layout with intended project structure
- What:
- move dags/ to airflow/dags/ and update compose mounts
- make SQL root resolution work in container and local runs
- update DAG path references in README, AGENTS, ARCHITECTURE, and plans
- remove tracked Python cache artifacts from old DAG location
- Check:
- airflow dags list
- airflow dags list-import-errors
- e2e success: ddl_init, kafka_load(limit=50), etl_pipeline
- Изменен путь volume с /tmp/kraft-combined-logs на /var/lib/kafka/data
- Решена проблема с правами доступа при старте Kafka в KRaft mode
- Kafka теперь корректно инициализирует метаданные при первом запуске
Add persistent volume for ClickHouse to preserve data across container
restarts. The volume `clickhouse-data` is mounted to `/var/lib/clickhouse`,
ensuring data remains when containers are recreated.
Add comprehensive DAG implementation for ClickHouse schema initialization
and ETL pipeline orchestration. The ddl_init_dag manages database schema
creation across stg/ods/dds/dm layers with verification capabilities. The
etl_pipeline_dag implements full ODS to DDS to DM transformation flow with
data quality checks, branching logic for full/incremental loads, and
timeout handling for data availability.
Additional changes:
- Upgrade Airflow from 2.9.3 to 2.10.5
- Fix ClickHouse connection to use native protocol port 9000
- Mount SQL directory in docker-compose for DAG execution
- Update project requirements and documentation comments
- Remove unused pandas dependency
Update Airflow configuration to integrate with ClickHouse DWH instead of
PostgreSQL training database. Changes include:
- Switch Airflow dependencies from PostgreSQL to ClickHouse connector
- Update docker-compose to use ClickHouse connection and correct Dockerfile
- Refactor airflow/requirements.txt to include only essential packages
- Add DAGs directory for ETL pipeline orchestration
- Update documentation to reflect Airflow integration and access credentials
- Adjust service dependencies to wait for ClickHouse startup
Add Apache Airflow infrastructure with webserver, scheduler, and metadata
database to enable DAG-based pipeline orchestration. Includes optimized
requirements file and Docker configuration for Airflow 2.9.3.
Add Apache Superset for data visualization to the docker-compose setup.
The custom Dockerfile installs additional tools and the clickhouse-connect
driver. The service is configured with health check, persistent volumes,
and environment variables.
Fix network definition name from 'ch_replicated' to 'cs_dwh' to match
service references. Comment out hardcoded container names to allow
Docker to generate unique names automatically and avoid conflicts.
ddadmin