Add persistent volume for ClickHouse to preserve data across container
restarts. The volume `clickhouse-data` is mounted to `/var/lib/clickhouse`,
ensuring data remains when containers are recreated.
Add comprehensive DAG implementation for ClickHouse schema initialization
and ETL pipeline orchestration. The ddl_init_dag manages database schema
creation across stg/ods/dds/dm layers with verification capabilities. The
etl_pipeline_dag implements full ODS to DDS to DM transformation flow with
data quality checks, branching logic for full/incremental loads, and
timeout handling for data availability.
Additional changes:
- Upgrade Airflow from 2.9.3 to 2.10.5
- Fix ClickHouse connection to use native protocol port 9000
- Mount SQL directory in docker-compose for DAG execution
- Update project requirements and documentation comments
- Remove unused pandas dependency
Update Airflow configuration to integrate with ClickHouse DWH instead of
PostgreSQL training database. Changes include:
- Switch Airflow dependencies from PostgreSQL to ClickHouse connector
- Update docker-compose to use ClickHouse connection and correct Dockerfile
- Refactor airflow/requirements.txt to include only essential packages
- Add DAGs directory for ETL pipeline orchestration
- Update documentation to reflect Airflow integration and access credentials
- Adjust service dependencies to wait for ClickHouse startup
Add Apache Airflow infrastructure with webserver, scheduler, and metadata
database to enable DAG-based pipeline orchestration. Includes optimized
requirements file and Docker configuration for Airflow 2.9.3.
Add Apache Superset for data visualization to the docker-compose setup.
The custom Dockerfile installs additional tools and the clickhouse-connect
driver. The service is configured with health check, persistent volumes,
and environment variables.
Fix network definition name from 'ch_replicated' to 'cs_dwh' to match
service references. Comment out hardcoded container names to allow
Docker to generate unique names automatically and avoid conflicts.
ddadmin