ddmitry/clickstream-ch-kafka-superset-demo
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(airflow): implement dag orchestration for ddl and etl

Browse Source 
Add comprehensive DAG implementation for ClickHouse schema initialization
and ETL pipeline orchestration. The ddl_init_dag manages database schema
creation across stg/ods/dds/dm layers with verification capabilities. The
etl_pipeline_dag implements full ODS to DDS to DM transformation flow with
data quality checks, branching logic for full/incremental loads, and
timeout handling for data availability.

Additional changes:
- Upgrade Airflow from 2.9.3 to 2.10.5
- Fix ClickHouse connection to use native protocol port 9000
- Mount SQL directory in docker-compose for DAG execution
- Update project requirements and documentation comments
- Remove unused pandas dependency
This commit is contained in:
ddadmin
2026-02-07 21:52:31 +03:00
parent de9d0429a2
commit 14d16c5cae
7 changed files with 459 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Dockerfile.airflow
+1 -1
View File
@@ -1,5 +1,5 @@
# Use the official Airflow image as base
FROM apache/airflow:2.9.3
FROM apache/airflow:2.10.5
# Set environment variables
ENV AIRFLOW_HOME=/opt/airflow