ddmitry/clickstream-ch-kafka-superset-demo
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(airflow): implement dag orchestration for ddl and etl

Browse Source 
Add comprehensive DAG implementation for ClickHouse schema initialization
and ETL pipeline orchestration. The ddl_init_dag manages database schema
creation across stg/ods/dds/dm layers with verification capabilities. The
etl_pipeline_dag implements full ODS to DDS to DM transformation flow with
data quality checks, branching logic for full/incremental loads, and
timeout handling for data availability.

Additional changes:
- Upgrade Airflow from 2.9.3 to 2.10.5
- Fix ClickHouse connection to use native protocol port 9000
- Mount SQL directory in docker-compose for DAG execution
- Update project requirements and documentation comments
- Remove unused pandas dependency
This commit is contained in:
ddadmin
2026-02-07 21:52:31 +03:00
parent de9d0429a2
commit 14d16c5cae
7 changed files with 459 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
airflow/requirements.txt
+3 -6
View File
@@ -1,10 +1,7 @@
# Airflow requirements для ClickHouse DWH проекта
# Airflow requirements для учебного ETL-проекта
# Core database connector для metadata
# Metadata DB для Airflow
psycopg2-binary==2.9.9
# ClickHouse provider для ETL
# ClickHouse operator/hook для DAG'ов
airflow-clickhouse-plugin==1.6.0
# Для работы с данными
pandas==2.1.4