feat(airflow): implement dag orchestration for ddl and etl

Browse Source

Add comprehensive DAG implementation for ClickHouse schema initialization
and ETL pipeline orchestration. The ddl_init_dag manages database schema
creation across stg/ods/dds/dm layers with verification capabilities. The
etl_pipeline_dag implements full ODS to DDS to DM transformation flow with
data quality checks, branching logic for full/incremental loads, and
timeout handling for data availability.

Additional changes:
- Upgrade Airflow from 2.9.3 to 2.10.5
- Fix ClickHouse connection to use native protocol port 9000
- Mount SQL directory in docker-compose for DAG execution
- Update project requirements and documentation comments
- Remove unused pandas dependency

...

This commit is contained in:

ddadmin

2026-02-07 21:52:31 +03:00

parent de9d0429a2

commit 14d16c5cae

7 changed files with 459 additions and 35 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

dags/__pycache__/ddl_init_dag.cpython-312.pyc

BIN

Binary file not shown.

dags/__pycache__/etl_pipeline_dag.cpython-312.pyc

BIN

Binary file not shown.