ddadmin
14d16c5cae
Add comprehensive DAG implementation for ClickHouse schema initialization and ETL pipeline orchestration. The ddl_init_dag manages database schema creation across stg/ods/dds/dm layers with verification capabilities. The etl_pipeline_dag implements full ODS to DDS to DM transformation flow with data quality checks, branching logic for full/incremental loads, and timeout handling for data availability. Additional changes: - Upgrade Airflow from 2.9.3 to 2.10.5 - Fix ClickHouse connection to use native protocol port 9000 - Mount SQL directory in docker-compose for DAG execution - Update project requirements and documentation comments - Remove unused pandas dependency
8 lines
196 B
Plaintext
8 lines
196 B
Plaintext
# Airflow requirements для учебного ETL-проекта
|
|
|
|
# Metadata DB для Airflow
|
|
psycopg2-binary==2.9.9
|
|
|
|
# ClickHouse operator/hook для DAG'ов
|
|
airflow-clickhouse-plugin==1.6.0
|