feat(airflow): configure ClickHouse connection and update infrastructure

Update Airflow configuration to integrate with ClickHouse DWH instead of
PostgreSQL training database. Changes include:

- Switch Airflow dependencies from PostgreSQL to ClickHouse connector
- Update docker-compose to use ClickHouse connection and correct Dockerfile
- Refactor airflow/requirements.txt to include only essential packages
- Add DAGs directory for ETL pipeline orchestration
- Update documentation to reflect Airflow integration and access credentials
- Adjust service dependencies to wait for ClickHouse startup
This commit is contained in:
2026-02-06 23:34:45 +03:00
parent d33cdb3fb0
commit fe9c15c0fe
9 changed files with 102 additions and 34 deletions
+7 -10
View File
@@ -1,14 +1,11 @@
# Optimized requirements for Airflow Docker setup
# Only includes packages actually used in DAGs
# Airflow requirements для ClickHouse DWH проекта
# Core database connector
# Core database connector для metadata
psycopg2-binary==2.9.9
# Data processing (used in data_processing_dag.py and file_operations_dag.py)
# ClickHouse provider для ETL
# Примечание: официальный провайдер deprecated, используем clickhouse-connect
clickhouse-connect==0.8.0
# Для работы с данными
pandas==2.1.4
# Data generation library (used in file_operations_dag.py)
mimesis==15.1.0
# Airflow PostgreSQL provider (used in sql_basic_dag.py and data_processing_dag.py)
apache-airflow-providers-postgres==5.11.1