feat(airflow): configure ClickHouse connection and update infrastructure
Update Airflow configuration to integrate with ClickHouse DWH instead of PostgreSQL training database. Changes include: - Switch Airflow dependencies from PostgreSQL to ClickHouse connector - Update docker-compose to use ClickHouse connection and correct Dockerfile - Refactor airflow/requirements.txt to include only essential packages - Add DAGs directory for ETL pipeline orchestration - Update documentation to reflect Airflow integration and access credentials - Adjust service dependencies to wait for ClickHouse startup
This commit is contained in:
@@ -1,14 +1,11 @@
|
||||
# Optimized requirements for Airflow Docker setup
|
||||
# Only includes packages actually used in DAGs
|
||||
# Airflow requirements для ClickHouse DWH проекта
|
||||
|
||||
# Core database connector
|
||||
# Core database connector для metadata
|
||||
psycopg2-binary==2.9.9
|
||||
|
||||
# Data processing (used in data_processing_dag.py and file_operations_dag.py)
|
||||
# ClickHouse provider для ETL
|
||||
# Примечание: официальный провайдер deprecated, используем clickhouse-connect
|
||||
clickhouse-connect==0.8.0
|
||||
|
||||
# Для работы с данными
|
||||
pandas==2.1.4
|
||||
|
||||
# Data generation library (used in file_operations_dag.py)
|
||||
mimesis==15.1.0
|
||||
|
||||
# Airflow PostgreSQL provider (used in sql_basic_dag.py and data_processing_dag.py)
|
||||
apache-airflow-providers-postgres==5.11.1
|
||||
|
||||
Reference in New Issue
Block a user