feat(airflow): configure ClickHouse connection and update infrastructure

Update Airflow configuration to integrate with ClickHouse DWH instead of
PostgreSQL training database. Changes include:

- Switch Airflow dependencies from PostgreSQL to ClickHouse connector
- Update docker-compose to use ClickHouse connection and correct Dockerfile
- Refactor airflow/requirements.txt to include only essential packages
- Add DAGs directory for ETL pipeline orchestration
- Update documentation to reflect Airflow integration and access credentials
- Adjust service dependencies to wait for ClickHouse startup
This commit is contained in:
2026-02-06 23:34:45 +03:00
parent d33cdb3fb0
commit fe9c15c0fe
9 changed files with 102 additions and 34 deletions
+9 -2
View File
@@ -586,9 +586,11 @@ INSERT INTO dm.daily_traffic SELECT * FROM dm.v_daily_traffic;
### Airflow-оркестрация
Инфраструктура Airflow развёрнута и готова к использованию:
```python
# dag.py
with DAG('clickhouse_etl'):
# dags/etl_pipeline_dag.py
with DAG('etl_pipeline'):
ddl = BashOperator(task_id='ddl', bash_command='make ddl')
load = BashOperator(task_id='load', bash_command='make data')
transform = BashOperator(task_id='transform', bash_command='make transform')
@@ -596,6 +598,11 @@ with DAG('clickhouse_etl'):
ddl >> load >> transform
```
**Подключение к ClickHouse:**
- Connection: `clickhouse_default`
- URL: `clickhouse://default:123456@clickhouse:8123/default`
- Provider: `clickhouse-connect``airflow/requirements.txt`)
---
## Полезные запросы