feat(airflow): configure ClickHouse connection and update infrastructure
Update Airflow configuration to integrate with ClickHouse DWH instead of PostgreSQL training database. Changes include: - Switch Airflow dependencies from PostgreSQL to ClickHouse connector - Update docker-compose to use ClickHouse connection and correct Dockerfile - Refactor airflow/requirements.txt to include only essential packages - Add DAGs directory for ETL pipeline orchestration - Update documentation to reflect Airflow integration and access credentials - Adjust service dependencies to wait for ClickHouse startup
This commit is contained in:
@@ -586,9 +586,11 @@ INSERT INTO dm.daily_traffic SELECT * FROM dm.v_daily_traffic;
|
||||
|
||||
### Airflow-оркестрация
|
||||
|
||||
Инфраструктура Airflow развёрнута и готова к использованию:
|
||||
|
||||
```python
|
||||
# dag.py
|
||||
with DAG('clickhouse_etl'):
|
||||
# dags/etl_pipeline_dag.py
|
||||
with DAG('etl_pipeline'):
|
||||
ddl = BashOperator(task_id='ddl', bash_command='make ddl')
|
||||
load = BashOperator(task_id='load', bash_command='make data')
|
||||
transform = BashOperator(task_id='transform', bash_command='make transform')
|
||||
@@ -596,6 +598,11 @@ with DAG('clickhouse_etl'):
|
||||
ddl >> load >> transform
|
||||
```
|
||||
|
||||
**Подключение к ClickHouse:**
|
||||
- Connection: `clickhouse_default`
|
||||
- URL: `clickhouse://default:123456@clickhouse:8123/default`
|
||||
- Provider: `clickhouse-connect` (в `airflow/requirements.txt`)
|
||||
|
||||
---
|
||||
|
||||
## Полезные запросы
|
||||
|
||||
Reference in New Issue
Block a user