ddmitry/clickstream-ch-kafka-superset-demo
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(airflow): move DAGs to airflow/dags and update paths

Browse Source 
- Why:
  - keep Airflow artifacts under a single airflow/ directory
  - align repository layout with intended project structure
- What:
  - move dags/ to airflow/dags/ and update compose mounts
  - make SQL root resolution work in container and local runs
  - update DAG path references in README, AGENTS, ARCHITECTURE, and plans
  - remove tracked Python cache artifacts from old DAG location
- Check:
  - airflow dags list
  - airflow dags list-import-errors
  - e2e success: ddl_init, kafka_load(limit=50), etl_pipeline
This commit is contained in:
ddadmin
2026-02-08 19:34:10 +03:00
parent 44a75691e8
commit 869c189fe8
14 changed files with 41 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ARCHITECTURE.md
+3 -3
View File
@@ -578,9 +578,9 @@ INSERT INTO dm.daily_traffic SELECT * FROM dm.v_daily_traffic;
Инфраструктура Airflow развёрнута и готова к использованию:
```python
# dags/ddl_init_dag.py — создание баз/таблиц (ручной запуск при bootstrap)
# dags/kafka_load_dag.py — загрузка JSONL в Kafka (через kafka-python)
# dags/etl_pipeline_dag.py — основной ETL (STG→ODS→DDS→DM)
# airflow/dags/ddl_init_dag.py — создание баз/таблиц (ручной запуск при bootstrap)
# airflow/dags/kafka_load_dag.py — загрузка JSONL в Kafka (через kafka-python)
# airflow/dags/etl_pipeline_dag.py — основной ETL (STG→ODS→DDS→DM)
# Учебный формат:
# - DDL и трансформации выполняются явными SQL-task через ClickHouseOperator;