refactor(airflow): move DAGs to airflow/dags and update paths

- Why:
  - keep Airflow artifacts under a single airflow/ directory
  - align repository layout with intended project structure
- What:
  - move dags/ to airflow/dags/ and update compose mounts
  - make SQL root resolution work in container and local runs
  - update DAG path references in README, AGENTS, ARCHITECTURE, and plans
  - remove tracked Python cache artifacts from old DAG location
- Check:
  - airflow dags list
  - airflow dags list-import-errors
  - e2e success: ddl_init, kafka_load(limit=50), etl_pipeline
This commit is contained in:
2026-02-08 19:34:10 +03:00
parent 44a75691e8
commit 869c189fe8
14 changed files with 41 additions and 17 deletions
+3 -3
View File
@@ -146,18 +146,18 @@ ddl_init -> kafka_load -> etl_pipeline
- публикации строк из `.jsonl` (`1 строка = 1 message value`).
### Общие helper-функции
- `dags/utils/clickhouse_helpers.py`:
- `airflow/dags/utils/clickhouse_helpers.py`:
- `execute_sql(sql: str) -> None`
- `execute_sql_file(path: str) -> None`
- `fetch_one(sql: str) -> tuple`
- `dags/utils/kafka_helpers.py`:
- `airflow/dags/utils/kafka_helpers.py`:
- `prepare_topics(reset: bool) -> None`
- `load_jsonl(file_path: str, topic: str, limit: int) -> int`
- `check_kafka_ready() -> None`
## Структура файлов
```text
dags/
airflow/dags/
├── __init__.py
├── ddl_init_dag.py # отдельный DAG для DDL (обязателен)
├── kafka_load_dag.py # отдельный DAG для ingest в Kafka (обязателен)