refactor(airflow): move DAGs to airflow/dags and update paths
- Why: - keep Airflow artifacts under a single airflow/ directory - align repository layout with intended project structure - What: - move dags/ to airflow/dags/ and update compose mounts - make SQL root resolution work in container and local runs - update DAG path references in README, AGENTS, ARCHITECTURE, and plans - remove tracked Python cache artifacts from old DAG location - Check: - airflow dags list - airflow dags list-import-errors - e2e success: ddl_init, kafka_load(limit=50), etl_pipeline
This commit is contained in:
@@ -146,18 +146,18 @@ ddl_init -> kafka_load -> etl_pipeline
|
||||
- публикации строк из `.jsonl` (`1 строка = 1 message value`).
|
||||
|
||||
### Общие helper-функции
|
||||
- `dags/utils/clickhouse_helpers.py`:
|
||||
- `airflow/dags/utils/clickhouse_helpers.py`:
|
||||
- `execute_sql(sql: str) -> None`
|
||||
- `execute_sql_file(path: str) -> None`
|
||||
- `fetch_one(sql: str) -> tuple`
|
||||
- `dags/utils/kafka_helpers.py`:
|
||||
- `airflow/dags/utils/kafka_helpers.py`:
|
||||
- `prepare_topics(reset: bool) -> None`
|
||||
- `load_jsonl(file_path: str, topic: str, limit: int) -> int`
|
||||
- `check_kafka_ready() -> None`
|
||||
|
||||
## Структура файлов
|
||||
```text
|
||||
dags/
|
||||
airflow/dags/
|
||||
├── __init__.py
|
||||
├── ddl_init_dag.py # отдельный DAG для DDL (обязателен)
|
||||
├── kafka_load_dag.py # отдельный DAG для ingest в Kafka (обязателен)
|
||||
|
||||
Reference in New Issue
Block a user