fix(airflow): simplify kafka_load params and align docs
- Why: - For DE task we only need full ingest or limit-based sample. - load_* and full_load params were redundant and unclear in current flow. - What: - Remove full_load and load_* params from kafka_load DAG contract. - Simplify kafka helpers (validate/check files) to fixed 4-stream ingest. - Sync AGENTS, README, runbook, architecture and airflow plan docs. - Check: - python3 -m py_compile dags/kafka_load_dag.py dags/utils/kafka_helpers.py - Airflow smoke/full runs: ddl_init -> kafka_load -> etl_pipeline (all success). - Legacy path: make data && make transform (success).
This commit is contained in:
@@ -615,7 +615,7 @@ INSERT INTO dm.daily_traffic SELECT * FROM dm.v_daily_traffic;
|
||||
- Загрузка данных из `data/*.jsonl` в Kafka через `kafka-python`
|
||||
- TaskGroup `precheck`: проверка Kafka, файлов, параметров
|
||||
- TaskGroup `ingest`: создание топиков → параллельная загрузка 4 потоков → проверка
|
||||
- Параметры: `limit` (0 = все), `reset_topics`, `load_*` (выбор потоков)
|
||||
- Параметры: `limit` (0 = все), `reset_topics`
|
||||
|
||||
**Подключение к ClickHouse:**
|
||||
- Connection: `clickhouse_default`
|
||||
|
||||
Reference in New Issue
Block a user