fix(airflow): simplify kafka_load params and align docs

- Why:

  - For DE task we only need full ingest or limit-based sample.

  - load_* and full_load params were redundant and unclear in current flow.

- What:

  - Remove full_load and load_* params from kafka_load DAG contract.

  - Simplify kafka helpers (validate/check files) to fixed 4-stream ingest.

  - Sync AGENTS, README, runbook, architecture and airflow plan docs.

- Check:

  - python3 -m py_compile dags/kafka_load_dag.py dags/utils/kafka_helpers.py

  - Airflow smoke/full runs: ddl_init -> kafka_load -> etl_pipeline (all success).

  - Legacy path: make data && make transform (success).
This commit is contained in:
2026-02-08 18:33:54 +03:00
parent 10f5bc3510
commit 1b9991f595
7 changed files with 22 additions and 179 deletions
-1
View File
@@ -33,7 +33,6 @@ docker compose exec -T airflow-webserver airflow dags trigger kafka_load \
Параметры `kafka_load`:
- `limit` — количество строк (default: 0 — все строки)
- `reset_topics` — пересоздать топики (default: true)
- `load_browser/load_location/load_device/load_geo` — выбор потоков (default: true)
**Вариант Б: Через shell-скрипт `make data` (устаревший)**
```bash