fix(airflow): simplify kafka_load params and align docs
- Why: - For DE task we only need full ingest or limit-based sample. - load_* and full_load params were redundant and unclear in current flow. - What: - Remove full_load and load_* params from kafka_load DAG contract. - Simplify kafka helpers (validate/check files) to fixed 4-stream ingest. - Sync AGENTS, README, runbook, architecture and airflow plan docs. - Check: - python3 -m py_compile dags/kafka_load_dag.py dags/utils/kafka_helpers.py - Airflow smoke/full runs: ddl_init -> kafka_load -> etl_pipeline (all success). - Legacy path: make data && make transform (success).
This commit is contained in:
@@ -108,15 +108,12 @@
|
||||
- Параметры:
|
||||
- `limit` (int, default 0) — количество строк (0 = все)
|
||||
- `reset_topics` (bool, default true) — пересоздать топики
|
||||
- `load_browser/load_location/load_device/load_geo` — выбор потоков
|
||||
- Примеры запуска:
|
||||
```json
|
||||
// Полная загрузка (по умолчанию)
|
||||
{}
|
||||
// Ограниченная загрузка — 100 строк
|
||||
{"limit": 100}
|
||||
// Только browser_events
|
||||
{"limit": 100, "load_location": false, "load_device": false, "load_geo": false}
|
||||
```
|
||||
|
||||
### `etl_pipeline` — ETL процесс
|
||||
@@ -143,7 +140,7 @@ make up
|
||||
|
||||
# 3. Загрузка данных через Airflow (вместо make data)
|
||||
# Airflow UI → DAGs → kafka_load → Trigger DAG with config
|
||||
# Параметры по умолчанию: limit=50, все потоки
|
||||
# Параметры по умолчанию: limit=0 (полная загрузка), reset_topics=true
|
||||
|
||||
# 4. Запуск ETL
|
||||
# Airflow UI → DAGs → etl_pipeline → Trigger DAG with config
|
||||
|
||||
Reference in New Issue
Block a user