- Why:
- For DE task we only need full ingest or limit-based sample.
- load_* and full_load params were redundant and unclear in current flow.
- What:
- Remove full_load and load_* params from kafka_load DAG contract.
- Simplify kafka helpers (validate/check files) to fixed 4-stream ingest.
- Sync AGENTS, README, runbook, architecture and airflow plan docs.
- Check:
- python3 -m py_compile dags/kafka_load_dag.py dags/utils/kafka_helpers.py
- Airflow smoke/full runs: ddl_init -> kafka_load -> etl_pipeline (all success).
- Legacy path: make data && make transform (success).
ddadmin
1b9991f595