- Why:
- keep Airflow artifacts under a single airflow/ directory
- align repository layout with intended project structure
- What:
- move dags/ to airflow/dags/ and update compose mounts
- make SQL root resolution work in container and local runs
- update DAG path references in README, AGENTS, ARCHITECTURE, and plans
- remove tracked Python cache artifacts from old DAG location
- Check:
- airflow dags list
- airflow dags list-import-errors
- e2e success: ddl_init, kafka_load(limit=50), etl_pipeline
- Why:\n - User-facing docs mixed Airflow and legacy CLI ingest paths and caused confusion\n- What:\n - Rework README quick start and status to use DAG chain ddl_init -> kafka_load -> etl_pipeline\n - Rewrite runbook as canonical Airflow-first execution flow\n - Sync architecture diagrams/sequence and DQ wording with current SQL and DAG behavior\n- Check:\n - Verified updated sections and removed stale markers with rg in README.md, docs/ARCHITECTURE.md, plans/runbook.md
- Why:
- For DE task we only need full ingest or limit-based sample.
- load_* and full_load params were redundant and unclear in current flow.
- What:
- Remove full_load and load_* params from kafka_load DAG contract.
- Simplify kafka helpers (validate/check files) to fixed 4-stream ingest.
- Sync AGENTS, README, runbook, architecture and airflow plan docs.
- Check:
- python3 -m py_compile dags/kafka_load_dag.py dags/utils/kafka_helpers.py
- Airflow smoke/full runs: ddl_init -> kafka_load -> etl_pipeline (all success).
- Legacy path: make data && make transform (success).
Move DDL files from flat ddl/ directory to sql/ddl/ with layer-based
subdirectories (stg, ods, dds, dm). Move batch transformation SQL from
jobs/ to sql/ layer directories. Update scripts and documentation to
reflect new paths for improved organization and Airflow integration.
Split Kafka ingestion into a dedicated `kafka_load` DAG to enable independent
experimentation with data loading without triggering the full ETL pipeline.
Restructure implementation phases: Stage 1 uses `make data` for MVP, Stage 2
adds the standalone Kafka DAG, Stage 3 adds monitoring. Update DAG numbering,
parameters, task groups, and acceptance criteria to reflect the new
architecture.
Consolidate the orchestration strategy by merging `kafka_load` and
`etl_batch_transform` into a unified `etl_pipeline`. Replace BashOperator
dependencies on Kafka CLI with PythonOperators utilizing `kafka-python`.
Add detailed technical specifications for helper functions, MVP stages,
and validation checks to align with current infrastructure constraints.
Detail the architecture for migrating ETL orchestration from make to
Airflow. Define DAG structures for database initialization, Kafka data
ingestion, batch transformation, and quality monitoring. Include
technical specifications, operator details, and implementation phases.
Replace materialized view joins with batch SQL transformations to avoid
consistency issues with out-of-order data. Document the reasoning for
using batch processing for ODS to DDS layer, including handling of
eventual consistency and versioning in ReplacingMergeTree. Update
data flow diagrams and remove MV creation DDL for DDS tables. Add
documentation for error handling tables and batch transformation jobs.
Add comprehensive plan for migrating executable DDL statements from markdown
to separate SQL files organized by layer. The plan outlines artifact structure,
execution requirements via make/Airflow, and environment parameters.
Existing inline DDL content is now marked as legacy in an appendix section,
providing clear separation between planned implementation and current state.
Add build automation via Makefile with targets for docker compose
management, DDL application, and data ingestion. Implement a robust bash
script for loading JSONL demo data into Kafka topics with configurable
options for limits, full dataset loading, and topic reset behavior.
Update Kafka broker address to use internal Docker network port (29092)
instead of external port (9092). Add kafka_ingest_plan.md with detailed
implementation strategy and runbook.md with user instructions.
- Описаны слои STG/ODS/DDS/DM и связи потоков (`event_id`/`click_id`)
- Добавлены DDL и MV-пайплайн для ingestion из Kafka (ClickHouse) + типизация/дедуп/DQ
- Добавлены витрины/VIEW для BI (Superset), mermaid-диаграмма и операционные заметки
ddadmin