Move DDL files from flat ddl/ directory to sql/ddl/ with layer-based
subdirectories (stg, ods, dds, dm). Move batch transformation SQL from
jobs/ to sql/ layer directories. Update scripts and documentation to
reflect new paths for improved organization and Airflow integration.
Replace materialized view joins with batch SQL transformations to avoid
consistency issues with out-of-order data. Document the reasoning for
using batch processing for ODS to DDS layer, including handling of
eventual consistency and versioning in ReplacingMergeTree. Update
data flow diagrams and remove MV creation DDL for DDS tables. Add
documentation for error handling tables and batch transformation jobs.
Add comprehensive plan for migrating executable DDL statements from markdown
to separate SQL files organized by layer. The plan outlines artifact structure,
execution requirements via make/Airflow, and environment parameters.
Existing inline DDL content is now marked as legacy in an appendix section,
providing clear separation between planned implementation and current state.
Update Kafka broker address to use internal Docker network port (29092)
instead of external port (9092). Add kafka_ingest_plan.md with detailed
implementation strategy and runbook.md with user instructions.
- Описаны слои STG/ODS/DDS/DM и связи потоков (`event_id`/`click_id`)
- Добавлены DDL и MV-пайплайн для ingestion из Kafka (ClickHouse) + типизация/дедуп/DQ
- Добавлены витрины/VIEW для BI (Superset), mermaid-диаграмма и операционные заметки
ddadmin