Add persistent volume for ClickHouse to preserve data across container
restarts. The volume `clickhouse-data` is mounted to `/var/lib/clickhouse`,
ensuring data remains when containers are recreated.
Move DDL files from flat ddl/ directory to sql/ddl/ with layer-based
subdirectories (stg, ods, dds, dm). Move batch transformation SQL from
jobs/ to sql/ layer directories. Update scripts and documentation to
reflect new paths for improved organization and Airflow integration.
Update Airflow configuration to integrate with ClickHouse DWH instead of
PostgreSQL training database. Changes include:
- Switch Airflow dependencies from PostgreSQL to ClickHouse connector
- Update docker-compose to use ClickHouse connection and correct Dockerfile
- Refactor airflow/requirements.txt to include only essential packages
- Add DAGs directory for ETL pipeline orchestration
- Update documentation to reflect Airflow integration and access credentials
- Adjust service dependencies to wait for ClickHouse startup
Add batch ETL pipeline with ODS→DDS→DM transformation jobs and scripts.
Create DDL infrastructure with automated database schema application.
Update Makefile with transform target for executing batch processes.
Rewrite README with complete Russian documentation including architecture
diagrams, quick start guide, and data flow visualization.
ddadmin