ddadmin
14d16c5cae
Add comprehensive DAG implementation for ClickHouse schema initialization and ETL pipeline orchestration. The ddl_init_dag manages database schema creation across stg/ods/dds/dm layers with verification capabilities. The etl_pipeline_dag implements full ODS to DDS to DM transformation flow with data quality checks, branching logic for full/incremental loads, and timeout handling for data availability. Additional changes: - Upgrade Airflow from 2.9.3 to 2.10.5 - Fix ClickHouse connection to use native protocol port 9000 - Mount SQL directory in docker-compose for DAG execution - Update project requirements and documentation comments - Remove unused pandas dependency
24 lines
650 B
Docker
24 lines
650 B
Docker
# Use the official Airflow image as base
|
|
FROM apache/airflow:2.10.5
|
|
|
|
# Set environment variables
|
|
ENV AIRFLOW_HOME=/opt/airflow
|
|
|
|
# Copy the project requirements file
|
|
COPY --chown=airflow:0 airflow/requirements.txt ${AIRFLOW_HOME}/requirements.txt
|
|
|
|
# Install the required Python packages
|
|
RUN pip install --no-cache-dir -r ${AIRFLOW_HOME}/requirements.txt
|
|
|
|
# Create necessary directories
|
|
RUN mkdir -p /opt/airflow/dags /opt/airflow/logs /opt/airflow/config /opt/airflow/data
|
|
|
|
# Set working directory
|
|
WORKDIR ${AIRFLOW_HOME}
|
|
|
|
# Expose the webserver port
|
|
EXPOSE 8080
|
|
|
|
# The image will use Airflow's default entrypoint
|
|
# Commands will be passed at runtime
|