ddmitry/clickstream-ch-kafka-superset-demo
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(infra): add airflow orchestration services

Browse Source 
Add Apache Airflow infrastructure with webserver, scheduler, and metadata
database to enable DAG-based pipeline orchestration. Includes optimized
requirements file and Docker configuration for Airflow 2.9.3.
This commit is contained in:
ddadmin
2026-02-06 23:20:58 +03:00
parent d7cff5ad1e
commit d33cdb3fb0
3 changed files with 145 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
airflow/requirements.txt
+14
View File
@@ -0,0 +1,14 @@
# Optimized requirements for Airflow Docker setup
# Only includes packages actually used in DAGs
# Core database connector
psycopg2-binary==2.9.9
# Data processing (used in data_processing_dag.py and file_operations_dag.py)
pandas==2.1.4
# Data generation library (used in file_operations_dag.py)
mimesis==15.1.0
# Airflow PostgreSQL provider (used in sql_basic_dag.py and data_processing_dag.py)
apache-airflow-providers-postgres==5.11.1