airflow-manual/airflow-docker/CHANGELOG.md

CHANGELOG - Educational Airflow Setup

📋 Summary of Changes Made

1. Fixed Docker Compose Structure

• Removed duplicate services: sections
• Added missing service definitions (postgres-metadata, postgres-training, airflow-webserver, airflow-scheduler, airflow-init)
• Configured proper service dependencies and health checks

2. Created Hardcoded Environment Variables

• .env file with all necessary credentials:
  • Airflow: admin/admin
  • PostgreSQL metadata: airflow/airflow
  • PostgreSQL training: student/student

3. Implemented Educational DAG Examples

Level 1: Basic Concepts

• hello_world_dag.py - Simple PythonOperator tasks with dependencies

Level 2: Intermediate Integration

• sql_basic_dag.py - PostgreSQL operations
• file_operations_dag.py - File processing and CSV operations

Level 3: Advanced Features

• data_processing_dag.py - ETL pipeline with multiple steps

Level 4: Conditional Logic

• branching_dag.py - BranchPythonOperator and conditional execution

Level 5: Error Handling

• error_handling_dag.py - Task retries and error management

4. Sample Data Files

• customers.csv - Customer data for exercises
• orders.csv - Order data for practical work

5. Database Configuration

• PostgreSQL for Airflow metadata (port 5433)
• PostgreSQL for training exercises (port 5432)

6. Configuration Files

• requirements.txt - Essential Python packages
• Updated README.md with comprehensive setup instructions

🎯 Key Features for Beginners

Simplified Setup

• All environment variables hardcoded in .env
• Clear port mappings and access credentials

Progressive Learning

• From basic "Hello World" to advanced ETL pipelines
• Hands-on exercises with real data

Technical Specifications

• Airflow version: 2.9.2
• PostgreSQL version: 16
• LocalExecutor for simplicity

🚀 Quick Start Commands

# Start all services
docker-compose up -d

# Access interfaces
- Airflow UI: http://localhost:8080
- Training PostgreSQL: localhost:5432
- Metadata PostgreSQL: localhost:5433