70 lines
2.0 KiB
Markdown
70 lines
2.0 KiB
Markdown
# CHANGELOG - Educational Airflow Setup
|
|
|
|
## 📋 Summary of Changes Made
|
|
|
|
### 1. Fixed Docker Compose Structure
|
|
- Removed duplicate `services:` sections
|
|
- Added missing service definitions (postgres-metadata, postgres-training, airflow-webserver, airflow-scheduler, airflow-init)
|
|
- Configured proper service dependencies and health checks
|
|
|
|
### 2. Created Hardcoded Environment Variables
|
|
- `.env` file with all necessary credentials:
|
|
- Airflow: admin/admin
|
|
- PostgreSQL metadata: airflow/airflow
|
|
- PostgreSQL training: student/student
|
|
|
|
### 3. Implemented Educational DAG Examples
|
|
|
|
**Level 1: Basic Concepts**
|
|
- `hello_world_dag.py` - Simple PythonOperator tasks with dependencies
|
|
|
|
**Level 2: Intermediate Integration**
|
|
- `sql_basic_dag.py` - PostgreSQL operations
|
|
- `file_operations_dag.py` - File processing and CSV operations
|
|
|
|
**Level 3: Advanced Features**
|
|
- `data_processing_dag.py` - ETL pipeline with multiple steps
|
|
|
|
**Level 4: Conditional Logic**
|
|
- `branching_dag.py` - BranchPythonOperator and conditional execution
|
|
|
|
**Level 5: Error Handling**
|
|
- `error_handling_dag.py` - Task retries and error management
|
|
|
|
### 4. Sample Data Files
|
|
- `customers.csv` - Customer data for exercises
|
|
- `orders.csv` - Order data for practical work
|
|
|
|
### 5. Database Configuration
|
|
- PostgreSQL for Airflow metadata (port 5433)
|
|
- PostgreSQL for training exercises (port 5432)
|
|
|
|
### 6. Configuration Files
|
|
- `requirements.txt` - Essential Python packages
|
|
- Updated `README.md` with comprehensive setup instructions
|
|
|
|
## 🎯 Key Features for Beginners
|
|
|
|
### Simplified Setup
|
|
- All environment variables hardcoded in `.env`
|
|
- Clear port mappings and access credentials
|
|
|
|
### Progressive Learning
|
|
- From basic "Hello World" to advanced ETL pipelines
|
|
- Hands-on exercises with real data
|
|
|
|
### Technical Specifications
|
|
- Airflow version: 2.9.2
|
|
- PostgreSQL version: 16
|
|
- LocalExecutor for simplicity
|
|
|
|
## 🚀 Quick Start Commands
|
|
|
|
```bash
|
|
# Start all services
|
|
docker-compose up -d
|
|
|
|
# Access interfaces
|
|
- Airflow UI: http://localhost:8080
|
|
- Training PostgreSQL: localhost:5432
|
|
- Metadata PostgreSQL: localhost:5433 |