Документация

2025-10-19 20:17:29 +03:00
parent 48911d7f62
commit a58670783c
3 changed files with 285 additions and 20 deletions
@@ -0,0 +1,70 @@
+# CHANGELOG - Educational Airflow Setup
+
+## 📋 Summary of Changes Made
+
+### 1. Fixed Docker Compose Structure
+- Removed duplicate `services:` sections
+- Added missing service definitions (postgres-metadata, postgres-training, airflow-webserver, airflow-scheduler, airflow-init)
+- Configured proper service dependencies and health checks
+
+### 2. Created Hardcoded Environment Variables
+- `.env` file with all necessary credentials:
+  - Airflow: admin/admin
+  - PostgreSQL metadata: airflow/airflow
+  - PostgreSQL training: student/student
+
+### 3. Implemented Educational DAG Examples
+
+**Level 1: Basic Concepts**
+- `hello_world_dag.py` - Simple PythonOperator tasks with dependencies
+
+**Level 2: Intermediate Integration**
+- `sql_basic_dag.py` - PostgreSQL operations
+- `file_operations_dag.py` - File processing and CSV operations
+
+**Level 3: Advanced Features**
+- `data_processing_dag.py` - ETL pipeline with multiple steps
+
+**Level 4: Conditional Logic**
+- `branching_dag.py` - BranchPythonOperator and conditional execution
+
+**Level 5: Error Handling**
+- `error_handling_dag.py` - Task retries and error management
+
+### 4. Sample Data Files
+- `customers.csv` - Customer data for exercises
+- `orders.csv` - Order data for practical work
+
+### 5. Database Configuration
+- PostgreSQL for Airflow metadata (port 5433)
+- PostgreSQL for training exercises (port 5432)
+
+### 6. Configuration Files
+- `requirements.txt` - Essential Python packages
+- Updated `README.md` with comprehensive setup instructions
+
+## 🎯 Key Features for Beginners
+
+### Simplified Setup
+- All environment variables hardcoded in `.env`
+- Clear port mappings and access credentials
+
+### Progressive Learning
+- From basic "Hello World" to advanced ETL pipelines
+- Hands-on exercises with real data
+
+### Technical Specifications
+- Airflow version: 2.9.2
+- PostgreSQL version: 16
+- LocalExecutor for simplicity
+
+## 🚀 Quick Start Commands
+
+```bash
+# Start all services
+docker-compose up -d
+
+# Access interfaces
+- Airflow UI: http://localhost:8080
+- Training PostgreSQL: localhost:5432
+- Metadata PostgreSQL: localhost:5433