ddadmin
eaea33e761
- Зачем: - убрать мелкие противоречия между спецификациями, учебным планом и заданиями. - сделать сценарий csv_to_postgres и csv_to_postgres_dq понятнее для студентов. - Что: - уточнено, что csv_to_postgres отвечает за загрузку и предпросмотр данных, а DQ вынесен в отдельный DAG. - добавлены явные указания про каталог data/output и соединение postgres_training. - исправлены ссылки на задачу data_quality_summary в учебных заданиях. - Проверка: - git diff -- airflow-docker/dag-specifications.md airflow-docker/educational-setup-plan.md airflow-docker/educational-tasks.md.
152 lines
4.1 KiB
Markdown
152 lines
4.1 KiB
Markdown
# Educational Airflow Setup Plan for Beginners
|
|
|
|
## Current Issues Identified
|
|
|
|
### 1. Docker Compose Structure Problems
|
|
- Duplicate `services:` sections in [`docker-compose.yml`](airflow-docker/docker-compose.yml:1,21)
|
|
- Inconsistent container naming
|
|
|
|
### 2. Missing Directory Structure
|
|
- No `airflow/dags/` directory
|
|
- No `data/` directory for sample files
|
|
- No initialization scripts
|
|
|
|
## Proposed Simple Educational Setup
|
|
|
|
### Core Components
|
|
|
|
#### 1. Fixed Docker Compose Structure
|
|
```yaml
|
|
services:
|
|
# PostgreSQL for Airflow metadata
|
|
postgres-metadata:
|
|
image: postgres:16
|
|
environment:
|
|
POSTGRES_USER: airflow
|
|
POSTGRES_PASSWORD: airflow
|
|
POSTGRES_DB: airflow
|
|
ports:
|
|
- "5434:5432"
|
|
volumes:
|
|
- pgmeta:/var/lib/postgresql/data
|
|
|
|
# PostgreSQL for training exercises
|
|
postgres-training:
|
|
image: postgres:16
|
|
environment:
|
|
POSTGRES_USER: student
|
|
POSTGRES_PASSWORD: student
|
|
POSTGRES_DB: training
|
|
ports:
|
|
- "5432:5432"
|
|
volumes:
|
|
- pg_data:/var/lib/postgresql/data
|
|
|
|
# Airflow services
|
|
airflow-webserver:
|
|
image: apache/airflow:2.9.2
|
|
environment:
|
|
AIRFLOW__CORE__LOAD_EXAMPLES: "False"
|
|
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres-metadata:5432/airflow
|
|
ports:
|
|
- "8080:8080"
|
|
volumes:
|
|
- ./dags:/opt/airflow/dags
|
|
- ./data:/opt/airflow/data
|
|
|
|
airflow-scheduler:
|
|
image: apache/airflow:2.9.2
|
|
environment:
|
|
AIRFLOW__CORE__LOAD_EXAMPLES: "False"
|
|
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres-metadata:5432/airflow
|
|
volumes:
|
|
- ./dags:/opt/airflow/dags
|
|
- ./data:/opt/airflow/data
|
|
|
|
airflow-init:
|
|
image: apache/airflow:2.9.2
|
|
environment:
|
|
AIRFLOW__CORE__LOAD_EXAMPLES: "False"
|
|
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres-metadata:5432/airflow
|
|
```
|
|
|
|
#### 2. Hardcoded Environment Variables
|
|
Create `.env` file with all variables hardcoded:
|
|
- Airflow admin: admin/admin
|
|
- PostgreSQL metadata: airflow/airflow
|
|
- PostgreSQL training: student/student
|
|
|
|
#### 3. Educational DAG Examples
|
|
|
|
**Level 1: Basic Concepts**
|
|
- `hello_world_dag.py` - Simple print statements
|
|
- `sql_basic_dag.py` - Basic SQL operations
|
|
- `file_operations_dag.py` - CSV file processing
|
|
|
|
**Level 2: Intermediate**
|
|
- `csv_to_postgres.py` - CSV to PostgreSQL pipeline
|
|
- `csv_to_postgres_dq.py` - separate data quality checks for loaded orders
|
|
- `data_processing_dag.py` - ETL pipeline with multiple steps
|
|
- `branching_dag.py` - Conditional task execution
|
|
|
|
#### 4. Sample Data Structure
|
|
```
|
|
data/
|
|
├── input/
|
|
│ ├── customers.csv
|
|
│ ├── orders.csv
|
|
│ └── products.csv
|
|
├── output/
|
|
└── logs/
|
|
```
|
|
|
|
#### 5. Learning Progression
|
|
|
|
**Week 1: Airflow Basics**
|
|
- DAG structure and syntax
|
|
- Basic operators (PythonOperator, BashOperator)
|
|
- Task dependencies
|
|
|
|
**Week 2: SQL Integration**
|
|
- PostgreSQL connections
|
|
- SQL execution in tasks
|
|
- Data transformation
|
|
|
|
**Week 3: Real-world Scenarios**
|
|
- Error handling and retries
|
|
- Parameter passing
|
|
- Scheduling
|
|
|
|
### Implementation Steps
|
|
|
|
1. **Fix Docker Compose** - Remove duplicate sections, add missing services
|
|
2. **Create .env file** - Hardcode all environment variables
|
|
3. **Setup directories** - Create dags/, data/input/, data/output/
|
|
4. **Create sample DAGs** - From simple to complex
|
|
5. **Add sample data** - CSV files for practical exercises
|
|
6. **Create requirements.txt** - Essential Python packages
|
|
7. **Update documentation** - Clear step-by-step instructions
|
|
8. **Test setup** - Ensure everything works end-to-end
|
|
|
|
### Key Educational Principles
|
|
|
|
- **Simplicity First** - Start with minimal configuration
|
|
- **Progressive Complexity** - Build skills step by step
|
|
- **Practical Focus** - Real data processing tasks
|
|
- **Immediate Feedback** - Students see results quickly
|
|
|
|
### Sample Exercise Structure
|
|
|
|
Each DAG will include:
|
|
- Clear comments explaining each component
|
|
- Step-by-step task definitions
|
|
- Expected output descriptions
|
|
- Common pitfalls and solutions
|
|
|
|
### Technical Requirements
|
|
|
|
- Docker and Docker Compose
|
|
- Basic Python knowledge
|
|
- Basic SQL knowledge
|
|
- Web browser for Airflow UI
|