docs(dags): обновлена документация для csv_to_postgres пайплайна

- Зачем:
  - отражение изменений после миграции с Greenplum на PostgreSQL
  - добавление описания новых DAG-ов для обучения
- Что:
  - удалено устаревшее упоминание Greenplum в educational-setup-plan.md
  - добавлено описание csv_to_postgres.py в educational-setup-plan.md
  - добавлено описание csv_to_postgres.py и csv_to_postgres_dq.py в dag-specifications.md
  - обновлена нумерация DAG-ов в dag-specifications.md
- Проверка:
  - просмотр файлов dag-specifications.md и educational-setup-plan.md
This commit is contained in:
2026-03-08 20:15:24 +03:00
parent 8028711f0c
commit 5a3e78c9b8
2 changed files with 54 additions and 2 deletions
+53 -1
View File
@@ -89,7 +89,59 @@ id,name,department,salary
3,Charlie,Sales,48000
```
#### 2.2 data_processing_dag.py
#### 2.2 csv_to_postgres.py
**Learning Objectives:**
- Load CSV data into PostgreSQL database
- Implement data quality checks
- Use XCom for passing file paths between tasks
- Work with PostgreSQL connections in Airflow
**Scenario:**
Generate sample orders data as CSV, load it into PostgreSQL, and verify data quality.
**Tasks:**
- `create_orders_table`: Create public.orders table in PostgreSQL
- `generate_csv`: Generate sample orders CSV file
- `preview_csv`: Display first few rows of CSV
- `load_csv_to_postgres`: Load CSV data into PostgreSQL using temporary table
**Database Connection:** Uses `postgres_training` connection (auto-provisioned by init script).
**Sample Data Structure:**
```csv
order_id,order_ts,customer_id,amount
1,2023-10-01 10:30:00,101,1250.50
2,2023-10-01 11:45:00,102,890.00
3,2023-10-02 09:15:00,103,2100.75
```
**Data Quality Checks:** See `csv_to_postgres_dq.py` for automated validation.
#### 2.3 csv_to_postgres_dq.py
**Learning Objectives:**
- Implement data quality validation in Airflow
- Use Python functions for data checks
- Handle data quality failures
- Separate validation from main ETL pipeline
**Scenario:**
Run automated data quality checks on the public.orders table after CSV loading.
**Tasks:**
- `check_table_exists`: Verify public.orders table exists
- `check_schema`: Validate table schema matches expected structure
- `check_row_count`: Ensure table has data
- `check_duplicates`: Verify no duplicate order_id values
**Quality Checks:**
- Table existence in public schema
- Column names and data types (order_id, order_ts, customer_id, amount)
- Minimum row count (> 0)
- Unique order_id values (no duplicates)
**Helper Functions:** Located in `dags/helpers/postgres.py`.
#### 2.4 data_processing_dag.py
**Learning Objectives:**
- ETL pipeline concepts
- Multiple data sources
+1 -1
View File
@@ -4,7 +4,6 @@
### 1. Docker Compose Structure Problems
- Duplicate `services:` sections in [`docker-compose.yml`](airflow-docker/docker-compose.yml:1,21)
- Missing Greenplum service (referenced in dependencies but not defined)
- Inconsistent container naming
### 2. Missing Directory Structure
@@ -85,6 +84,7 @@ Create `.env` file with all variables hardcoded:
- `file_operations_dag.py` - CSV file processing
**Level 2: Intermediate**
- `csv_to_postgres.py` - CSV to PostgreSQL pipeline with data quality checks
- `data_processing_dag.py` - ETL pipeline with multiple steps
- `branching_dag.py` - Conditional task execution