docs(dags): обновлена документация для csv_to_postgres пайплайна
- Зачем: - отражение изменений после миграции с Greenplum на PostgreSQL - добавление описания новых DAG-ов для обучения - Что: - удалено устаревшее упоминание Greenplum в educational-setup-plan.md - добавлено описание csv_to_postgres.py в educational-setup-plan.md - добавлено описание csv_to_postgres.py и csv_to_postgres_dq.py в dag-specifications.md - обновлена нумерация DAG-ов в dag-specifications.md - Проверка: - просмотр файлов dag-specifications.md и educational-setup-plan.md
This commit is contained in:
@@ -89,7 +89,59 @@ id,name,department,salary
|
|||||||
3,Charlie,Sales,48000
|
3,Charlie,Sales,48000
|
||||||
```
|
```
|
||||||
|
|
||||||
#### 2.2 data_processing_dag.py
|
#### 2.2 csv_to_postgres.py
|
||||||
|
**Learning Objectives:**
|
||||||
|
- Load CSV data into PostgreSQL database
|
||||||
|
- Implement data quality checks
|
||||||
|
- Use XCom for passing file paths between tasks
|
||||||
|
- Work with PostgreSQL connections in Airflow
|
||||||
|
|
||||||
|
**Scenario:**
|
||||||
|
Generate sample orders data as CSV, load it into PostgreSQL, and verify data quality.
|
||||||
|
|
||||||
|
**Tasks:**
|
||||||
|
- `create_orders_table`: Create public.orders table in PostgreSQL
|
||||||
|
- `generate_csv`: Generate sample orders CSV file
|
||||||
|
- `preview_csv`: Display first few rows of CSV
|
||||||
|
- `load_csv_to_postgres`: Load CSV data into PostgreSQL using temporary table
|
||||||
|
|
||||||
|
**Database Connection:** Uses `postgres_training` connection (auto-provisioned by init script).
|
||||||
|
|
||||||
|
**Sample Data Structure:**
|
||||||
|
```csv
|
||||||
|
order_id,order_ts,customer_id,amount
|
||||||
|
1,2023-10-01 10:30:00,101,1250.50
|
||||||
|
2,2023-10-01 11:45:00,102,890.00
|
||||||
|
3,2023-10-02 09:15:00,103,2100.75
|
||||||
|
```
|
||||||
|
|
||||||
|
**Data Quality Checks:** See `csv_to_postgres_dq.py` for automated validation.
|
||||||
|
|
||||||
|
#### 2.3 csv_to_postgres_dq.py
|
||||||
|
**Learning Objectives:**
|
||||||
|
- Implement data quality validation in Airflow
|
||||||
|
- Use Python functions for data checks
|
||||||
|
- Handle data quality failures
|
||||||
|
- Separate validation from main ETL pipeline
|
||||||
|
|
||||||
|
**Scenario:**
|
||||||
|
Run automated data quality checks on the public.orders table after CSV loading.
|
||||||
|
|
||||||
|
**Tasks:**
|
||||||
|
- `check_table_exists`: Verify public.orders table exists
|
||||||
|
- `check_schema`: Validate table schema matches expected structure
|
||||||
|
- `check_row_count`: Ensure table has data
|
||||||
|
- `check_duplicates`: Verify no duplicate order_id values
|
||||||
|
|
||||||
|
**Quality Checks:**
|
||||||
|
- Table existence in public schema
|
||||||
|
- Column names and data types (order_id, order_ts, customer_id, amount)
|
||||||
|
- Minimum row count (> 0)
|
||||||
|
- Unique order_id values (no duplicates)
|
||||||
|
|
||||||
|
**Helper Functions:** Located in `dags/helpers/postgres.py`.
|
||||||
|
|
||||||
|
#### 2.4 data_processing_dag.py
|
||||||
**Learning Objectives:**
|
**Learning Objectives:**
|
||||||
- ETL pipeline concepts
|
- ETL pipeline concepts
|
||||||
- Multiple data sources
|
- Multiple data sources
|
||||||
|
|||||||
@@ -4,7 +4,6 @@
|
|||||||
|
|
||||||
### 1. Docker Compose Structure Problems
|
### 1. Docker Compose Structure Problems
|
||||||
- Duplicate `services:` sections in [`docker-compose.yml`](airflow-docker/docker-compose.yml:1,21)
|
- Duplicate `services:` sections in [`docker-compose.yml`](airflow-docker/docker-compose.yml:1,21)
|
||||||
- Missing Greenplum service (referenced in dependencies but not defined)
|
|
||||||
- Inconsistent container naming
|
- Inconsistent container naming
|
||||||
|
|
||||||
### 2. Missing Directory Structure
|
### 2. Missing Directory Structure
|
||||||
@@ -85,6 +84,7 @@ Create `.env` file with all variables hardcoded:
|
|||||||
- `file_operations_dag.py` - CSV file processing
|
- `file_operations_dag.py` - CSV file processing
|
||||||
|
|
||||||
**Level 2: Intermediate**
|
**Level 2: Intermediate**
|
||||||
|
- `csv_to_postgres.py` - CSV to PostgreSQL pipeline with data quality checks
|
||||||
- `data_processing_dag.py` - ETL pipeline with multiple steps
|
- `data_processing_dag.py` - ETL pipeline with multiple steps
|
||||||
- `branching_dag.py` - Conditional task execution
|
- `branching_dag.py` - Conditional task execution
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user