ddadmin
688c46c68b
- Зачем: - нужен воспроизводимый вход в курс и единое место хранения планов по модулям. - Что: - добавлены `plans/README.md`, living plan Модуля 1, `START_HERE.md` и канонический ноутбук `01_environment_and_smoke_test.ipynb`, а `cluster_smoke.py` расширен до reusable helper и CLI smoke test. - уточнены onboarding-материалы и окружение: добавлены Spark UI в `README.md`, ресурсы хоста и креды MinIO в `START_HERE.md`, использован `NB_GID` в `jupyter/Dockerfile`, в ноутбуке усилены самостоятельные задания и добавлены `cell id`, а пояснения в `cluster_smoke.py` переведены на русский для студентов. - обновлены `README.md`, `AGENTS.md` и archive howto, удалены устаревшие `spark-basic-test.ipynb` и `03_partitioning_and_schema_evolution.ipynb`. - Проверка: - `python3 -m py_compile src/spark/cluster_smoke.py src/spark/__init__.py`. - `docker compose build spark-master` и `docker compose build jupyter`. - `docker compose up -d`, `docker compose exec jupyter python3 -c "from spark.cluster_smoke import create_spark_session, run_cluster_smoke; spark=create_spark_session(app_name='module-01-validation'); print(run_cluster_smoke(spark)); spark.stop()"` и `docker compose exec jupyter jupyter nbconvert --to notebook --execute /opt/work/01_environment_and_smoke_test.ipynb --output-dir /tmp --output module1-validation-2.ipynb`.
58 lines
2.8 KiB
Markdown
Executable File
58 lines
2.8 KiB
Markdown
Executable File
# Repository Guidelines
|
|
|
|
This repository contains a small teaching Lakehouse stack (Spark + Trino + Iceberg + MinIO) intended for demos and mentoring.
|
|
|
|
## Project Structure & Module Organization
|
|
|
|
- `docker-compose.yml`: orchestrates Spark, Trino, MinIO, PostgreSQL, and Jupyter.
|
|
- `spark/`: Spark image (`Dockerfile`) and `spark-defaults.conf`.
|
|
- `jupyter/`: Jupyter image (`Dockerfile`) based on the Spark image.
|
|
- `trino/`: catalog config, e.g. `trino/catalog/lakehouse.properties`.
|
|
- `src/`: PySpark and SQL examples split by engine (`src/spark/...`, `src/trino/...`).
|
|
- `notebooks/`: demo notebooks (mounted into Jupyter at `/opt/work`).
|
|
- `plans/`: internal living docs for implementation plans.
|
|
|
|
Keep new examples in `src/` or `notebooks/`, and avoid mixing configuration and code.
|
|
|
|
## Build, Test, and Development Commands
|
|
|
|
Run from the repo root:
|
|
|
|
- `docker compose build`: build custom Spark and Jupyter images.
|
|
- `docker compose up -d`: start the full stack in the background.
|
|
- `docker compose ps`: check container status.
|
|
- `docker compose down -v`: stop the stack and remove volumes (for a clean slate).
|
|
|
|
Use `docker compose logs -f <service>` when debugging (`spark-master`, `trino`, `minio`, etc.).
|
|
|
|
## Coding Style & Naming Conventions
|
|
|
|
- Python: PEP 8, 4-space indentation, snake_case for functions, lower_snake_case for files (e.g. `spark_join_demo.py`).
|
|
- SQL: uppercase keywords, `schema.table` naming, short English identifiers; comments may be in Russian.
|
|
- Compose/Docker: service names kebab-case (`spark-master`), env vars UPPER_SNAKE_CASE.
|
|
- Prefer small, focused examples; reuse helpers from `src/` in notebooks where possible.
|
|
|
|
## Testing Guidelines
|
|
|
|
There is no formal automated test suite yet. Validate changes by:
|
|
|
|
- building and starting the stack, then
|
|
- running `src/spark/cluster_smoke.py` or the SQL in `src/spark/iceberg_demo.sql`,
|
|
- opening `notebooks/01_environment_and_smoke_test.ipynb` in Jupyter and checking it runs end-to-end.
|
|
|
|
If you add tests, prefer `pytest` under `tests/` and mark slow, integration-heavy tests clearly.
|
|
|
|
## Commit & Pull Request Guidelines
|
|
|
|
- Commits: short, descriptive subject in Russian or English, present tense (e.g. `Спарк запускается, создаётся тестовая таблица`, `Add Iceberg join demo`).
|
|
- Keep changes small and focused; update `README.md` when behavior, ports, or images change.
|
|
- PRs should describe the problem, the solution, and any impact on local setup; include screenshots (Trino UI, MinIO, Jupyter) when UI changes are relevant.
|
|
|
|
## Agent-Specific Instructions
|
|
|
|
When editing as an automated agent:
|
|
|
|
- Respect this file and avoid large refactors without an explicit request.
|
|
- Preserve Russian user-facing text unless the change is explicitly about translation.
|
|
- Prefer minimal changes that keep the demo simple and robust for newcomers.
|