ddmitry/mini-lakehouse-lab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(module-1): доработаны материалы модуля 1 и удалён legacy-ноутбук

Browse Source 
- Зачем:
  - нужен воспроизводимый вход в курс и единое место хранения планов по модулям.
- Что:
  - добавлены `plans/README.md`, living plan Модуля 1, `START_HERE.md` и канонический ноутбук `01_environment_and_smoke_test.ipynb`, а `cluster_smoke.py` расширен до reusable helper и CLI smoke test.
  - уточнены onboarding-материалы и окружение: добавлены Spark UI в `README.md`, ресурсы хоста и креды MinIO в `START_HERE.md`, использован `NB_GID` в `jupyter/Dockerfile`, в ноутбуке усилены самостоятельные задания и добавлены `cell id`, а пояснения в `cluster_smoke.py` переведены на русский для студентов.
  - обновлены `README.md`, `AGENTS.md` и archive howto, удалены устаревшие `spark-basic-test.ipynb` и `03_partitioning_and_schema_evolution.ipynb`.
- Проверка:
  - `python3 -m py_compile src/spark/cluster_smoke.py src/spark/__init__.py`.
  - `docker compose build spark-master` и `docker compose build jupyter`.
  - `docker compose up -d`, `docker compose exec jupyter python3 -c "from spark.cluster_smoke import create_spark_session, run_cluster_smoke; spark=create_spark_session(app_name='module-01-validation'); print(run_cluster_smoke(spark)); spark.stop()"` и `docker compose exec jupyter jupyter nbconvert --to notebook --execute /opt/work/01_environment_and_smoke_test.ipynb --output-dir /tmp --output module1-validation-2.ipynb`.
This commit is contained in:
ddadmin
2026-03-07 00:24:40 +03:00
parent 9d84e15e01
commit 688c46c68b
12 changed files with 554 additions and 269 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+2 -1
View File
@@ -10,6 +10,7 @@ This repository contains a small teaching Lakehouse stack (Spark + Trino + Icebe
- `trino/`: catalog config, e.g. `trino/catalog/lakehouse.properties`.
- `src/`: PySpark and SQL examples split by engine (`src/spark/...`, `src/trino/...`).
- `notebooks/`: demo notebooks (mounted into Jupyter at `/opt/work`).
- `plans/`: internal living docs for implementation plans.
Keep new examples in `src/` or `notebooks/`, and avoid mixing configuration and code.
@@ -37,7 +38,7 @@ There is no formal automated test suite yet. Validate changes by:
- building and starting the stack, then
- running `src/spark/cluster_smoke.py` or the SQL in `src/spark/iceberg_demo.sql`,
- opening `notebooks/spark-basic-test.ipynb` in Jupyter and checking it runs end-to-end.
- opening `notebooks/01_environment_and_smoke_test.ipynb` in Jupyter and checking it runs end-to-end.
If you add tests, prefer `pytest` under `tests/` and mark slow, integration-heavy tests clearly.