ddmitry/clickstream-ch-kafka-superset-demo

T

ddadmin 78b8b29fc7 docs(plans): update ddl architecture to use batch transforms for ods to dds

Replace materialized view joins with batch SQL transformations to avoid
consistency issues with out-of-order data. Document the reasoning for
using batch processing for ODS to DDS layer, including handling of
eventual consistency and versioning in ReplacingMergeTree. Update
data flow diagrams and remove MV creation DDL for DDS tables. Add
documentation for error handling tables and batch transformation jobs.

2026-02-05 23:07:57 +03:00

configs

Наброски инфраструктуры

2026-02-04 22:47:19 +03:00

data

Текст задания

2026-02-04 22:11:07 +03:00

plans

docs(plans): update ddl architecture to use batch transforms for ods to dds

2026-02-05 23:07:57 +03:00

scripts

feat(infra): add makefile and kafka data loading script

2026-02-05 22:10:48 +03:00

AGENTS.md

docs: add runbook reference and makefile commands

2026-02-05 22:12:51 +03:00

docker-compose.yml

feat(docker): add superset service

2026-02-04 23:17:11 +03:00

Dockerfile.superset

feat(docker): add superset service

2026-02-04 23:17:11 +03:00

Makefile

feat(infra): add makefile and kafka data loading script

2026-02-05 22:10:48 +03:00

README.md

docs: add project readme

2026-02-05 22:11:10 +03:00

README.md

Kaniskin docker ETL (mini clickstream demo)

Мини‑демо аналитического стека: Kafka → ClickHouse (STG → ODS → DDS → DM) + Superset + мониторинг.

Документация

Схема слоёв и DDL (в т.ч. Kafka → STG): plans/clickhouse_ddl.md
Runbook (как поднимать/применять DDL/лить данные): plans/runbook.md

Что где лежит

Исходные данные: data/*_events.jsonl
Скрипты:
- применение DDL: scripts/apply_clickhouse_ddl.sh
- загрузка данных в Kafka: scripts/load_kafka_data.sh

Порты сервисов

См. docker-compose.yml:

ClickHouse native: localhost:8002
ClickHouse HTTP: localhost:9123
Kafka: localhost:9092
Kafka UI: http://localhost:8082
Prometheus: http://localhost:9090
Grafana: http://localhost:3000
Superset: http://localhost:8088