ddmitry/clickstream-ch-kafka-superset-demo
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity
142 Commits 1 Branch 0 Tags
d38124263af241d1f622be1425c1868c5f5a9658
Commit Graph
4 Commits
Author SHA1 Message Date
ddadminandDmitry Dementiev 04541677ff feat(generator): доработка по ревью rev5 — тики, метрики, история 
- Зачем:
  - ревью rev5 выявило несоответствие плану: дефолт 60s против 5s,
    неработающий jitter, отсутствие Prometheus-метрик и персистентности
- Что:
  - дефолт GEN_TICK_SECONDS изменён с 60s на 5s (steady-stream режим)
  - реализован GEN_JITTER_PCT в расчёте объёма тика (вариативность)
  - добавлены Prometheus метрики: generator_events_total,
    generator_publish_errors_total, generator_tick_duration_seconds,
    generator_last_success_timestamp (порт 9109)
  - добавлен ClickHouseBatchHistory с записью в meta.generator_batches
  - lazy import kafka-python для тестов без Kafka
  - обновлён scrape_config в configs/prometheus.yml
  - удалён нерабочий verify_kafka.py
- Проверка:
  - test_comprehensive.py: 9/9 тестов пройдено
  - docker build -t generator:rev5 . — успешно
  - docker-compose.yml валиден
 2026-06-09 17:27:01 +03:00
ddadmin 4917a9a6ad feat(monitoring): add Airflow monitoring via statsd-exporter 
- Add statsd-exporter service to docker-compose.yml (prom/statsd-exporter:v0.27.1)
- Add StatsD env vars to airflow-default-env for metrics export
- Add airflow job to prometheus.yml scrape configs
- Add Airflow Overview dashboard (Grafana provisioning)
- Add Airflow alert rules: scheduler down, queue backlog, failures, parse time
- Add configs/statsd_mapping.yml for StatsD → Prometheus conversion
- Use Prometheus naming convention (_total for counters, _seconds for timers)
- Add monitoring plan at plans/monitoring_airflow_plan.md
- Update OPERATIONS.md and Makefile for airflow monitoring

Tested: all 3 jobs (airflow, clickhouse, kafka) showing UP in Prometheus,
metrics flowing (dagbag_size=3, executor slots, heartbeats with _total suffix),
all 4 alert rules loaded in Grafana
 2026-02-08 22:27:32 +03:00
ddadmin ae593fd08c feat(monitoring): add Kafka monitoring via kafka-exporter 
- Add kafka-exporter service to docker-compose.yml
- Add kafka job to prometheus.yml scrape configs
- Add Kafka Overview dashboard (Grafana provisioning)
- Add Kafka alert rules (broker down, consumer lag, etc.)
- Add make reload-monitoring command for easy updates
- Update OPERATIONS.md with TL;DR and troubleshooting

API verified via Context7:
- /danielqsj/kafka_exporter for exporter config
- /prometheus/docs for scrape_configs format
 2026-02-08 21:25:31 +03:00
ddadmin 06ffb0cf16 Наброски инфраструктуры 2026-02-04 22:47:19 +03:00