ddadmin
4917a9a6ad
- Add statsd-exporter service to docker-compose.yml (prom/statsd-exporter:v0.27.1) - Add StatsD env vars to airflow-default-env for metrics export - Add airflow job to prometheus.yml scrape configs - Add Airflow Overview dashboard (Grafana provisioning) - Add Airflow alert rules: scheduler down, queue backlog, failures, parse time - Add configs/statsd_mapping.yml for StatsD → Prometheus conversion - Use Prometheus naming convention (_total for counters, _seconds for timers) - Add monitoring plan at plans/monitoring_airflow_plan.md - Update OPERATIONS.md and Makefile for airflow monitoring Tested: all 3 jobs (airflow, clickhouse, kafka) showing UP in Prometheus, metrics flowing (dagbag_size=3, executor slots, heartbeats with _total suffix), all 4 alert rules loaded in Grafana
30 lines
812 B
YAML
30 lines
812 B
YAML
global:
|
|
scrape_interval: 15s
|
|
|
|
scrape_configs:
|
|
- job_name: "clickhouse"
|
|
metrics_path: "/metrics"
|
|
static_configs:
|
|
- targets: ["clickhouse:9126"]
|
|
labels:
|
|
instance: Clickhouse-1
|
|
honor_labels: true
|
|
|
|
# Kafka metrics via kafka-exporter
|
|
# Проверено через Context7: формат scrape_configs подтверждён (/prometheus/docs)
|
|
- job_name: "kafka"
|
|
metrics_path: "/metrics"
|
|
static_configs:
|
|
- targets: ["kafka-exporter:9308"]
|
|
labels:
|
|
instance: Kafka-1
|
|
honor_labels: true
|
|
|
|
# Airflow metrics via statsd-exporter (StatsD → Prometheus)
|
|
- job_name: "airflow"
|
|
metrics_path: "/metrics"
|
|
static_configs:
|
|
- targets: ["statsd-exporter:9102"]
|
|
labels:
|
|
instance: Airflow-1
|
|
honor_labels: true |