refactor(generator): история batch в Kafka вместо ClickHouse
- Зачем:
- ревью rev5: ClickHouse-интеграция была проблемной (порт 9000 native vs HTTP,
неработающий fallback, отсутствие DDL для базы meta)
- архитектурно чище: генератор остаётся pure Kafka producer,
история доступна для аналитики через стандартный ingestion
- Что:
- удален ClickHouseBatchHistory, clickhouse-connect зависимость
- добавлен KafkaBatchHistory с записью в топик generator_batch_history
- добавлен BatchRecord.to_dict() для JSON-сериализации
- добавлен рабочий fallback: Kafka → InMemory при недоступности
- удален pytest-asyncio (не использовался)
- добавлены тесты test_kafka_history.py (15 тестов) и test_service.py (6 тестов)
- обновлена документация: топик вместо таблицы ClickHouse
- Проверка:
- make generator-test: 44/44 тестов пройдено
- docker-compose валиден, генератор не зависит от clickhouse
This commit is contained in:
@@ -2,7 +2,7 @@
|
|||||||
reload-monitoring recover-monitoring \
|
reload-monitoring recover-monitoring \
|
||||||
superset-init superset-dashboard superset-ui superset-restart \
|
superset-init superset-dashboard superset-ui superset-restart \
|
||||||
generator-up generator-down generator-logs generator-restart \
|
generator-up generator-down generator-logs generator-restart \
|
||||||
generator-test generator-test-build
|
generator-test generator-test-build generator-test-cov
|
||||||
|
|
||||||
COMPOSE ?= docker compose
|
COMPOSE ?= docker compose
|
||||||
|
|
||||||
|
|||||||
@@ -300,8 +300,6 @@ services:
|
|||||||
GEN_DATA_DIR: /data
|
GEN_DATA_DIR: /data
|
||||||
GEN_ENABLED: "true"
|
GEN_ENABLED: "true"
|
||||||
GEN_METRICS_PORT: "9109"
|
GEN_METRICS_PORT: "9109"
|
||||||
CLICKHOUSE_HOST: clickhouse
|
|
||||||
CLICKHOUSE_PORT: "9000"
|
|
||||||
# PYTHONUNBUFFERED для сразу видеть логи
|
# PYTHONUNBUFFERED для сразу видеть логи
|
||||||
PYTHONUNBUFFERED: "1"
|
PYTHONUNBUFFERED: "1"
|
||||||
ports:
|
ports:
|
||||||
@@ -312,7 +310,6 @@ services:
|
|||||||
- cs_dwh
|
- cs_dwh
|
||||||
depends_on:
|
depends_on:
|
||||||
- kafka
|
- kafka
|
||||||
- clickhouse
|
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD", "python", "-c", "import sys; sys.exit(0)"]
|
test: ["CMD", "python", "-c", "import sys; sys.exit(0)"]
|
||||||
|
|||||||
+15
-19
@@ -36,8 +36,6 @@ generator-service -> Kafka topics -> (потребители отдельно)
|
|||||||
| `GEN_SEED` | Сид для воспроизводимости | — |
|
| `GEN_SEED` | Сид для воспроизводимости | — |
|
||||||
| `GEN_ENABLED` | Включить генерацию | `true` |
|
| `GEN_ENABLED` | Включить генерацию | `true` |
|
||||||
| `GEN_METRICS_PORT` | Порт для Prometheus | `9109` |
|
| `GEN_METRICS_PORT` | Порт для Prometheus | `9109` |
|
||||||
| `CLICKHOUSE_HOST` | Хост ClickHouse для истории | `clickhouse` |
|
|
||||||
| `CLICKHOUSE_PORT` | Порт ClickHouse | `9000` |
|
|
||||||
|
|
||||||
### Режим "раз в минуту" (для демо)
|
### Режим "раз в минуту" (для демо)
|
||||||
|
|
||||||
@@ -62,6 +60,9 @@ make generator-logs
|
|||||||
|
|
||||||
# Перезапуск с пересборкой
|
# Перезапуск с пересборкой
|
||||||
make generator-restart
|
make generator-restart
|
||||||
|
|
||||||
|
# Запуск тестов
|
||||||
|
make generator-test
|
||||||
```
|
```
|
||||||
|
|
||||||
## Метрики Prometheus
|
## Метрики Prometheus
|
||||||
@@ -84,27 +85,25 @@ curl http://localhost:9090/api/v1/targets | grep generator
|
|||||||
|
|
||||||
## История batch
|
## История batch
|
||||||
|
|
||||||
История сохраняется в таблице `meta.generator_batches` (ClickHouse):
|
История пишется в Kafka-топик `generator_batch_history` (JSON). При недоступности Kafka используется in-memory fallback (последние 1000 записей).
|
||||||
|
|
||||||
```sql
|
Поля сообщения:
|
||||||
SELECT
|
|
||||||
batch_id,
|
|
||||||
started_at,
|
|
||||||
sent_total,
|
|
||||||
status
|
|
||||||
FROM meta.generator_batches
|
|
||||||
ORDER BY started_at DESC
|
|
||||||
LIMIT 10
|
|
||||||
```
|
|
||||||
|
|
||||||
Поля:
|
|
||||||
- `batch_id` — идентификатор батча
|
- `batch_id` — идентификатор батча
|
||||||
- `started_at` / `finished_at` — время начала/окончания
|
- `started_at` / `finished_at` — время начала/окончания (ISO format)
|
||||||
- `sent_total` — всего отправлено
|
- `sent_total` — всего отправлено
|
||||||
- `sent_browser/location/device/geo` — по топикам
|
- `sent_browser/location/device/geo` — по топикам
|
||||||
- `status` — success/partial/error
|
- `status` — success/partial/error
|
||||||
- `error_message` — описание ошибки (если есть)
|
- `error_message` — описание ошибки (если есть)
|
||||||
|
|
||||||
|
### Чтение истории из Kafka
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose exec kafka /opt/kafka/bin/kafka-console-consumer.sh \
|
||||||
|
--bootstrap-server kafka:29092 \
|
||||||
|
--topic generator_batch_history \
|
||||||
|
--from-beginning
|
||||||
|
```
|
||||||
|
|
||||||
## Тестирование
|
## Тестирование
|
||||||
|
|
||||||
Тесты написаны на **pytest**.
|
Тесты написаны на **pytest**.
|
||||||
@@ -115,9 +114,6 @@ LIMIT 10
|
|||||||
# Через Makefile (рекомендуется)
|
# Через Makefile (рекомендуется)
|
||||||
make generator-test
|
make generator-test
|
||||||
|
|
||||||
# С покрытием
|
|
||||||
make generator-test-cov
|
|
||||||
|
|
||||||
# Вручную через Docker
|
# Вручную через Docker
|
||||||
docker build -t generator:test .
|
docker build -t generator:test .
|
||||||
docker run --rm -v $(PWD):/workspace -w /workspace/generator generator:test pytest tests/ -v
|
docker run --rm -v $(PWD):/workspace -w /workspace/generator generator:test pytest tests/ -v
|
||||||
|
|||||||
+95
-114
@@ -122,12 +122,9 @@ class Config:
|
|||||||
default_factory=lambda: int(os.getenv("GEN_METRICS_PORT", "9109"))
|
default_factory=lambda: int(os.getenv("GEN_METRICS_PORT", "9109"))
|
||||||
)
|
)
|
||||||
|
|
||||||
# ClickHouse для истории batch
|
# Топик для истории batch
|
||||||
clickhouse_host: str = field(
|
history_topic: str = field(
|
||||||
default_factory=lambda: os.getenv("CLICKHOUSE_HOST", "clickhouse")
|
default_factory=lambda: os.getenv("GEN_HISTORY_TOPIC", "generator_batch_history")
|
||||||
)
|
|
||||||
clickhouse_port: int = field(
|
|
||||||
default_factory=lambda: int(os.getenv("CLICKHOUSE_PORT", "9000"))
|
|
||||||
)
|
)
|
||||||
|
|
||||||
def __post_init__(self):
|
def __post_init__(self):
|
||||||
@@ -313,7 +310,7 @@ class EventGenerator:
|
|||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Batch history в ClickHouse
|
# Batch record для истории
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@dataclass
|
@dataclass
|
||||||
class BatchRecord:
|
class BatchRecord:
|
||||||
@@ -330,111 +327,20 @@ class BatchRecord:
|
|||||||
status: str # 'success', 'partial', 'error'
|
status: str # 'success', 'partial', 'error'
|
||||||
error_message: str | None = None
|
error_message: str | None = None
|
||||||
|
|
||||||
|
def to_dict(self) -> dict:
|
||||||
class ClickHouseBatchHistory:
|
"""Конвертирует в словарь для сериализации."""
|
||||||
"""Хранение истории batch в ClickHouse."""
|
return {
|
||||||
|
"batch_id": self.batch_id,
|
||||||
def __init__(self, host: str, port: int):
|
"started_at": self.started_at.isoformat(),
|
||||||
self.host = host
|
"finished_at": self.finished_at.isoformat(),
|
||||||
self.port = port
|
"sent_total": self.sent_total,
|
||||||
self._client = None
|
"sent_browser": self.sent_browser,
|
||||||
self._initialized = False
|
"sent_location": self.sent_location,
|
||||||
|
"sent_device": self.sent_device,
|
||||||
def _get_client(self):
|
"sent_geo": self.sent_geo,
|
||||||
"""Lazy инициализация клиента ClickHouse."""
|
"status": self.status,
|
||||||
if self._client is None:
|
"error_message": self.error_message,
|
||||||
try:
|
}
|
||||||
import clickhouse_connect
|
|
||||||
self._client = clickhouse_connect.get_client(
|
|
||||||
host=self.host,
|
|
||||||
port=self.port,
|
|
||||||
username="default",
|
|
||||||
password="123456",
|
|
||||||
database="default"
|
|
||||||
)
|
|
||||||
self._ensure_table()
|
|
||||||
self._initialized = True
|
|
||||||
except Exception as e:
|
|
||||||
logger.warning(f"Failed to connect to ClickHouse: {e}")
|
|
||||||
self._initialized = False
|
|
||||||
return self._client
|
|
||||||
|
|
||||||
def _ensure_table(self):
|
|
||||||
"""Создаёт таблицу для истории batch если не существует."""
|
|
||||||
try:
|
|
||||||
self._client.command("""
|
|
||||||
CREATE TABLE IF NOT EXISTS meta.generator_batches (
|
|
||||||
batch_id String,
|
|
||||||
started_at DateTime64(6),
|
|
||||||
finished_at DateTime64(6),
|
|
||||||
sent_total Int32,
|
|
||||||
sent_browser Int32,
|
|
||||||
sent_location Int32,
|
|
||||||
sent_device Int32,
|
|
||||||
sent_geo Int32,
|
|
||||||
status String,
|
|
||||||
error_message Nullable(String)
|
|
||||||
) ENGINE = MergeTree()
|
|
||||||
ORDER BY (started_at, batch_id)
|
|
||||||
""")
|
|
||||||
logger.info("ClickHouse table meta.generator_batches ready")
|
|
||||||
except Exception as e:
|
|
||||||
logger.warning(f"Failed to create table: {e}")
|
|
||||||
|
|
||||||
def add(self, record: BatchRecord):
|
|
||||||
"""Добавляет запись в историю."""
|
|
||||||
client = self._get_client()
|
|
||||||
if client is None or not self._initialized:
|
|
||||||
logger.debug("ClickHouse not available, skipping batch history")
|
|
||||||
return
|
|
||||||
|
|
||||||
try:
|
|
||||||
client.insert(
|
|
||||||
"meta.generator_batches",
|
|
||||||
[[
|
|
||||||
record.batch_id,
|
|
||||||
record.started_at,
|
|
||||||
record.finished_at,
|
|
||||||
record.sent_total,
|
|
||||||
record.sent_browser,
|
|
||||||
record.sent_location,
|
|
||||||
record.sent_device,
|
|
||||||
record.sent_geo,
|
|
||||||
record.status,
|
|
||||||
record.error_message
|
|
||||||
]],
|
|
||||||
columns=[
|
|
||||||
"batch_id", "started_at", "finished_at", "sent_total",
|
|
||||||
"sent_browser", "sent_location", "sent_device", "sent_geo",
|
|
||||||
"status", "error_message"
|
|
||||||
]
|
|
||||||
)
|
|
||||||
except Exception as e:
|
|
||||||
logger.warning(f"Failed to write batch history: {e}")
|
|
||||||
|
|
||||||
def get_stats(self) -> dict:
|
|
||||||
"""Возвращает статистику по истории."""
|
|
||||||
client = self._get_client()
|
|
||||||
if client is None or not self._initialized:
|
|
||||||
return {"error": "ClickHouse not available"}
|
|
||||||
|
|
||||||
try:
|
|
||||||
result = client.query("""
|
|
||||||
SELECT
|
|
||||||
count() as total_batches,
|
|
||||||
sumIf(1, status = 'success') as success_count,
|
|
||||||
max(started_at) as last_batch
|
|
||||||
FROM meta.generator_batches
|
|
||||||
""")
|
|
||||||
row = result.result_rows[0]
|
|
||||||
return {
|
|
||||||
"total_batches": row[0],
|
|
||||||
"success_rate": row[1] / row[0] if row[0] > 0 else 0,
|
|
||||||
"last_batch": row[2]
|
|
||||||
}
|
|
||||||
except Exception as e:
|
|
||||||
logger.warning(f"Failed to get stats: {e}")
|
|
||||||
return {"error": str(e)}
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -463,6 +369,70 @@ class InMemoryBatchHistory:
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Kafka history - пишет историю в отдельный топик
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
class KafkaBatchHistory:
|
||||||
|
"""Хранение истории batch в Kafka (отдельный топик)."""
|
||||||
|
|
||||||
|
HISTORY_TOPIC = "generator_batch_history"
|
||||||
|
|
||||||
|
def __init__(self, bootstrap_servers: str):
|
||||||
|
self.bootstrap_servers = bootstrap_servers
|
||||||
|
self.producer = None
|
||||||
|
self._initialized = False
|
||||||
|
self._connect()
|
||||||
|
|
||||||
|
def _connect(self):
|
||||||
|
"""Устанавливает соединение с Kafka."""
|
||||||
|
KafkaProducerCls, KafkaErrorCls = _import_kafka()
|
||||||
|
|
||||||
|
logger.info(f"Connecting to Kafka for history at {self.bootstrap_servers}")
|
||||||
|
try:
|
||||||
|
self.producer = KafkaProducerCls(
|
||||||
|
bootstrap_servers=self.bootstrap_servers,
|
||||||
|
value_serializer=lambda v: json.dumps(v).encode("utf-8"),
|
||||||
|
key_serializer=lambda k: k.encode("utf-8") if k else None,
|
||||||
|
retries=3,
|
||||||
|
retry_backoff_ms=1000,
|
||||||
|
)
|
||||||
|
self._initialized = True
|
||||||
|
logger.info("Connected to Kafka for history successfully")
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Failed to connect to Kafka for history: {e}")
|
||||||
|
self._initialized = False
|
||||||
|
|
||||||
|
def add(self, record: BatchRecord):
|
||||||
|
"""Добавляет запись в историю (топик Kafka)."""
|
||||||
|
if not self._initialized or self.producer is None:
|
||||||
|
logger.debug("Kafka history not available, skipping")
|
||||||
|
return
|
||||||
|
|
||||||
|
try:
|
||||||
|
key = record.batch_id
|
||||||
|
value = record.to_dict()
|
||||||
|
self.producer.send(self.HISTORY_TOPIC, key=key, value=value)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Failed to write batch history to Kafka: {e}")
|
||||||
|
|
||||||
|
def flush(self):
|
||||||
|
"""Сбрасывает буфер."""
|
||||||
|
if self.producer:
|
||||||
|
self.producer.flush()
|
||||||
|
|
||||||
|
def close(self):
|
||||||
|
"""Закрывает соединение."""
|
||||||
|
if self.producer:
|
||||||
|
self.producer.close()
|
||||||
|
|
||||||
|
def get_stats(self) -> dict:
|
||||||
|
"""Возвращает статус подключения."""
|
||||||
|
return {
|
||||||
|
"initialized": self._initialized,
|
||||||
|
"topic": self.HISTORY_TOPIC,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Kafka publisher
|
# Kafka publisher
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -555,8 +525,7 @@ class GeneratorService:
|
|||||||
self.dictionary = EventDictionary.load(config.data_dir)
|
self.dictionary = EventDictionary.load(config.data_dir)
|
||||||
self.generator = EventGenerator(self.dictionary, config)
|
self.generator = EventGenerator(self.dictionary, config)
|
||||||
self.publisher: KafkaPublisher | None = None
|
self.publisher: KafkaPublisher | None = None
|
||||||
# Пробуем ClickHouse, если не доступен - используем in-memory
|
self.history: KafkaBatchHistory | InMemoryBatchHistory | None = None
|
||||||
self.history = ClickHouseBatchHistory(config.clickhouse_host, config.clickhouse_port)
|
|
||||||
self._running = False
|
self._running = False
|
||||||
|
|
||||||
def start(self):
|
def start(self):
|
||||||
@@ -574,7 +543,15 @@ class GeneratorService:
|
|||||||
f"lambda_base={self.config.lambda_base_per_min}/min, "
|
f"lambda_base={self.config.lambda_base_per_min}/min, "
|
||||||
f"jitter={self.config.jitter_pct}%")
|
f"jitter={self.config.jitter_pct}%")
|
||||||
|
|
||||||
|
# Подключаемся к Kafka для публикации событий
|
||||||
self.publisher = KafkaPublisher(self.config.kafka_bootstrap_servers)
|
self.publisher = KafkaPublisher(self.config.kafka_bootstrap_servers)
|
||||||
|
|
||||||
|
# Инициализируем историю (Kafka с fallback на in-memory)
|
||||||
|
self.history = KafkaBatchHistory(self.config.kafka_bootstrap_servers)
|
||||||
|
if not self.history._initialized:
|
||||||
|
logger.warning("Falling back to InMemoryBatchHistory")
|
||||||
|
self.history = InMemoryBatchHistory()
|
||||||
|
|
||||||
self._running = True
|
self._running = True
|
||||||
|
|
||||||
try:
|
try:
|
||||||
@@ -590,6 +567,8 @@ class GeneratorService:
|
|||||||
self._running = False
|
self._running = False
|
||||||
if self.publisher:
|
if self.publisher:
|
||||||
self.publisher.close()
|
self.publisher.close()
|
||||||
|
if isinstance(self.history, KafkaBatchHistory):
|
||||||
|
self.history.close()
|
||||||
|
|
||||||
def _main_loop(self):
|
def _main_loop(self):
|
||||||
"""Основной цикл тиков."""
|
"""Основной цикл тиков."""
|
||||||
@@ -627,6 +606,8 @@ class GeneratorService:
|
|||||||
total_errors += errors
|
total_errors += errors
|
||||||
|
|
||||||
self.publisher.flush()
|
self.publisher.flush()
|
||||||
|
if isinstance(self.history, KafkaBatchHistory):
|
||||||
|
self.history.flush()
|
||||||
pub_duration = time.time() - pub_start
|
pub_duration = time.time() - pub_start
|
||||||
|
|
||||||
# Определяем статус
|
# Определяем статус
|
||||||
|
|||||||
@@ -1,15 +1,11 @@
|
|||||||
# Kafka клиент
|
# Kafka клиент
|
||||||
kafka-python==2.0.5
|
kafka-python==2.0.5
|
||||||
|
|
||||||
# ClickHouse драйвер (для истории batch)
|
|
||||||
clickhouse-connect==0.8.0
|
|
||||||
|
|
||||||
# Prometheus метрики
|
# Prometheus метрики
|
||||||
prometheus-client==0.21.1
|
prometheus-client==0.21.1
|
||||||
|
|
||||||
# Тестирование
|
# Тестирование
|
||||||
pytest==8.3.4
|
pytest==8.3.4
|
||||||
pytest-asyncio==0.25.3
|
|
||||||
|
|
||||||
# Утилиты
|
# Утилиты
|
||||||
python-json-logger==2.0.7
|
python-json-logger==2.0.7
|
||||||
|
|||||||
@@ -38,8 +38,7 @@ def base_config(data_dir):
|
|||||||
seed=42,
|
seed=42,
|
||||||
enabled=True,
|
enabled=True,
|
||||||
metrics_port=9109,
|
metrics_port=9109,
|
||||||
clickhouse_host="localhost",
|
history_topic="generator_batch_history",
|
||||||
clickhouse_port=9000,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -58,8 +58,7 @@ class TestConfigDefaults:
|
|||||||
seed=None,
|
seed=None,
|
||||||
enabled=True,
|
enabled=True,
|
||||||
metrics_port=9109,
|
metrics_port=9109,
|
||||||
clickhouse_host="localhost",
|
history_topic="generator_batch_history",
|
||||||
clickhouse_port=9000,
|
|
||||||
)
|
)
|
||||||
assert config.tick_seconds == 5
|
assert config.tick_seconds == 5
|
||||||
finally:
|
finally:
|
||||||
|
|||||||
@@ -0,0 +1,263 @@
|
|||||||
|
"""
|
||||||
|
Тесты для KafkaBatchHistory.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from unittest.mock import MagicMock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from generator import BatchRecord, KafkaBatchHistory
|
||||||
|
|
||||||
|
|
||||||
|
class TestBatchRecordSerialization:
|
||||||
|
"""Тесты сериализации BatchRecord."""
|
||||||
|
|
||||||
|
def test_to_dict_serializes_all_fields(self):
|
||||||
|
"""to_dict сериализует все поля."""
|
||||||
|
now = datetime(2024, 1, 15, 10, 30, 0, tzinfo=timezone.utc)
|
||||||
|
record = BatchRecord(
|
||||||
|
batch_id="abc123",
|
||||||
|
started_at=now,
|
||||||
|
finished_at=now,
|
||||||
|
sent_total=100,
|
||||||
|
sent_browser=25,
|
||||||
|
sent_location=25,
|
||||||
|
sent_device=25,
|
||||||
|
sent_geo=25,
|
||||||
|
status="success",
|
||||||
|
error_message=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
data = record.to_dict()
|
||||||
|
|
||||||
|
assert data["batch_id"] == "abc123"
|
||||||
|
assert data["started_at"] == "2024-01-15T10:30:00+00:00"
|
||||||
|
assert data["finished_at"] == "2024-01-15T10:30:00+00:00"
|
||||||
|
assert data["sent_total"] == 100
|
||||||
|
assert data["sent_browser"] == 25
|
||||||
|
assert data["sent_location"] == 25
|
||||||
|
assert data["sent_device"] == 25
|
||||||
|
assert data["sent_geo"] == 25
|
||||||
|
assert data["status"] == "success"
|
||||||
|
assert data["error_message"] is None
|
||||||
|
|
||||||
|
def test_to_dict_with_error_message(self):
|
||||||
|
"""to_dict сериализует error_message если есть."""
|
||||||
|
now = datetime.now(timezone.utc)
|
||||||
|
record = BatchRecord(
|
||||||
|
batch_id="err456",
|
||||||
|
started_at=now,
|
||||||
|
finished_at=now,
|
||||||
|
sent_total=0,
|
||||||
|
sent_browser=0,
|
||||||
|
sent_location=0,
|
||||||
|
sent_device=0,
|
||||||
|
sent_geo=0,
|
||||||
|
status="error",
|
||||||
|
error_message="Connection failed",
|
||||||
|
)
|
||||||
|
|
||||||
|
data = record.to_dict()
|
||||||
|
|
||||||
|
assert data["error_message"] == "Connection failed"
|
||||||
|
|
||||||
|
def test_isoformat_includes_timezone(self):
|
||||||
|
"""ISO формат включает timezone."""
|
||||||
|
now = datetime.now(timezone.utc)
|
||||||
|
record = BatchRecord(
|
||||||
|
batch_id="tz789",
|
||||||
|
started_at=now,
|
||||||
|
finished_at=now,
|
||||||
|
sent_total=50,
|
||||||
|
sent_browser=12,
|
||||||
|
sent_location=13,
|
||||||
|
sent_device=12,
|
||||||
|
sent_geo=13,
|
||||||
|
status="partial",
|
||||||
|
error_message="Some errors",
|
||||||
|
)
|
||||||
|
|
||||||
|
data = record.to_dict()
|
||||||
|
|
||||||
|
# Должен содержать +00:00 или Z
|
||||||
|
assert "+" in data["started_at"] or "Z" in data["started_at"]
|
||||||
|
|
||||||
|
|
||||||
|
class TestKafkaBatchHistory:
|
||||||
|
"""Тесты KafkaBatchHistory."""
|
||||||
|
|
||||||
|
def test_history_topic_constant(self):
|
||||||
|
"""Константа топика истории."""
|
||||||
|
assert KafkaBatchHistory.HISTORY_TOPIC == "generator_batch_history"
|
||||||
|
|
||||||
|
@patch("generator._import_kafka")
|
||||||
|
def test_init_connects_to_kafka(self, mock_import):
|
||||||
|
"""Инициализация подключается к Kafka."""
|
||||||
|
mock_producer_class = MagicMock()
|
||||||
|
mock_import.return_value = (mock_producer_class, Exception)
|
||||||
|
|
||||||
|
history = KafkaBatchHistory("localhost:9092")
|
||||||
|
|
||||||
|
assert history._initialized is True
|
||||||
|
assert history.bootstrap_servers == "localhost:9092"
|
||||||
|
mock_producer_class.assert_called_once()
|
||||||
|
|
||||||
|
@patch("generator._import_kafka")
|
||||||
|
def test_init_handles_connection_error(self, mock_import):
|
||||||
|
"""Инициализация обрабатывает ошибку подключения."""
|
||||||
|
# Симулируем ошибку при создании producer
|
||||||
|
mock_producer_class = MagicMock(side_effect=Exception("Connection failed"))
|
||||||
|
mock_import.return_value = (mock_producer_class, Exception)
|
||||||
|
|
||||||
|
history = KafkaBatchHistory("localhost:9092")
|
||||||
|
|
||||||
|
assert history._initialized is False
|
||||||
|
assert history.producer is None
|
||||||
|
|
||||||
|
@patch("generator._import_kafka")
|
||||||
|
def test_add_sends_to_kafka(self, mock_import):
|
||||||
|
"""add отправляет сообщение в Kafka."""
|
||||||
|
mock_producer = MagicMock()
|
||||||
|
mock_producer_class = MagicMock(return_value=mock_producer)
|
||||||
|
mock_import.return_value = (mock_producer_class, Exception)
|
||||||
|
|
||||||
|
history = KafkaBatchHistory("localhost:9092")
|
||||||
|
now = datetime.now(timezone.utc)
|
||||||
|
record = BatchRecord(
|
||||||
|
batch_id="test123",
|
||||||
|
started_at=now,
|
||||||
|
finished_at=now,
|
||||||
|
sent_total=100,
|
||||||
|
sent_browser=25,
|
||||||
|
sent_location=25,
|
||||||
|
sent_device=25,
|
||||||
|
sent_geo=25,
|
||||||
|
status="success",
|
||||||
|
error_message=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
history.add(record)
|
||||||
|
|
||||||
|
mock_producer.send.assert_called_once()
|
||||||
|
call_args = mock_producer.send.call_args
|
||||||
|
assert call_args[0][0] == "generator_batch_history"
|
||||||
|
assert call_args[1]["key"] == "test123"
|
||||||
|
assert "value" in call_args[1]
|
||||||
|
|
||||||
|
@patch("generator._import_kafka")
|
||||||
|
def test_add_skips_if_not_initialized(self, mock_import):
|
||||||
|
"""add пропускает если не инициализирован."""
|
||||||
|
mock_producer_class = MagicMock(side_effect=Exception("Connection failed"))
|
||||||
|
mock_import.return_value = (mock_producer_class, Exception)
|
||||||
|
|
||||||
|
history = KafkaBatchHistory("localhost:9092")
|
||||||
|
now = datetime.now(timezone.utc)
|
||||||
|
record = BatchRecord(
|
||||||
|
batch_id="skip456",
|
||||||
|
started_at=now,
|
||||||
|
finished_at=now,
|
||||||
|
sent_total=0,
|
||||||
|
sent_browser=0,
|
||||||
|
sent_location=0,
|
||||||
|
sent_device=0,
|
||||||
|
sent_geo=0,
|
||||||
|
status="error",
|
||||||
|
error_message="Test",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Не должно упасть
|
||||||
|
history.add(record)
|
||||||
|
|
||||||
|
@patch("generator._import_kafka")
|
||||||
|
def test_add_handles_send_error(self, mock_import):
|
||||||
|
"""add обрабатывает ошибку отправки."""
|
||||||
|
mock_producer = MagicMock()
|
||||||
|
mock_producer.send.side_effect = Exception("Send failed")
|
||||||
|
mock_producer_class = MagicMock(return_value=mock_producer)
|
||||||
|
mock_import.return_value = (mock_producer_class, Exception)
|
||||||
|
|
||||||
|
history = KafkaBatchHistory("localhost:9092")
|
||||||
|
now = datetime.now(timezone.utc)
|
||||||
|
record = BatchRecord(
|
||||||
|
batch_id="fail789",
|
||||||
|
started_at=now,
|
||||||
|
finished_at=now,
|
||||||
|
sent_total=50,
|
||||||
|
sent_browser=12,
|
||||||
|
sent_location=13,
|
||||||
|
sent_device=12,
|
||||||
|
sent_geo=13,
|
||||||
|
status="success",
|
||||||
|
error_message=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Не должно упасть
|
||||||
|
history.add(record)
|
||||||
|
|
||||||
|
@patch("generator._import_kafka")
|
||||||
|
def test_flush_calls_producer_flush(self, mock_import):
|
||||||
|
"""flush вызывает flush у producer."""
|
||||||
|
mock_producer = MagicMock()
|
||||||
|
mock_producer_class = MagicMock(return_value=mock_producer)
|
||||||
|
mock_import.return_value = (mock_producer_class, Exception)
|
||||||
|
|
||||||
|
history = KafkaBatchHistory("localhost:9092")
|
||||||
|
history.flush()
|
||||||
|
|
||||||
|
mock_producer.flush.assert_called_once()
|
||||||
|
|
||||||
|
@patch("generator._import_kafka")
|
||||||
|
def test_flush_noop_if_not_initialized(self, mock_import):
|
||||||
|
"""flush ничего не делает если не инициализирован."""
|
||||||
|
mock_producer_class = MagicMock(side_effect=Exception("Connection failed"))
|
||||||
|
mock_import.return_value = (mock_producer_class, Exception)
|
||||||
|
|
||||||
|
history = KafkaBatchHistory("localhost:9092")
|
||||||
|
# Не должно упасть
|
||||||
|
history.flush()
|
||||||
|
|
||||||
|
@patch("generator._import_kafka")
|
||||||
|
def test_close_calls_producer_close(self, mock_import):
|
||||||
|
"""close вызывает close у producer."""
|
||||||
|
mock_producer = MagicMock()
|
||||||
|
mock_producer_class = MagicMock(return_value=mock_producer)
|
||||||
|
mock_import.return_value = (mock_producer_class, Exception)
|
||||||
|
|
||||||
|
history = KafkaBatchHistory("localhost:9092")
|
||||||
|
history.close()
|
||||||
|
|
||||||
|
mock_producer.close.assert_called_once()
|
||||||
|
|
||||||
|
@patch("generator._import_kafka")
|
||||||
|
def test_close_noop_if_not_initialized(self, mock_import):
|
||||||
|
"""close ничего не делает если не инициализирован."""
|
||||||
|
mock_producer_class = MagicMock(side_effect=Exception("Connection failed"))
|
||||||
|
mock_import.return_value = (mock_producer_class, Exception)
|
||||||
|
|
||||||
|
history = KafkaBatchHistory("localhost:9092")
|
||||||
|
# Не должно упасть
|
||||||
|
history.close()
|
||||||
|
|
||||||
|
@patch("generator._import_kafka")
|
||||||
|
def test_get_stats_returns_status(self, mock_import):
|
||||||
|
"""get_stats возвращает статус инициализации."""
|
||||||
|
mock_producer_class = MagicMock()
|
||||||
|
mock_import.return_value = (mock_producer_class, Exception)
|
||||||
|
|
||||||
|
history = KafkaBatchHistory("localhost:9092")
|
||||||
|
stats = history.get_stats()
|
||||||
|
|
||||||
|
assert stats["initialized"] is True
|
||||||
|
assert stats["topic"] == "generator_batch_history"
|
||||||
|
|
||||||
|
@patch("generator._import_kafka")
|
||||||
|
def test_get_stats_handles_not_initialized(self, mock_import):
|
||||||
|
"""get_stats корректен при неинициализированном состоянии."""
|
||||||
|
mock_producer_class = MagicMock(side_effect=Exception("Connection failed"))
|
||||||
|
mock_import.return_value = (mock_producer_class, Exception)
|
||||||
|
|
||||||
|
history = KafkaBatchHistory("localhost:9092")
|
||||||
|
stats = history.get_stats()
|
||||||
|
|
||||||
|
assert stats["initialized"] is False
|
||||||
|
assert stats["topic"] == "generator_batch_history"
|
||||||
@@ -0,0 +1,107 @@
|
|||||||
|
"""
|
||||||
|
Тесты для GeneratorService и интеграционных сценариев.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from unittest.mock import MagicMock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from generator import (
|
||||||
|
Config, EventDictionary, GeneratorService,
|
||||||
|
InMemoryBatchHistory, KafkaBatchHistory
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class TestConfigHistoryTopic:
|
||||||
|
"""Тесты для history_topic в конфигурации."""
|
||||||
|
|
||||||
|
def test_default_history_topic(self, base_config):
|
||||||
|
"""По умолчанию топик истории."""
|
||||||
|
assert base_config.history_topic == "generator_batch_history"
|
||||||
|
|
||||||
|
def test_custom_history_topic(self, base_config):
|
||||||
|
"""Кастомный топик истории."""
|
||||||
|
from dataclasses import replace
|
||||||
|
custom_config = replace(base_config, history_topic="custom_history")
|
||||||
|
assert custom_config.history_topic == "custom_history"
|
||||||
|
|
||||||
|
|
||||||
|
class TestBatchRecordWithDictConversion:
|
||||||
|
"""Тесты конвертации BatchRecord в dict."""
|
||||||
|
|
||||||
|
def test_dict_contains_all_batch_info(self):
|
||||||
|
"""Словарь содержит всю информацию о батче."""
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from generator import BatchRecord
|
||||||
|
|
||||||
|
started = datetime(2024, 6, 15, 12, 0, 0, tzinfo=timezone.utc)
|
||||||
|
finished = datetime(2024, 6, 15, 12, 0, 5, tzinfo=timezone.utc)
|
||||||
|
|
||||||
|
record = BatchRecord(
|
||||||
|
batch_id="batch_001",
|
||||||
|
started_at=started,
|
||||||
|
finished_at=finished,
|
||||||
|
sent_total=400,
|
||||||
|
sent_browser=100,
|
||||||
|
sent_location=100,
|
||||||
|
sent_device=100,
|
||||||
|
sent_geo=100,
|
||||||
|
status="success",
|
||||||
|
error_message=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
data = record.to_dict()
|
||||||
|
|
||||||
|
# Проверяем структуру JSON
|
||||||
|
assert isinstance(data, dict)
|
||||||
|
assert data["batch_id"] == "batch_001"
|
||||||
|
assert data["sent_total"] == 400
|
||||||
|
assert data["status"] == "success"
|
||||||
|
assert "started_at" in data
|
||||||
|
assert "finished_at" in data
|
||||||
|
|
||||||
|
# Проверяем что можно сериализовать в JSON
|
||||||
|
import json
|
||||||
|
json_str = json.dumps(data)
|
||||||
|
assert isinstance(json_str, str)
|
||||||
|
|
||||||
|
# Проверяем что можно десериализовать
|
||||||
|
restored = json.loads(json_str)
|
||||||
|
assert restored["batch_id"] == "batch_001"
|
||||||
|
|
||||||
|
|
||||||
|
class TestGeneratorServiceInit:
|
||||||
|
"""Тесты инициализации GeneratorService."""
|
||||||
|
|
||||||
|
def test_service_initializes_dictionary(self, base_config):
|
||||||
|
"""Service загружает словарь при инициализации."""
|
||||||
|
service = GeneratorService(base_config)
|
||||||
|
|
||||||
|
assert service.dictionary is not None
|
||||||
|
assert len(service.dictionary.browser_events) == 1000
|
||||||
|
assert service.generator is not None
|
||||||
|
assert service.config == base_config
|
||||||
|
|
||||||
|
def test_service_history_is_none_before_start(self, base_config):
|
||||||
|
"""История None до вызова start."""
|
||||||
|
service = GeneratorService(base_config)
|
||||||
|
|
||||||
|
# История и publisher инициализируются в start()
|
||||||
|
assert service.history is None
|
||||||
|
assert service.publisher is None
|
||||||
|
|
||||||
|
|
||||||
|
class TestGeneratorServiceDisabled:
|
||||||
|
"""Тесты отключенного генератора."""
|
||||||
|
|
||||||
|
def test_disabled_generator_logs_warning(self, base_config, caplog):
|
||||||
|
"""Отключенный генератор логирует warning."""
|
||||||
|
from dataclasses import replace
|
||||||
|
import logging
|
||||||
|
|
||||||
|
disabled_config = replace(base_config, enabled=False)
|
||||||
|
service = GeneratorService(disabled_config)
|
||||||
|
|
||||||
|
with caplog.at_level(logging.WARNING):
|
||||||
|
service.start()
|
||||||
|
|
||||||
|
assert "disabled" in caplog.text.lower() or "GEN_ENABLED" in caplog.text
|
||||||
Reference in New Issue
Block a user