ddmitry/clickstream-ch-kafka-superset-demo

Files

T

ddadmin f4971e94ca docs(scratch): handoff — триаж #5 закрыт, ревью APPROVED, идёт приёмка

- Зачем:
  - зафиксировать состояние конвейера #5 перед долгой живой приёмкой,
    чтобы новая сессия продолжила без потери контекста.
- Что:
  - в handoff добавлена дельта 23:10: обе находки FIXED (фрагменты ID
    с SHA-256-цепочкой), перепроверка линией B — APPROVED;
  - обновлено состояние стенда: им владеет сценарий приёмки;
  - в .scratch/issue5-run добавлены свежие отчёты, перепроверка и
    сценарий приёмки acceptance-issue5.sh.
- Проверка:
  - git show --stat; лог приёмки — до строки SCRIPT_EXIT_CODE=.

2026-07-22 23:11:44 +03:00

4.5 KiB

Raw Blame History

Mandate: issue #5 — incremental manifest counters (next-day without full Kafka reread)

Repo: /home/dementev/sources/clickstream-ch-kafka-superset-demo Branch: feature/mentee-path (work in place, do NOT commit). Python via uv. Docs/comments/code text that stays in repo: clear Russian. This exchange file and your report: English.

Source of truth (read first, in this order)

Issue #5 criteria — reproduced below (GitHub CLI may not work in your sandbox).
Spec: docs/specs/2026-07-19-mentee-path-redesign.md — «Решения» item 4, «Чего здесь не делаем».
.scratch/hitl-findings.md, finding F9 — the code-level analysis of WHERE the full Kafka reread happens. This is your map into the code.

Goal

The scheduled next-day stops getting more expensive as the world ages. Today every next-day run rereads the ENTIRE Kafka history to rebuild manifest counters: cost grows ~quadratically over days (measured: day 2 — 638 s), memory grows linearly (OOM risk).

Acceptance criteria (from issue #5)

Cumulative counter state (sums, uid/click_id sets, rolling checksum) is stored in state/manifest.
A new day is append-only: NO full reread of Kafka history remains in the next-day path.
Generation+manifest task on day N takes ~the same time as day 1.
make generated-history-chain-check green on day seams.
make test and make lint green.

Architecture guidance (coordinator's preferred direction)

Open question you must resolve and JUSTIFY in the report: how cumulative counters get seeded when the world comes from the reference artifact in git (data/startup_history/reference-world.json.xz), whose stored state predates the new counters.

Preferred direction: seed the cumulative counters wherever the event stream already flows through the generator — i.e. during import (artifact is fully read to load Kafka anyway) and during backfill (events are produced by us). Then next-day only updates counters incrementally. The artifact format in git should NOT need to change; only the locally-stored state/manifest gains the cumulative section. If state lacks the cumulative section (old local state), fail with a clear Russian error telling the user to re-run import — do not silently fall back to a full reread.

If you find this infeasible or substantially worse than an alternative, STOP and write your reasoning + alternative to the report file, then return with a question instead of implementing something else.

Determinism invariants (must hold):

single-threaded PRNG usage unchanged; do NOT parallelize generation;
for the same world, manifest numbers (events, visits, users, checksum) must be identical whether computed the old way (full read) or via the new incremental path — the rolling checksum design must guarantee this (e.g. order-independent combine or strictly deterministic order);
check operation semantics preserved: it must still be able to verify ClickHouse against the manifest.

Boundaries

Do NOT parallelize generation (breaks determinism — single PRNG stream, visits crossing day boundaries).
Do NOT do incremental ETL: world_next_day keeps calling etl_pipeline full_refresh (tech debt tracked separately in #8).
Do NOT touch retention beyond the incremental counters.
Do NOT change the reference artifact file in git.
Do NOT touch docs/course/.
Do NOT commit.
Update docs affected by the state/manifest change in the same change (at minimum docs/runbooks/startup-history.md if state format/semantics visible there; check docs/ARCHITECTURE.md, docs/OPERATIONS.md).

Verification (run before reporting done)

make test and make lint green (exit codes in report).
make generated-history-chain-check if runnable in your environment; if it needs the live stand and you cannot run it, say so explicitly — the coordinator will run it during live acceptance.
Unit tests for the new counter state: seeding on import/backfill, incremental update on next-day, equality of incremental vs full-recompute manifest numbers on a small fixture (this equality test is REQUIRED).
Follow repo data rules: tests use small slices, not full jsonl files.

Report

Write full report to /tmp/claude-1000/-home-dementev-sources-clickstream-ch-kafka-superset-demo/0836e970-ee07-4f86-a61f-291f32da525a/scratchpad/report-issue5.md: design decision on seeding + why, what changed (file list), how the rolling checksum stays equal to the full recompute, test/lint exit codes and counters, anything NOT done or uncertain. Return only a short summary + the report path.

4.5 KiB Raw Blame History