docs(scratch): handoff — триаж #5 закрыт, ревью APPROVED, идёт приёмка
- Зачем: - зафиксировать состояние конвейера #5 перед долгой живой приёмкой, чтобы новая сессия продолжила без потери контекста. - Что: - в handoff добавлена дельта 23:10: обе находки FIXED (фрагменты ID с SHA-256-цепочкой), перепроверка линией B — APPROVED; - обновлено состояние стенда: им владеет сценарий приёмки; - в .scratch/issue5-run добавлены свежие отчёты, перепроверка и сценарий приёмки acceptance-issue5.sh. - Проверка: - git show --stat; лог приёмки — до строки SCRIPT_EXIT_CODE=.
This commit is contained in:
@@ -0,0 +1,94 @@
|
||||
# Mandate: issue #5 — incremental manifest counters (next-day without full Kafka reread)
|
||||
|
||||
Repo: /home/dementev/sources/clickstream-ch-kafka-superset-demo
|
||||
Branch: feature/mentee-path (work in place, do NOT commit).
|
||||
Python via uv. Docs/comments/code text that stays in repo: clear Russian.
|
||||
This exchange file and your report: English.
|
||||
|
||||
## Source of truth (read first, in this order)
|
||||
|
||||
1. Issue #5 criteria — reproduced below (GitHub CLI may not work in your sandbox).
|
||||
2. Spec: `docs/specs/2026-07-19-mentee-path-redesign.md` — «Решения» item 4,
|
||||
«Чего здесь не делаем».
|
||||
3. `.scratch/hitl-findings.md`, finding F9 — the code-level analysis of WHERE
|
||||
the full Kafka reread happens. This is your map into the code.
|
||||
|
||||
## Goal
|
||||
|
||||
The scheduled `next-day` stops getting more expensive as the world ages.
|
||||
Today every next-day run rereads the ENTIRE Kafka history to rebuild manifest
|
||||
counters: cost grows ~quadratically over days (measured: day 2 — 638 s),
|
||||
memory grows linearly (OOM risk).
|
||||
|
||||
## Acceptance criteria (from issue #5)
|
||||
|
||||
- Cumulative counter state (sums, uid/click_id sets, rolling checksum) is
|
||||
stored in state/manifest.
|
||||
- A new day is append-only: NO full reread of Kafka history remains in the
|
||||
next-day path.
|
||||
- Generation+manifest task on day N takes ~the same time as day 1.
|
||||
- `make generated-history-chain-check` green on day seams.
|
||||
- `make test` and `make lint` green.
|
||||
|
||||
## Architecture guidance (coordinator's preferred direction)
|
||||
|
||||
Open question you must resolve and JUSTIFY in the report: how cumulative
|
||||
counters get seeded when the world comes from the reference artifact in git
|
||||
(`data/startup_history/reference-world.json.xz`), whose stored state predates
|
||||
the new counters.
|
||||
|
||||
Preferred direction: seed the cumulative counters wherever the event stream
|
||||
already flows through the generator — i.e. during `import` (artifact is fully
|
||||
read to load Kafka anyway) and during `backfill` (events are produced by us).
|
||||
Then `next-day` only updates counters incrementally. The artifact format in
|
||||
git should NOT need to change; only the locally-stored state/manifest gains
|
||||
the cumulative section. If state lacks the cumulative section (old local
|
||||
state), fail with a clear Russian error telling the user to re-run import —
|
||||
do not silently fall back to a full reread.
|
||||
|
||||
If you find this infeasible or substantially worse than an alternative, STOP
|
||||
and write your reasoning + alternative to the report file, then return with a
|
||||
question instead of implementing something else.
|
||||
|
||||
Determinism invariants (must hold):
|
||||
- single-threaded PRNG usage unchanged; do NOT parallelize generation;
|
||||
- for the same world, manifest numbers (events, visits, users, checksum)
|
||||
must be identical whether computed the old way (full read) or via the new
|
||||
incremental path — the rolling checksum design must guarantee this
|
||||
(e.g. order-independent combine or strictly deterministic order);
|
||||
- `check` operation semantics preserved: it must still be able to verify
|
||||
ClickHouse against the manifest.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Do NOT parallelize generation (breaks determinism — single PRNG stream,
|
||||
visits crossing day boundaries).
|
||||
- Do NOT do incremental ETL: world_next_day keeps calling etl_pipeline
|
||||
full_refresh (tech debt tracked separately in #8).
|
||||
- Do NOT touch retention beyond the incremental counters.
|
||||
- Do NOT change the reference artifact file in git.
|
||||
- Do NOT touch `docs/course/`.
|
||||
- Do NOT commit.
|
||||
- Update docs affected by the state/manifest change in the same change
|
||||
(at minimum `docs/runbooks/startup-history.md` if state format/semantics
|
||||
visible there; check `docs/ARCHITECTURE.md`, `docs/OPERATIONS.md`).
|
||||
|
||||
## Verification (run before reporting done)
|
||||
|
||||
- `make test` and `make lint` green (exit codes in report).
|
||||
- `make generated-history-chain-check` if runnable in your environment;
|
||||
if it needs the live stand and you cannot run it, say so explicitly —
|
||||
the coordinator will run it during live acceptance.
|
||||
- Unit tests for the new counter state: seeding on import/backfill,
|
||||
incremental update on next-day, equality of incremental vs full-recompute
|
||||
manifest numbers on a small fixture (this equality test is REQUIRED).
|
||||
- Follow repo data rules: tests use small slices, not full jsonl files.
|
||||
|
||||
## Report
|
||||
|
||||
Write full report to
|
||||
`/tmp/claude-1000/-home-dementev-sources-clickstream-ch-kafka-superset-demo/0836e970-ee07-4f86-a61f-291f32da525a/scratchpad/report-issue5.md`:
|
||||
design decision on seeding + why, what changed (file list), how the rolling
|
||||
checksum stays equal to the full recompute, test/lint exit codes and
|
||||
counters, anything NOT done or uncertain. Return only a short summary + the
|
||||
report path.
|
||||
Reference in New Issue
Block a user