docs(scratch): handoff — триаж #5 закрыт, ревью APPROVED, идёт приёмка

- Зачем:
  - зафиксировать состояние конвейера #5 перед долгой живой приёмкой,
    чтобы новая сессия продолжила без потери контекста.
- Что:
  - в handoff добавлена дельта 23:10: обе находки FIXED (фрагменты ID
    с SHA-256-цепочкой), перепроверка линией B — APPROVED;
  - обновлено состояние стенда: им владеет сценарий приёмки;
  - в .scratch/issue5-run добавлены свежие отчёты, перепроверка и
    сценарий приёмки acceptance-issue5.sh.
- Проверка:
  - git show --stat; лог приёмки — до строки SCRIPT_EXIT_CODE=.
This commit is contained in:
2026-07-22 23:11:44 +03:00
parent ceb30a95c0
commit f4971e94ca
10 changed files with 803 additions and 3 deletions
+94
View File
@@ -0,0 +1,94 @@
# Mandate: issue #5 — incremental manifest counters (next-day without full Kafka reread)
Repo: /home/dementev/sources/clickstream-ch-kafka-superset-demo
Branch: feature/mentee-path (work in place, do NOT commit).
Python via uv. Docs/comments/code text that stays in repo: clear Russian.
This exchange file and your report: English.
## Source of truth (read first, in this order)
1. Issue #5 criteria — reproduced below (GitHub CLI may not work in your sandbox).
2. Spec: `docs/specs/2026-07-19-mentee-path-redesign.md` — «Решения» item 4,
«Чего здесь не делаем».
3. `.scratch/hitl-findings.md`, finding F9 — the code-level analysis of WHERE
the full Kafka reread happens. This is your map into the code.
## Goal
The scheduled `next-day` stops getting more expensive as the world ages.
Today every next-day run rereads the ENTIRE Kafka history to rebuild manifest
counters: cost grows ~quadratically over days (measured: day 2 — 638 s),
memory grows linearly (OOM risk).
## Acceptance criteria (from issue #5)
- Cumulative counter state (sums, uid/click_id sets, rolling checksum) is
stored in state/manifest.
- A new day is append-only: NO full reread of Kafka history remains in the
next-day path.
- Generation+manifest task on day N takes ~the same time as day 1.
- `make generated-history-chain-check` green on day seams.
- `make test` and `make lint` green.
## Architecture guidance (coordinator's preferred direction)
Open question you must resolve and JUSTIFY in the report: how cumulative
counters get seeded when the world comes from the reference artifact in git
(`data/startup_history/reference-world.json.xz`), whose stored state predates
the new counters.
Preferred direction: seed the cumulative counters wherever the event stream
already flows through the generator — i.e. during `import` (artifact is fully
read to load Kafka anyway) and during `backfill` (events are produced by us).
Then `next-day` only updates counters incrementally. The artifact format in
git should NOT need to change; only the locally-stored state/manifest gains
the cumulative section. If state lacks the cumulative section (old local
state), fail with a clear Russian error telling the user to re-run import —
do not silently fall back to a full reread.
If you find this infeasible or substantially worse than an alternative, STOP
and write your reasoning + alternative to the report file, then return with a
question instead of implementing something else.
Determinism invariants (must hold):
- single-threaded PRNG usage unchanged; do NOT parallelize generation;
- for the same world, manifest numbers (events, visits, users, checksum)
must be identical whether computed the old way (full read) or via the new
incremental path — the rolling checksum design must guarantee this
(e.g. order-independent combine or strictly deterministic order);
- `check` operation semantics preserved: it must still be able to verify
ClickHouse against the manifest.
## Boundaries
- Do NOT parallelize generation (breaks determinism — single PRNG stream,
visits crossing day boundaries).
- Do NOT do incremental ETL: world_next_day keeps calling etl_pipeline
full_refresh (tech debt tracked separately in #8).
- Do NOT touch retention beyond the incremental counters.
- Do NOT change the reference artifact file in git.
- Do NOT touch `docs/course/`.
- Do NOT commit.
- Update docs affected by the state/manifest change in the same change
(at minimum `docs/runbooks/startup-history.md` if state format/semantics
visible there; check `docs/ARCHITECTURE.md`, `docs/OPERATIONS.md`).
## Verification (run before reporting done)
- `make test` and `make lint` green (exit codes in report).
- `make generated-history-chain-check` if runnable in your environment;
if it needs the live stand and you cannot run it, say so explicitly —
the coordinator will run it during live acceptance.
- Unit tests for the new counter state: seeding on import/backfill,
incremental update on next-day, equality of incremental vs full-recompute
manifest numbers on a small fixture (this equality test is REQUIRED).
- Follow repo data rules: tests use small slices, not full jsonl files.
## Report
Write full report to
`/tmp/claude-1000/-home-dementev-sources-clickstream-ch-kafka-superset-demo/0836e970-ee07-4f86-a61f-291f32da525a/scratchpad/report-issue5.md`:
design decision on seeding + why, what changed (file list), how the rolling
checksum stays equal to the full recompute, test/lint exit codes and
counters, anything NOT done or uncertain. Return only a short summary + the
report path.