# Mandate: issue #5 — incremental manifest counters (next-day without full Kafka reread)

Repo: /home/dementev/sources/clickstream-ch-kafka-superset-demo
Branch: feature/mentee-path (work in place, do NOT commit).
Python via uv. Docs/comments/code text that stays in repo: clear Russian.
This exchange file and your report: English.

## Source of truth (read first, in this order)

1. Issue #5 criteria — reproduced below (GitHub CLI may not work in your sandbox).
2. Spec: `docs/specs/2026-07-19-mentee-path-redesign.md` — «Решения» item 4,
   «Чего здесь не делаем».
3. `.scratch/hitl-findings.md`, finding F9 — the code-level analysis of WHERE
   the full Kafka reread happens. This is your map into the code.

## Goal

The scheduled `next-day` stops getting more expensive as the world ages.
Today every next-day run rereads the ENTIRE Kafka history to rebuild manifest
counters: cost grows ~quadratically over days (measured: day 2 — 638 s),
memory grows linearly (OOM risk).

## Acceptance criteria (from issue #5)

- Cumulative counter state (sums, uid/click_id sets, rolling checksum) is
  stored in state/manifest.
- A new day is append-only: NO full reread of Kafka history remains in the
  next-day path.
- Generation+manifest task on day N takes ~the same time as day 1.
- `make generated-history-chain-check` green on day seams.
- `make test` and `make lint` green.

## Architecture guidance (coordinator's preferred direction)

Open question you must resolve and JUSTIFY in the report: how cumulative
counters get seeded when the world comes from the reference artifact in git
(`data/startup_history/reference-world.json.xz`), whose stored state predates
the new counters.

Preferred direction: seed the cumulative counters wherever the event stream
already flows through the generator — i.e. during `import` (artifact is fully
read to load Kafka anyway) and during `backfill` (events are produced by us).
Then `next-day` only updates counters incrementally. The artifact format in
git should NOT need to change; only the locally-stored state/manifest gains
the cumulative section. If state lacks the cumulative section (old local
state), fail with a clear Russian error telling the user to re-run import —
do not silently fall back to a full reread.

If you find this infeasible or substantially worse than an alternative, STOP
and write your reasoning + alternative to the report file, then return with a
question instead of implementing something else.

Determinism invariants (must hold):
- single-threaded PRNG usage unchanged; do NOT parallelize generation;
- for the same world, manifest numbers (events, visits, users, checksum)
  must be identical whether computed the old way (full read) or via the new
  incremental path — the rolling checksum design must guarantee this
  (e.g. order-independent combine or strictly deterministic order);
- `check` operation semantics preserved: it must still be able to verify
  ClickHouse against the manifest.

## Boundaries

- Do NOT parallelize generation (breaks determinism — single PRNG stream,
  visits crossing day boundaries).
- Do NOT do incremental ETL: world_next_day keeps calling etl_pipeline
  full_refresh (tech debt tracked separately in #8).
- Do NOT touch retention beyond the incremental counters.
- Do NOT change the reference artifact file in git.
- Do NOT touch `docs/course/`.
- Do NOT commit.
- Update docs affected by the state/manifest change in the same change
  (at minimum `docs/runbooks/startup-history.md` if state format/semantics
  visible there; check `docs/ARCHITECTURE.md`, `docs/OPERATIONS.md`).

## Verification (run before reporting done)

- `make test` and `make lint` green (exit codes in report).
- `make generated-history-chain-check` if runnable in your environment;
  if it needs the live stand and you cannot run it, say so explicitly —
  the coordinator will run it during live acceptance.
- Unit tests for the new counter state: seeding on import/backfill,
  incremental update on next-day, equality of incremental vs full-recompute
  manifest numbers on a small fixture (this equality test is REQUIRED).
- Follow repo data rules: tests use small slices, not full jsonl files.

## Report

Write full report to
`/tmp/claude-1000/-home-dementev-sources-clickstream-ch-kafka-superset-demo/0836e970-ee07-4f86-a61f-291f32da525a/scratchpad/report-issue5.md`:
design decision on seeding + why, what changed (file list), how the rolling
checksum stays equal to the full recompute, test/lint exit codes and
counters, anything NOT done or uncertain. Return only a short summary + the
report path.