clickstream-ch-kafka-superset-demo/.scratch/issue5-run/report-issue5.md

Issue 5 report: incremental cumulative manifest counters

Outcome

Implemented the mandate's preferred seeding design. backfill seeds counters while generated batches already flow through StartupHistoryArtifactBuilder; import seeds them from the artifact topics that are already loaded and validated before Kafka publication. next-day restores the cumulative state and adds only the newly generated batches. The full-history KafkaDataTopicReader.load() call and its construction were removed from the next-day service path.

No git commit was created, and data/startup_history/reference-world.json.xz was not changed.

Seeding and storage design

• The reference artifact remains version 1.0 and contains no cumulative section. StartupHistoryArtifactBuilder.to_artifact() deliberately removes the local cumulative sections from its state and manifest copies.
• On import, the existing artifact checksum is still accepted and validated. The importer builds the new cumulative state from the already loaded topic events, then writes the enriched local state and manifest to Kafka.
• On backfill, the existing builder counters are persisted directly after successful publication.
• The local manifest stores:
  • per-topic row totals and timestamp bounds;
  • exact click_id and user_domain_id sets, encoded as sorted JSON, zlib-compressed, and base64-encoded;
  • the rolling checksum state.
• The local generator state stores a compact SHA-256 reference to the full cumulative section in the manifest. This prevents mismatched state/manifest pairs without duplicating the large exact ID sets in both Kafka compact-topic messages.
• The split was chosen after checking the real reference manifest: 26,083 visits and 4,056 users. Duplicating both open UUID arrays in state and manifest would risk Kafka's normal message-size limit. The full exact sets still exist in the manifest, while state proves which manifest counter state it belongs to.
• Airflow precheck and the generator both validate the state reference, cumulative payload, visible topic counters, and totals before generating. An old local state fails in Russian with an instruction to re-run import; there is no fallback to a Kafka history reread.

Rolling checksum and determinism

Each event is serialized with the existing canonical JSON settings (sort_keys=True, ensure_ascii=True) and hashed with SHA-256. A topic's rolling checksum is the sum of those 256-bit digests modulo 2^256. This operation is associative and commutative, so restoring the saved accumulator and adding a new day gives the same result as recomputing all events in one pass, independent of batch boundaries. Generation and PRNG usage remain single-threaded and unchanged.

The required equality unit test is test_incremental_counters_equal_full_recompute. It seeds the first fixture day, serializes/restores the cumulative state, adds a second day, and compares every topic statistic and every total with a full recompute over the combined fixture. A separate compatibility test proves that an artifact carrying the previous concatenated SHA-256 checksum remains valid and does not need regeneration.

Verification

make test

Exit code: 0

Summary:

• Generator image build: successful.
• Generator tests: 214 passed in 16.00s.
• Repository contract tests: 31 passed in 0.26s.
• docker compose config --quiet: successful.

make lint

Exit code: 0

Summary:

• Generator image build: successful.
• Python compileall: successful.
• Bash syntax checks: successful.
• docker compose config --quiet: successful.
• git diff --check: successful.

make generated-history-chain-check

Attempted, exit code: 2.

The live check was not runnable in this execution context. The script failed while connecting to the Docker API:

permission denied while trying to connect to the docker API at unix:///var/run/docker.sock
Ошибка: manifest boundaries должен содержать T0 и T_end
make: *** [Makefile:64: generated-history-chain-check] Error 1

The second message follows from the missing Docker result rather than a completed chain check. The coordinator should re-run this command in the live acceptance environment.

Files changed

• generator/src/clickstream_generator/startup_history_artifact.py
• generator/src/clickstream_generator/state.py
• generator/src/clickstream_generator/service.py
• generator/src/clickstream_generator/airflow_control.py
• generator/tests/test_startup_history_artifact.py
• generator/tests/test_state.py
• generator/tests/test_service.py
• generator/tests/test_airflow_control.py
• docs/runbooks/startup-history.md
• docs/OPERATIONS.md
• docs/ARCHITECTURE.md

Open questions and follow-up

• Re-run make generated-history-chain-check against a live stand with Docker API access.
• Live acceptance should compare generation-plus-manifest duration across multiple next-day runs. The code path is now O(events in the new day) for generation/counter work, but this sandbox run did not perform the multi-day wall-clock benchmark.
• ETL remains full_refresh exactly as required; its age-dependent cost is still tracked separately.

Triage round 1

Findings

• CODE-1/TASK-1 — FIXED. Exact ID sets are no longer embedded in the single cumulative manifest record. They are split into immutable, content-addressed chunks of at most 10,000 IDs in the separate compact topic generator_startup_history_counter_chunks. The main manifest stores cumulative totals, rolling topic checksums, and the SHA-256 head/count of the chunk chain. next-day writes only chunks for IDs first seen in the new day; it does not rewrite or scan historical chunks.
• CODE-2 — FIXED. next-day now flushes new counter chunks, then the main manifest, and saves state last. A manifest failure therefore cannot leave state pointing at an unpublished manifest. Import and backfill use the same chunks -> manifest -> state order.

Representation and exactness

Raising Kafka message limits alone was rejected because it would preserve the O(history) rewrite and only move the ceiling. Chunking keeps each Kafka value bounded while allowing total stored counter data to grow linearly with world age.

Each chunk contains the exact compressed click_id and user_domain_id values and the SHA-256 of the previous chunk. The chunk key is its own SHA-256 and is checked before send. The main manifest remains the source of exact visits and users totals used by the ClickHouse uniqExact check. On resume, IDs that can legally reappear across the day boundary come from generator state: active visits, and users that already have an active or completed visit. Unused pre-created population members are deliberately excluded. New visits still receive a new deterministic PRNG UUID. The incremental-vs-full-recompute test compares both topic statistics and complete manifest totals.

Both the main manifest and each chunk have an explicit 900,000-byte pre-send guard with a Russian error. Chunks are in a separate topic so loading the single main manifest key does not scan historical chunk payloads.

The reference artifact in git remains unchanged.

Measured payload and growth

Measured against data/startup_history/reference-world.json.xz (280,437 events, 26,083 visits, 4,056 users):

• main manifest JSON: 3,152 bytes;
• compact cumulative counter section inside it: 1,219 bytes;
• exact ID chain: 4 chunks, 846,718 bytes total;
• largest chunk: 282,324 bytes, 31% of the explicit 900,000-byte guard.

At the reference world's observed average, exact-set storage grows by about 282,239 bytes per model day. Projection: about 28,223,933 bytes (26.9 MiB) at 100 days and 103,017,357 bytes (98.2 MiB) at 365 days, spread across bounded messages. The main manifest grows only through the short boundaries list: measured synthetic sizes were 6,023 bytes at 100 days and 13,708 bytes at 365 days.

Tests changed

• Added a hash-chain segmentation test with a deliberately tiny chunk limit, including continuation from a saved chain and exact duplicate handling at a day boundary.
• Added explicit oversized-message and wrong-content-hash rejection tests.
• Updated the required incremental-vs-full-recompute test to provide the exact boundary IDs available from generator state.
• Updated import tests to assert chunk persistence and the manifest chain head.
• Updated the two-day service test to assert chunks -> manifest -> state ordering and full manifest totals, including users that were pre-created but had not appeared before the boundary.
• Existing mismatch, old-state, rolling-checksum, artifact-compatibility, publish-failure, and no-Kafka-reread coverage remains green.

Verification after fixes

• make test: exit code 0; generator tests 216 passed in 16.46s, repository contract tests 31 passed in 0.27s, compose validation successful.
• make lint: exit code 0; image build, Python compileall, Bash syntax, compose validation, and git diff --check successful.
• make generated-history-chain-check: exit code 2; still not runnable in this execution context because the script cannot access /var/run/docker.sock. It then reports missing manifest boundaries as a consequence. The live acceptance environment must rerun it.
• Working tree remains uncommitted; no git commit was created.