fix(skill): findings from round-4 adversarial review

- Зачем:
  - round-4 self-review на ветке нашёл HIGH-bug (пропущен плейсхолдер в Step 2, буквальный Opus мог не подставить литерал) и три MEDIUM (лишний abort на APPROVED при пустом `~/.codex/sessions/`, ложное обещание детекции wrong-session, same-epoch race при быстром codex). Исправляю до PR, чтобы не оставлять известных дефектов.
- Что:
  - SKILL.md Step 2: `${CODEX_SESSIONS_BEFORE}` добавлен в список placeholders с явным пояснением роли.
  - SKILL.md Step 4/7 + Rules: timestamp capture через `$(($(date +%s) - 1))` — сдвиг на секунду защищает от same-epoch race против `-newermt` strict-greater.
  - SKILL.md Step 4: перестановка проверок — review-sanity (check 3) раньше session-id capture (check 4); на APPROVED session-id не нужен → skip entirely. Убирает лишний abort валидной APPROVED round.
  - SKILL.md Step 4 check 4: parallel-codex caveat переписан честно — wrong-session resume НЕ детектируется автоматически, риск silent corruption зафиксирован.
  - SKILL.md Step 4 check 4: при zero-lines перед abort выводятся диагностические артефакты (jsonl, stderr, 3 недавних rollout).
  - DESIGN.md §4.8: обновлён с 4-check порядком и объяснением почему review раньше session-id.
  - DESIGN.md §4.1: alternative-considered про filesystem-only-primary переписан — убрано ложное обещание §4.8-детекции; trade-offs расширены `-1` shift и skip-on-APPROVED.
  - README.md: добавлены permissions `Bash(ls -t ~/.codex/sessions*)` для диагностики.
- Проверка:
  - Заново прогнать §7.1 smoke-test: EXIT=0, rollout-UUID извлекается даже при пустом JSONL.
  - Edge case: codex exec с cached response <1s + `CODEX_SESSIONS_BEFORE=T` → `find -newermt "@$((T-1))"` всё равно находит rollout.
This commit is contained in:
2026-04-17 17:31:24 +03:00
parent af0ad8ab11
commit bbf4499b71
3 changed files with 65 additions and 26 deletions
+22 -14
View File
@@ -23,7 +23,7 @@ Sends current work for adversarial review through an external AI model (OpenAI C
## Instructions
> **Placeholders:** `${REVIEW_ID}`, `${CODEX_SESSION_ID}`, `${REPO_ROOT}`, and `${BASE_BRANCH}` in the steps below are template placeholders, NOT shell variables. Substitute literal values directly into each tool call. In particular, `${REPO_ROOT}` is ALWAYS an absolute path captured at Step 2; never replace it with `$(pwd)`.
> **Placeholders:** `${REVIEW_ID}`, `${CODEX_SESSION_ID}`, `${CODEX_SESSIONS_BEFORE}`, `${REPO_ROOT}`, and `${BASE_BRANCH}` in the steps below are template placeholders, NOT shell variables. Substitute literal values directly into each tool call. In particular, `${REPO_ROOT}` is ALWAYS an absolute path captured at Step 2 (never `$(pwd)`); `${CODEX_SESSIONS_BEFORE}` is a unix-timestamp integer captured immediately before every `codex exec` / `codex exec resume` (see Steps 4 and 7), used by the filesystem session-id fallback — substitute the integer verbatim into `find -newermt "@<integer>"`, never leave `${CODEX_SESSIONS_BEFORE}` as a shell variable reference.
### Step 1: Determine review mode
@@ -308,7 +308,7 @@ And the following items are added to `<attack_surface>`:
**Launching Codex — command template:**
Flags:
- `--json` — stdout becomes JSONL events (primary path for session-ID capture). In some sandbox configurations this stream ends up empty; the filesystem fallback in check 3 below handles that case.
- `--json` — stdout becomes JSONL events (primary path for session-ID capture). In some sandbox configurations this stream ends up empty; the filesystem fallback in check 4 below handles that case.
- `-m gpt-5.4` — model (overridden by `model:...` argument)
- `-c model_reasoning_effort=high` — reasoning depth (overridden by `xhigh`, `low`, etc.)
- `-s read-only` — reviewer only reads, does not write
@@ -319,13 +319,13 @@ Flags:
**Plan Mode note:** Writing to `/tmp` via Write tool may trigger a permission prompt or exit Plan Mode. This is a known Claude Code limitation — Plan Mode restricts edits to the plan file only. If this happens, it does not affect review correctness: the review mode is already determined, and the skill only edits the plan file and `/tmp` temp files.
**Capture pre-exec timestamp** (for filesystem fallback of session-id; see check 3 below). Substitute the value literally:
**Capture pre-exec timestamp** (for filesystem fallback of session-id; see the secondary-path check below). Substitute the value literally:
```bash
date +%s
echo $(($(date +%s) - 1))
```
Save as `CODEX_SESSIONS_BEFORE` (a template placeholder — a Unix timestamp as an integer).
Save as `CODEX_SESSIONS_BEFORE` (a template placeholder — a Unix timestamp as an integer). The `- 1` shifts the window back one second to avoid a same-epoch race: `find -newermt "@N"` treats mtime **strictly greater** than N, so if codex finishes in the same epoch-second as the capture (fast path, cached response), the rollout file would be missed without this shift. Cost: the lookup window widens by 1 second, which is irrelevant against the codex exec duration (seconds to minutes).
```bash
cat /tmp/codex-prompt-${REVIEW_ID}.md | timeout 600 codex exec --json \
@@ -359,7 +359,12 @@ cat /tmp/codex-prompt-${REVIEW_ID}.md | timeout 600 codex exec --json \
- If file contains a line matching `^Error:` or `Failed to write` → codex reported an infrastructure failure despite exit 0. Show stderr to user, route to launch-failure retry (max 1 per round; after retry failure → hard abort).
- Otherwise → proceed.
3. **Capture `CODEX_SESSION_ID` — two-tier.**
3. **Review file sanity.** Read `/tmp/codex-review-${REVIEW_ID}.md`. It must exist and contain a line matching `^VERDICT: (APPROVED|REVISE)$`; if REVISE, it must also contain at least one line matching `\[severity:\s*(critical|high|medium)` (a structured finding). The full semantic-check logic is in Step 5; do the same thing here.
- Fails → route to launch-failure retry (max 1 per round; after retry failure → hard abort). Do NOT capture `CODEX_SESSION_ID` — if the review itself is broken, the session is of no use.
- Passes with `VERDICT: APPROVED``CODEX_SESSION_ID` is not needed (no Step 7 resume will happen). Skip the capture below entirely and proceed to Step 5.
- Passes with `VERDICT: REVISE` → capture `CODEX_SESSION_ID` next (check 4).
4. **Capture `CODEX_SESSION_ID` — two-tier.** Only reached when the review was valid AND the verdict is REVISE (Step 7 resume is about to happen).
**What you are looking for.** A UUID string (format `[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}`) that identifies the codex session, so `codex exec resume <UUID>` in Step 7 can continue this conversation. Try the cheap source first, fall back to the filesystem only if needed.
@@ -367,7 +372,7 @@ cat /tmp/codex-prompt-${REVIEW_ID}.md | timeout 600 codex exec --json \
```json
{"type":"thread.started","thread_id":"<uuid>","...":...}
```
Parse it as JSON, take `thread_id`, save as `CODEX_SESSION_ID`, proceed to check 4.
Parse it as JSON, take `thread_id`, save as `CODEX_SESSION_ID`, proceed to Step 5.
**Secondary: rollout filename.** In some Claude Code sandbox configurations the `--json` stdout file is empty (0 bytes) even when the review completes successfully (`-o` is populated, exit 0, stderr clean). In that case the session is still recoverable from disk: every `codex exec` writes a rollout file named `rollout-<ISO-timestamp>-<UUID>.jsonl` under `~/.codex/sessions/YYYY/MM/DD/` (see `DESIGN.md §2.3`). The trailing UUID in the filename is the session id.
@@ -379,16 +384,19 @@ cat /tmp/codex-prompt-${REVIEW_ID}.md | timeout 600 codex exec --json \
This prints zero or more lines of `<epoch-mtime> <filename>` for rollout files created after the pre-exec timestamp. From the result:
- **Zero lines** → no rollout file was created; treat as launch failure, show stderr, retry once, then abort.
- **Zero lines** → no rollout file found. Before aborting, surface useful diagnostic context to the user: the contents of `/tmp/codex-stdout-${REVIEW_ID}.jsonl` (if non-empty), `/tmp/codex-stderr-${REVIEW_ID}.txt`, and the 3 most-recent rollout filenames (`ls -t ~/.codex/sessions/*/*/*/rollout-*.jsonl 2>/dev/null | head -3`). Then treat as launch failure, retry once, then abort.
- **One or more lines** → pick the line with the largest epoch-mtime (there is usually just one; multiple lines indicate a parallel codex invocation in the same second). Extract the trailing UUID from that filename (the 36-char hex-and-dashes pattern above) and save as `CODEX_SESSION_ID`.
A single `find` invocation keeps the permission rule simple (`Bash(find ~/.codex/sessions*)`) and the parsing stays in your head — no shell pipeline needed.
**Platform note.** `-newermt "@<epoch>"` and `-printf` are GNU extensions. On macOS (BSD find) they are unsupported — substitute an equivalent that achieves the same goal: "list rollout files modified since `${CODEX_SESSIONS_BEFORE}`, newest first". For example, `find ~/.codex/sessions -name 'rollout-*.jsonl' -type f` plus `stat -f '%m %N' <path>` per result, or `ls -t ~/.codex/sessions/*/*/*/rollout-*.jsonl` and filter by a reference file's mtime. The goal is what matters, not the exact flags.
**Parallel-codex caveat:** if you happened to pick a rollout from a parallel codex invocation, Step 7's resume will either succeed against the wrong session (detected later via VERDICT / severity checks) or fail at the stderr/exit-code check and route to the standard fallback (§4.11). Either outcome is recoverable.
**Parallel-codex caveat — real silent-corruption risk, not benign.** If the user runs `codex` in parallel in the same cwd during the pre-exec timestamp window, the newest rollout may belong to that other invocation. Step 7's resume against the wrong session returns a normally-shaped response (VERDICT + severity markers), so the skill's post-resume checks in Step 7 will NOT detect the mismatch. The skill then applies "fixes" guided by a review of the wrong artifact. Two mitigations in effect:
4. **Review file sanity.** (Performed again in Step 5, but note upfront.) `/tmp/codex-review-${REVIEW_ID}.md` must exist and contain a line matching `^VERDICT: (APPROVED|REVISE)$`. If not → Step 5 will handle it via retry/abort.
1. The `CODEX_SESSIONS_BEFORE` timestamp is captured immediately before the exec, so the window is narrow (seconds).
2. `--last` is never used (`§4.5`) — explicit UUID is always passed to resume.
Neither eliminates the race. If you see suspicious behavior mid-review (reviewer mentions files or sections not in this work), halt and tell the user before applying any fix.
**Where `thread_id` / session id is NOT:**
@@ -483,10 +491,10 @@ End with VERDICT: APPROVED or VERDICT: REVISE
**2. Run resume.** Resume does NOT accept `-C`, so prefix the command with an explicit `cd` to `${REPO_ROOT}` (captured at Step 2). Use single quotes around `${REPO_ROOT}` — the path was validated at Step 2 to contain no single quotes.
Capture a pre-resume timestamp for the secondary session-id fallback (same pattern as Step 4):
Capture a pre-resume timestamp for the secondary session-id fallback (same pattern as Step 4, with the `-1` shift against same-epoch race):
```bash
date +%s
echo $(($(date +%s) - 1))
```
Save as `CODEX_SESSIONS_BEFORE`. Then launch resume via the same `cat | ... -` pattern:
@@ -518,7 +526,7 @@ Use `timeout: 620000` in Bash tool parameters.
3. **Review file sanity.** Read `/tmp/codex-review-${REVIEW_ID}.md` and apply the same checks as Step 5.2:
- Missing / empty / no `^VERDICT: (APPROVED|REVISE)$` line / REVISE without `[severity:` lines → route to fallback. Do NOT update `CODEX_SESSION_ID`.
**4. Only if all three checks pass** → refresh `CODEX_SESSION_ID` with the same two-tier approach from Step 4 check 3 (primary = first JSONL line of `/tmp/codex-stdout-${REVIEW_ID}.jsonl`; secondary = newest rollout filename with mtime > `CODEX_SESSIONS_BEFORE`, UUID extracted from the basename).
**4. Only if all three checks pass AND the verdict is REVISE** → refresh `CODEX_SESSION_ID` with the same two-tier approach from Step 4 check 4 (primary = first JSONL line of `/tmp/codex-stdout-${REVIEW_ID}.jsonl`; secondary = newest rollout filename with mtime > `CODEX_SESSIONS_BEFORE`, UUID extracted from the basename). On APPROVED verdict, skip the refresh — there is no round N+1.
Per `DESIGN.md §2.4.4`, successful resume does not rotate the thread id — the new value equals the previous one, so this refresh is defensive. If both tiers yield nothing but the three checks passed → the resume itself was fine; keep the previous `CODEX_SESSION_ID` unchanged and continue.
@@ -669,7 +677,7 @@ Do NOT delete plan files that existed before the review (only temp files created
- **`REPO_ROOT` is captured at Step 2** via `git rev-parse --show-toplevel` and substituted as an absolute literal path into every codex command. Never use `$(pwd)` inside codex commands — cwd drift between Bash calls makes it unreliable.
- **Resume requires `cd '${REPO_ROOT}' && ...`** because `codex exec resume` has no `-C` flag; cwd is inherited from the shell. The initial exec uses `-C "${REPO_ROOT}"` instead.
- **`CODEX_SESSION_ID` is updated only on full success** — ALL of (exit=0 AND stderr has no `Error:`/`thread/resume failed` line AND review file contains a valid `VERDICT:` line with findings on REVISE). On any failure, leave it unchanged and route to the fallback.
- **Session ID capture is two-tier.** Primary: `thread_id` from the first JSONL line of stdout. Secondary (when stdout is empty — env-specific): UUID from the trailing component of the newest `~/.codex/sessions/**/rollout-*.jsonl` filename with mtime > `CODEX_SESSIONS_BEFORE`. Capture `CODEX_SESSIONS_BEFORE=$(date +%s)` **before** every `codex exec` / `codex exec resume` call.
- **Session ID capture is two-tier.** Primary: `thread_id` from the first JSONL line of stdout. Secondary (when stdout is empty — env-specific): UUID from the trailing component of the newest `~/.codex/sessions/**/rollout-*.jsonl` filename with mtime > `CODEX_SESSIONS_BEFORE`. Capture `CODEX_SESSIONS_BEFORE=$(($(date +%s) - 1))` **before** every `codex exec` / `codex exec resume` call (the `-1` shift prevents a same-epoch race against `-newermt`'s strict-greater semantics).
- **Prompt delivery is `cat file | codex exec ... -`.** The `- < file` stdin-redirect form is accepted by codex but exits 1 with empty stderr in some Claude Code sandbox configurations. Pipe is portable across both envs observed.
- **The `--json` stdout stream is never human-readable review text** — JSONL events when populated, empty when suppressed by sandbox. Never treat Bash result as review content; the review lives exclusively in `/tmp/codex-review-*.md`.
- **Launch-failure retry** is capped at 1 per round and does NOT consume the 5-round counter. The retry counter is per-round; it resets at the start of every new round and is tracked only in that round's reasoning.