feat(skill): refresh adversarial-review под Codex 0.132
- Зачем:
- актуализация под новый Codex CLI (0.132 убрал флаг -a,
ввёл -c approval_policy / -c approvals_reviewer);
- переход на безопасные операторские дефолты
(sandbox=workspace-write для эмпирической верификации,
явные overrides sandbox:* и approvals:*);
- наведение порядка в принятии решений по раунду ревью:
review_quality + evaluation matrix + structural operator gate
вместо «оператор сам разбирается».
- Что:
- SKILL.md: override-граммар (sandbox:*, approvals:*, model:*,
low/medium/high/xhigh), детект OPERATOR_LANGUAGE, runtime hint,
conditional dirty-file warning в Step 2 (после захвата REPO_ROOT),
dual-layer mutation snapshots (git status + sha256sum input-файлов),
operation-aware dispatch table, evaluation matrix + structural
operator gate (batch-pause), structured resume body
(Applied / Re-scoped / Rejected / Specific asks), final operator
summary на OPERATOR_LANGUAGE.
- references/runner.md: переведён на Sonnet runner, добавлен
Step R2.5 bwrap preflight, Step R4.5 review_quality + bounded
triage, расширена 11-полевая JSON-схема результата.
- docs/DESIGN.md: §4.14–§4.21 с обоснованиями новых решений,
§7.8 refresh-era smoke checks, §8 version log с двумя раундами
dogfood-а этого refresh-а (включая R2-корректировку
обоснования residual gap для уже-грязных tracked-файлов).
- README.md: таблица дефолтов, Safety considerations (честно
задокументирован residual gap), Linux sandbox prerequisites
(bwrap + AppArmor user namespaces), Operator language,
Final operator summary, troubleshooting.
- examples/review-output.md: модель в сэмпле обновлена на gpt-5.5.
- docs/superpowers/specs/2026-05-20-...: сохранена спека дизайна
с уточнениями после dogfood.
- Проверка:
- dogfood: 2 раунда /adversarial-review code на этом refresh-е,
R2 верифицировал R2#1 (ordering) и R2#2 (Option B, honest
residual gap);
- smoke: codex --version ≥ 0.132.0, bwrap preflight зелёный
на reference WSL2.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
+442
-7
@@ -829,6 +829,311 @@ Each decision below follows the same template:
|
||||
command (codex), which matches what the skill already checks; no
|
||||
`pipefail` needed.
|
||||
|
||||
### §4.14. Default model `gpt-5.5` (refresh 2026-05-20)
|
||||
|
||||
- **Decision.** The default reviewer model is `gpt-5.5`. Earlier defaults
|
||||
were `gpt-5.4` and `gpt-5.3-codex`.
|
||||
- **Why.** Each Codex generation reduces false-positive rate on
|
||||
adversarial review and improves citation accuracy. The default tracks
|
||||
the latest stable Codex model that ChatGPT+API auth can both reach.
|
||||
- **Rejected alternatives.**
|
||||
- *Pin to a fixed model forever.* Causes the skill to silently drift
|
||||
out of date.
|
||||
- *Default to the latest available regardless of stability.* Mid-loop
|
||||
behavior changes invalidate the round-by-round severity trajectory
|
||||
that the loop relies on.
|
||||
- **Trade-offs accepted.** The version-log table in §8 needs a new row
|
||||
per default bump so users can verify the smoke protocol on the chosen
|
||||
model.
|
||||
|
||||
### §4.15. Default sandbox `workspace-write`, not `read-only` (refresh 2026-05-20)
|
||||
|
||||
- **Decision.** Initial / fresh-exec dispatches default to
|
||||
`-s workspace-write`. `read-only` is a deliberate operator opt-in via
|
||||
`sandbox:read-only`, never an automatic per-mode default.
|
||||
- **Why.** The reviewer's value proposition is empirical verification —
|
||||
"the reviewer ran the tests and X failed", "the reviewer built the
|
||||
project and Y broke", "the reviewer queried the live API and the
|
||||
assumed signature does not exist." Each of those requires write-class
|
||||
operations: tests write cache and stdout files, builds emit
|
||||
artifacts, CLIs touch local state. Read-only blocks all of them. The
|
||||
remaining read-only-compatible verification (file inspection, `rg`,
|
||||
`--help` introspection, MCP doc lookups that don't cache) is a small
|
||||
fraction of the high-value path.
|
||||
- **Rejected alternatives.**
|
||||
- *Default `read-only` for plan reviews.* The original 2026-05
|
||||
design tried this; reverted because plan reviewers also benefit
|
||||
from running a test the plan relies on, building to confirm a
|
||||
structural claim, or hitting an external API to validate an
|
||||
assumption.
|
||||
- *Sandbox-level enforcement of "no project file writes".* Codex
|
||||
does not expose a "write to `/tmp` only, not `${REPO_ROOT}`"
|
||||
sandbox tier — the closest is `read-only`, which blocks exec.
|
||||
- *Auto-create a `git worktree` per review.* Adds setup latency to
|
||||
every dispatch and complicates `RUNNER_SPEC_PATH` resolution and
|
||||
`REPO_ROOT` capture (the worktree path is not the operator's
|
||||
canonical repo root). Documented as an operator-side mitigation
|
||||
in `README.md` instead.
|
||||
- **Trade-offs accepted.** Reviewer-side writes to gitignored state
|
||||
inside `REPO_ROOT` are real and not architecturally prevented; they
|
||||
are mitigated through documentation (`README.md` "Safety
|
||||
considerations"), the per-review runtime hint, and the
|
||||
`sandbox:read-only` opt-out. Tracked-file mutation IS detected via
|
||||
`git status --porcelain` snapshots — see §4.20.
|
||||
|
||||
### §4.16. Approval policy `on-request` + `auto_review` reviewer (refresh 2026-05-20)
|
||||
|
||||
- **Decision.** The default approval flow is
|
||||
`-c approval_policy='"on-request"'` plus
|
||||
`-c approvals_reviewer='"auto_review"'`. The top-level
|
||||
`-a` / `--ask-for-approval` flag (removed in Codex CLI 0.132) is NOT
|
||||
emitted by the runner.
|
||||
- **Why.**
|
||||
- `on-request` keeps boundary crossings explicit so the reviewer can
|
||||
still ask before running a destructive operation, rather than
|
||||
silently failing.
|
||||
- `auto_review` for the approval reviewer avoids nested human
|
||||
approval prompts inside the Codex subprocess. From the parent
|
||||
Claude session, the operator cannot reliably see, answer, or even
|
||||
detect those prompts — they appear as the run "hanging". The
|
||||
`auto_review` policy auto-approves benign operations and rejects
|
||||
obviously harmful ones, matching the boundary semantics the skill
|
||||
needs.
|
||||
- The config-based `-c approval_policy=...` form is the only form
|
||||
supported on Codex CLI 0.132+; the runner emits it
|
||||
unconditionally for forward compatibility with the unknown shape
|
||||
of future flag churn.
|
||||
- **Rejected alternatives.**
|
||||
- *Pass `-a on-request` directly.* Removed in 0.132; would silently
|
||||
fail on modern CLI versions.
|
||||
- *Default to `approvals:user` (nested human prompt).* The nested
|
||||
prompt is invisible from the parent Claude session and will hang
|
||||
the run.
|
||||
- *Default to `approvals:never` (boundary crossings fail
|
||||
automatically).* Loses graceful degradation for routine operations
|
||||
that just need a one-time approval.
|
||||
- *Expose Codex's `untrusted` approval policy as an override.* The
|
||||
skill needs predictable boundary semantics, not per-command trust
|
||||
prompts.
|
||||
- **Trade-offs accepted.** `auto_review` is not a security boundary —
|
||||
it reviews approval requests, not actions already permitted by the
|
||||
selected sandbox. The skill relies on the `<reviewer_permissions>`
|
||||
prompt contract plus the workspace mutation snapshot for the real
|
||||
guardrails.
|
||||
|
||||
### §4.17. Sonnet runner triage as a hint, not a decision (refresh 2026-05-20)
|
||||
|
||||
- **Decision.** The runner subagent (Sonnet) produces a compact
|
||||
`triage` object — finding counts, max severity, per-severity
|
||||
coverage counts, truncation flag, lead-judgment hint. This object is
|
||||
returned in the result JSON and consumed by the lead as a hint. The
|
||||
lead's evaluation matrix in Step 6 is built from the verbatim
|
||||
review, NOT from triage. Triage is never passed back to Codex.
|
||||
- **Why.**
|
||||
- Codex returns a long-form review; main historically had to read
|
||||
the entire review file to extract simple counts ("how many
|
||||
critical findings did the reviewer raise?"). With triage, main
|
||||
gets cheap structured signal without reading the file twice.
|
||||
- The runner is the right layer to do triage because (a) it already
|
||||
has the review file in context after Step R4 checks, and (b) the
|
||||
runner's context is disposable — its 1M context can hold the
|
||||
review and triage logic without polluting main.
|
||||
- Keeping triage as a hint (not a decision) preserves the rule that
|
||||
final `accept` / `reject` / `re-scope` outcomes are the lead's.
|
||||
Triage that ranked findings or pre-emptively rejected them would
|
||||
move the decision boundary into the cheaper model, which is
|
||||
exactly the kind of authority drift past adversarial-review
|
||||
rounds warned against.
|
||||
- **Rejected alternatives.**
|
||||
- *Let the runner pre-apply a heuristic accept/reject.* Drifts
|
||||
final-decision authority into Sonnet. Rejected on principle.
|
||||
- *Let the runner re-write the review file with a triage summary.*
|
||||
Mutates the artifact main shows verbatim. Rejected to preserve
|
||||
the "show review verbatim" contract.
|
||||
- *Forward triage metadata to Codex on resume.* Codex would see
|
||||
Sonnet's judgments and either echo them (false agreement) or
|
||||
push back on them (wasted round on a triage artifact). Rejected.
|
||||
- **Trade-offs accepted.** Triage may be wrong — wrong counts,
|
||||
miscategorized severity, false `needs_lead_judgment` flag. The lead
|
||||
re-derives counts during the evaluation matrix anyway; the hint
|
||||
saves a small amount of cognitive load on round 1 but does not
|
||||
load-bear on correctness.
|
||||
|
||||
### §4.18. Operator-language detection from recent messages (refresh 2026-05-20)
|
||||
|
||||
- **Decision.** At Step 1, main inspects the last few human-authored
|
||||
conversation turns and captures `OPERATOR_LANGUAGE` (English by
|
||||
default; non-English when detection is unambiguous). Runtime prose
|
||||
uses `OPERATOR_LANGUAGE`. Repository files (this `DESIGN.md`,
|
||||
`SKILL.md`, `README.md`, `references/runner.md`, specs under
|
||||
`docs/superpowers/specs/`) stay in English regardless. The reviewer
|
||||
is asked to produce findings in `OPERATOR_LANGUAGE` while keeping
|
||||
machine-readable literals (severity tags, `VERDICT:`, section
|
||||
headers) in English.
|
||||
- **Why.** Operators routinely work in non-English; verbatim review
|
||||
show-back, structural-gate prompts, and final summaries are more
|
||||
useful in the operator's language. Repository files stay in English
|
||||
because the contributor pool and AI-assistant training mass are
|
||||
predominantly English, and mixed-language docs are harder to
|
||||
maintain and grep.
|
||||
- **Rejected alternatives.**
|
||||
- *Ask the operator to declare language at invocation time.* Extra
|
||||
friction; the conversation already encodes the signal.
|
||||
- *Translate everything including section headers and severity
|
||||
tags.* Breaks the parser regexes (`^VERDICT:`, `\[severity:\s*`,
|
||||
`Summary` / `Findings` / `Verdict` section anchors).
|
||||
- *Default to the operator's locale.* Locale != active language —
|
||||
a Russian operator may be working through an English-language
|
||||
codebase and want English review prose.
|
||||
- **Trade-offs accepted.** Detection is heuristic and can be wrong on
|
||||
short conversations or code-switching. Falling back to English on
|
||||
ambiguity is safe; the operator can still read it, and the
|
||||
machine-readable literals are stable.
|
||||
|
||||
### §4.19. Lead evaluation matrix + structural operator gate (refresh 2026-05-20)
|
||||
|
||||
- **Decision.** Step 6 builds an explicit evaluation matrix
|
||||
(`finding | severity | verified? | type | action`) with three
|
||||
first-class actions: `accept`, `reject with reasoning`, `re-scope`.
|
||||
Step 6.3 applies a batch-pause rule: one operator prompt per round
|
||||
listing structural / non-structural / rejected, only when the
|
||||
structural count is ≥ 1 and an operator is reachable. Headless and
|
||||
autonomous runs apply structural fixes anyway but record the fact
|
||||
in the final operator summary.
|
||||
- **Why.**
|
||||
- Pre-refresh Step 6 used a flat "fix everything the reviewer
|
||||
raised" loop. This is the failure mode the
|
||||
`superpowers:receiving-code-review` skill exists to prevent — the
|
||||
reviewer can be technically wrong, and applying its findings
|
||||
blindly produces large structural edits based on misreadings.
|
||||
The matrix forces an evaluation pass before any fix.
|
||||
- `reject with reasoning` and `re-scope` are equal first-class
|
||||
outcomes, not exceptions. The structured resume prompt's
|
||||
"Rejected with reasoning" section gives the reviewer a chance to
|
||||
contest, which keeps the loop honest.
|
||||
- Structural-vs-non-structural gating exists because the highest-
|
||||
leverage refactor a misread finding can trigger is "rewrite the
|
||||
invocation grammar to match the reviewer's mental model." That's
|
||||
the exact case where one operator pause prevents large
|
||||
irrecoverable damage.
|
||||
- **Rejected alternatives.**
|
||||
- *Pause once per fix.* Confirmation fatigue; operator stops
|
||||
reading after pause 3.
|
||||
- *Apply all fixes automatically and let the operator review
|
||||
post-hoc.* Reverting a misread structural fix is far more
|
||||
expensive than one upfront prompt.
|
||||
- *Let the runner classify structural vs non-structural and
|
||||
pre-gate.* Returns to the §4.17 problem — moves final-decision
|
||||
authority into the cheaper model.
|
||||
- **Trade-offs accepted.** The matrix adds verbosity to the
|
||||
conversation. The structural-gate prompt adds latency. Both are
|
||||
small per round and saturate at low rounds because severity
|
||||
declines (see §4.19 in spirit — Step 6.5 severity-decline soft
|
||||
signal).
|
||||
|
||||
### §4.20. Workspace mutation snapshots (dual-layer, mandatory) (refresh 2026-05-20)
|
||||
|
||||
- **Decision.** Main captures `git status --porcelain` of `REPO_ROOT`
|
||||
AND `sha256sum` of `/tmp/codex-{body,plan,resume-body}-*` BEFORE
|
||||
and AFTER every Codex dispatch. Any tracked-file mutation, or any
|
||||
`/tmp` review-input mutation, is a hard stop before applying fixes.
|
||||
Gitignored mutation inside `REPO_ROOT` is NOT detected (deliberate
|
||||
trade-off; see §4.15 and `README.md` "Safety considerations").
|
||||
- **Why.** The default `workspace-write` sandbox allows the reviewer
|
||||
to run commands that can mutate the workspace. The
|
||||
`<reviewer_permissions>` prompt contract is the primary control,
|
||||
but a prompt-only safeguard has a non-zero failure rate
|
||||
(reviewer-side bug, prompt drift, ambiguous toolchain command). The
|
||||
snapshot provides a secondary, after-the-fact detection layer that
|
||||
catches mutation regardless of the cause.
|
||||
- **Rejected alternatives.**
|
||||
- *Snapshot only `git status`.* Misses reviewer mutation of the
|
||||
skill's own `/tmp` prompt body — a real attack surface where a
|
||||
reviewer could rewrite the prompt to suppress future findings.
|
||||
- *Snapshot the entire `REPO_ROOT` tree including gitignored
|
||||
files.* Too expensive per round (`node_modules`, `target/`,
|
||||
`.next/`) and produces confirmation fatigue on every benign
|
||||
side-effect.
|
||||
- *Run the snapshot once at start-of-review only.* Misses
|
||||
mid-loop mutation; mid-loop is when the round counter has
|
||||
already advanced and rollback is most expensive.
|
||||
- *Content-aware tracked-file snapshot (`git diff HEAD` patch +
|
||||
sha256sum) on every dispatch.* Considered during round-1 dogfood
|
||||
review of this refresh, and reconsidered in round-2 after a
|
||||
reviewer pushed back on the original rejection rationale (see
|
||||
Round-2 correction below). Would close the already-dirty
|
||||
tracked-file residual risk listed in "Trade-offs accepted" below,
|
||||
at the cost of an extra patch-file write + hash per dispatch, four
|
||||
additional permissions entries (`Bash(git -C * diff HEAD*)`,
|
||||
`Write/Read(/tmp/codex-tracked-pre-*)`,
|
||||
`Write/Read(/tmp/codex-tracked-post-*)`,
|
||||
`Bash(diff -q /tmp/codex-tracked-*)`), two additional cleanup
|
||||
glob paths, and a third parallel snapshot layer in Step 4.
|
||||
Rejected on a complexity-vs-frequency trade-off: the vector is a
|
||||
legitimately-invoked verification command with unintended write
|
||||
side effects on an already-dirty tracked file, with the
|
||||
`<reviewer_permissions>` prompt contract as the only skill-level
|
||||
protection layer and `sandbox:read-only` as the operator-side
|
||||
opt-out. The skill ships the residual gap honestly documented
|
||||
rather than hidden behind a third snapshot layer; if the risk
|
||||
materializes in practice the design notes the implementation path
|
||||
above.
|
||||
- *Round-1 rejection rationale that cited the
|
||||
`approval_policy=on-request` + `auto_review` gate as a protection
|
||||
layer for this vector.* Withdrawn in round-2. Workspace writes are
|
||||
permitted by the `workspace-write` sandbox and do not generate an
|
||||
approval request, so `auto_review` never evaluates them (consistent
|
||||
with §4.16's own statement that auto-review "reviews approval
|
||||
requests, not actions already permitted by the selected sandbox").
|
||||
The realistic vector reduces to a reviewer-invoked verification
|
||||
command (formatter, snapshot updater, lockfile regen, codegen)
|
||||
with unintended side effects on an already-dirty file. See
|
||||
`README.md` "Residual risk: already-dirty tracked files" for the
|
||||
operator-facing wording.
|
||||
- **Trade-offs accepted.** Per-dispatch overhead is two `git`
|
||||
commands and one `sha256sum` over three small files — negligible.
|
||||
Gitignored mutation inside `REPO_ROOT` remains the documented
|
||||
residual risk. **Already-dirty tracked-file content drift** is the
|
||||
second documented residual risk: the porcelain snapshot tracks status
|
||||
codes, not content hashes, so a reviewer that mutates a file already
|
||||
marked ` M` will not trip a hard stop. The single skill-level
|
||||
protection is the `<reviewer_permissions>` prompt contract; no
|
||||
approval-policy gate exists for workspace-permitted writes.
|
||||
Mitigated by (a) a runtime warning in `SKILL.md` Step 2 that fires
|
||||
when pre-review `git status --porcelain` shows already-modified
|
||||
tracked entries, (b) `README.md` "Safety considerations" explicitly
|
||||
documenting the vector and the lack of an approval-policy gate, and
|
||||
(c) the operator-side opt-out of `sandbox:read-only` (loses
|
||||
reviewer-side verification, gains write-protection).
|
||||
|
||||
### §4.21. Final operator summary at every terminal state (refresh 2026-05-20)
|
||||
|
||||
- **Decision.** Step 8 emits an operator-facing summary in
|
||||
`OPERATOR_LANGUAGE` at every terminal state — approved, max rounds
|
||||
reached, not verified, aborted. The summary follows the canonical
|
||||
per-state header block AND the final verbatim reviewer response;
|
||||
it does not replace either. Built from per-round decision accounts
|
||||
already shown in conversation; never from `/tmp` codex logs.
|
||||
- **Why.** Pre-refresh Step 8 ended with the verbatim review and a
|
||||
one-line status. For multi-round reviews this is insufficient —
|
||||
the operator has scrolled past round 1 and 2 by the time they see
|
||||
the final state, and "approved after 4 rounds" hides which
|
||||
rejections and re-scopes survived to the end. The summary
|
||||
consolidates the decision audit trail at the place where the
|
||||
operator decides whether to merge.
|
||||
- **Rejected alternatives.**
|
||||
- *Show full diffs in the summary.* Too long; operator can `git
|
||||
diff` for that.
|
||||
- *Re-read Codex stdout/stderr to assemble the summary.* Violates
|
||||
the architectural rule that main does not consume codex
|
||||
transport logs (§12.x of subagent architecture).
|
||||
- *Skip the summary on `Approved`.* The `Approved` summary is
|
||||
actually the most useful for the merge-decision audience.
|
||||
- **Trade-offs accepted.** Summary quality depends on context
|
||||
compaction. The skill states this limitation explicitly when
|
||||
history is incomplete rather than fabricating round-by-round
|
||||
details.
|
||||
|
||||
---
|
||||
|
||||
## §5. Rejected ideas
|
||||
@@ -1127,8 +1432,11 @@ End the LAST line with exactly: VERDICT: APPROVED
|
||||
EOF
|
||||
|
||||
cat /tmp/codex-prompt-${REVIEW_ID}.md | timeout 300 codex exec --json \
|
||||
-m gpt-5.4 -c model_reasoning_effort=low \
|
||||
-s read-only -C "${REPO_ROOT}" \
|
||||
-m gpt-5.5 -c model_reasoning_effort=low \
|
||||
-s workspace-write \
|
||||
-c approval_policy='"on-request"' \
|
||||
-c approvals_reviewer='"auto_review"' \
|
||||
-C "${REPO_ROOT}" \
|
||||
-o /tmp/codex-review-${REVIEW_ID}.md \
|
||||
- \
|
||||
> /tmp/codex-stdout-${REVIEW_ID}.jsonl \
|
||||
@@ -1223,11 +1531,17 @@ mkdir -p /tmp/smoke-cwd-a /tmp/smoke-cwd-b
|
||||
( cd /tmp/smoke-cwd-b && git init -q )
|
||||
# run codex in each; capture session ids
|
||||
ID_A=$(cd /tmp/smoke-cwd-a && echo "ALPHA" | \
|
||||
codex exec -s read-only -m gpt-5.4 \
|
||||
-c model_reasoning_effort=low - 2>&1 | grep 'session id:' | awk '{print $3}')
|
||||
codex exec -s workspace-write -m gpt-5.5 \
|
||||
-c model_reasoning_effort=low \
|
||||
-c approval_policy='"on-request"' \
|
||||
-c approvals_reviewer='"auto_review"' \
|
||||
- 2>&1 | grep 'session id:' | awk '{print $3}')
|
||||
ID_B=$(cd /tmp/smoke-cwd-b && echo "BRAVO" | \
|
||||
codex exec -s read-only -m gpt-5.4 \
|
||||
-c model_reasoning_effort=low - 2>&1 | grep 'session id:' | awk '{print $3}')
|
||||
codex exec -s workspace-write -m gpt-5.5 \
|
||||
-c model_reasoning_effort=low \
|
||||
-c approval_policy='"on-request"' \
|
||||
-c approvals_reviewer='"auto_review"' \
|
||||
- 2>&1 | grep 'session id:' | awk '{print $3}')
|
||||
echo "A=${ID_A}"
|
||||
echo "B=${ID_B}"
|
||||
# resume --last from cwd-a; expect to resume ID_A, not ID_B
|
||||
@@ -1264,6 +1578,125 @@ If §7.1–§7.5 do not produce the expected outputs:
|
||||
before modifying `SKILL.md`. Future contributors should know which
|
||||
facts they can still trust.
|
||||
|
||||
### §7.8. Refresh-era checks (added 2026-05-20)
|
||||
|
||||
These supplement §7.1–§7.6 and verify the behavior introduced by the
|
||||
2026-05-20 refresh. Run them on the installed Codex CLI version and
|
||||
record results in `§8`.
|
||||
|
||||
**Flag emission.**
|
||||
|
||||
- [ ] `codex exec --help | rg 'approval_policy|approvals_reviewer'` — confirm the
|
||||
installed Codex CLI accepts the `-c` form.
|
||||
- [ ] On a defaulted invocation, the runner subagent's launch command
|
||||
contains `-s workspace-write`, `-c approval_policy='"on-request"'`,
|
||||
and `-c approvals_reviewer='"auto_review"'`. Verify by reading the
|
||||
runner's R3 step (the runner does NOT emit the launch command to
|
||||
main, so this is checked by reading the runner spec, not by
|
||||
inspecting main's logs).
|
||||
- [ ] `codex exec --help` does NOT list `-a` / `--ask-for-approval` on
|
||||
0.132+. Expected; the runner does not emit `-a` regardless.
|
||||
- [ ] `codex exec resume --help` does NOT list `-s`, `-m`, or
|
||||
approval-related `-c` overrides. The runner must not pass them
|
||||
on resume.
|
||||
|
||||
**Sandbox preflight.**
|
||||
|
||||
- [ ] On a host where `bwrap --dev-bind / / --unshare-net /bin/echo ok`
|
||||
fails, dispatch initial review with default sandbox. Expected:
|
||||
runner returns `success + degraded_environmental` with a
|
||||
`user_warning` pointing at README "Linux sandbox prerequisites";
|
||||
main treats as terminal infrastructure failure; no fake review
|
||||
round is shown.
|
||||
- [ ] Apply the AppArmor profile per `README.md`. Re-run. Expected:
|
||||
preflight passes; review proceeds normally.
|
||||
- [ ] Pass `sandbox:read-only`. Expected: preflight still runs
|
||||
(read-only is also bwrap-backed); review proceeds; the runtime
|
||||
hint is suppressed.
|
||||
- [ ] Pass `sandbox:inherit`. Expected: preflight is skipped; review
|
||||
runs with whatever sandbox the user's Codex config selects.
|
||||
|
||||
**Review-quality classification.**
|
||||
|
||||
- [ ] A clean review with concrete findings → `review_quality=valid`,
|
||||
`triage.status=ok`, counts populated.
|
||||
- [ ] Synthesize a review file whose only content is "bwrap: setting up
|
||||
uid map: Permission denied" (no findings, no real verdict) and
|
||||
have the runner classify it. Expected:
|
||||
`review_quality=degraded_environmental`,
|
||||
`triage.status=skipped`.
|
||||
- [ ] Synthesize a review with `VERDICT: REVISE` plus severity tags but
|
||||
no concrete finding bodies (<80 chars each). Expected:
|
||||
`review_quality=degraded_content`, `triage.status=ok`.
|
||||
- [ ] Stop the triage-step Bash tool mid-run (or remove `rg` from
|
||||
PATH). Expected: `triage.status=failed`,
|
||||
`review_quality=valid` (the review itself is fine).
|
||||
|
||||
**Operation-aware dispatch.**
|
||||
|
||||
- [ ] Inject `success + degraded_environmental` on `OPERATION=initial`.
|
||||
Expected: main emits `user_warning`, does NOT show review
|
||||
verbatim, aborts with the "ABORTED — environmental failure"
|
||||
Step 8 template.
|
||||
- [ ] Inject `success + degraded_environmental` on `OPERATION=resume`.
|
||||
Expected: main emits `user_warning`, does NOT show review
|
||||
verbatim, does NOT consume a round, routes to Step 7.4 fallback
|
||||
with prior round's severity.
|
||||
- [ ] Inject `success + degraded_environmental` on
|
||||
`OPERATION=fresh-exec`. Expected: main treats as terminal
|
||||
not-verified.
|
||||
- [ ] Inject `success + degraded_content` on any operation. Expected:
|
||||
main emits `user_warning`, shows Step 5 verbatim, prompts the
|
||||
operator (interactive) or auto-advances after a short wait
|
||||
(headless).
|
||||
|
||||
**Workspace mutation snapshots.**
|
||||
|
||||
- [ ] Simulate the reviewer modifying a tracked file (touch a file in
|
||||
`REPO_ROOT` during a synthetic dispatch). Expected: post-snapshot
|
||||
`diff` is non-empty; main hard-stops before fixes with the
|
||||
tracked-file diagnostic.
|
||||
- [ ] Simulate the reviewer modifying `/tmp/codex-body-<REVIEW_ID>.md`
|
||||
mid-dispatch. Expected: `sha256sum` diff is non-empty; main hard-
|
||||
stops with the `/tmp` input diagnostic.
|
||||
- [ ] Clean dispatch with no mutation. Expected: both diffs empty;
|
||||
Step 5 proceeds.
|
||||
|
||||
**Backward compat for legacy runner results.**
|
||||
|
||||
- [ ] Manually write a runner result JSON missing `review_quality` and
|
||||
`triage`. Expected: main treats as `review_quality=unknown`,
|
||||
`triage.status=skipped`; no abort, no operator prompt.
|
||||
|
||||
**Runtime hint.**
|
||||
|
||||
- [ ] Default invocation: the hint appears exactly once, before Step 2.
|
||||
- [ ] Default invocation, round 2/3/4/5: hint does NOT repeat.
|
||||
- [ ] Invocation with `sandbox:read-only`: hint is suppressed.
|
||||
- [ ] Invocation with `sandbox:workspace-write` (explicit, same as
|
||||
default): hint is suppressed.
|
||||
|
||||
**Operator language.**
|
||||
|
||||
- [ ] Recent operator messages in Russian → `OPERATOR_LANGUAGE = Russian`.
|
||||
Reviewer output is in Russian; severity tags, `VERDICT:`, and
|
||||
section headers stay in English. Final operator summary is in
|
||||
Russian.
|
||||
- [ ] Recent operator messages mixed or short → `OPERATOR_LANGUAGE`
|
||||
falls back to English; no language block is appended to the
|
||||
prompt.
|
||||
|
||||
**Final operator summary.**
|
||||
|
||||
- [ ] Approved terminal: summary appears in `OPERATOR_LANGUAGE` after
|
||||
the verbatim approved review.
|
||||
- [ ] Max rounds: summary appears with remaining findings.
|
||||
- [ ] Not verified: summary appears with the
|
||||
`WARNING: This is NOT an approval` framing.
|
||||
- [ ] Aborted environmental: summary appears with the diagnostic.
|
||||
- [ ] Compacted history: summary states "round N details unavailable
|
||||
due to compaction" instead of fabricating.
|
||||
|
||||
---
|
||||
|
||||
## §8. Version and verification log
|
||||
@@ -1274,6 +1707,8 @@ If §7.1–§7.5 do not produce the expected outputs:
|
||||
| 2026-04-17 | 0.121.0 | containerized sandbox (yantar-k8s) | external agent + lead | §7.1 `- < file` form fails EXIT=1 with empty stderr. `cat \| pipe` form works for `-o` review, but `--json` stdout is empty. Filesystem secondary session-id capture (§4.1b) verified functional: UUID extracted from rollout filename successfully resumes. Not a version issue (reproduced on 0.120.0 and 0.121.0). Root cause undiagnosed — see §6.6. Skill adapted: `§4.1` now two-tier, `§4.13` switches canonical form to `cat \| pipe`. |
|
||||
| 2026-04-17 | 0.121.0 | reference env (WSL2) | live dogfood + team review | Round 6: timestamp-only secondary (§4.1b as of round 5) flagged for silent wrong-session hazard against parallel codex. Verified empirically that rollout JSONL contains prompt text (3 matches of prompt content via grep). Replaced with positive content-binding: prompt marker `<!-- ADVERSARIAL-REVIEW-SESSION: ${REVIEW_ID} -->` + `find -newer <prompt> -exec grep -l <REVIEW_ID> {} +`. All flags POSIX — GNU-find dependency of earlier §9.5 goes away. See §6.7. |
|
||||
| 2026-04-17 | 0.121.0 | reference env (WSL2) | live dogfood round 2 | Round 7: review-stable marker flagged as insufficient — SKILL.md's own launch-retry flow can leave multiple rollouts matching the same `${REVIEW_ID}`, and "pick any" reintroduces silent intra-review session drift. Fixed by adding per-launch `${ATTEMPT_ID}` (6-digit random regenerated for every exec/retry/resume/fresh-exec). Marker is now `${REVIEW_ID}-${ATTEMPT_ID}`. Multi-match changed from "pick any" to fail-closed. See §6.8. |
|
||||
| 2026-05-20 | TBD (refresh-era ≥ 0.132.0 required for `-c approval_policy`) | reference env (WSL2) | refresh design + implementation | Default model bump `gpt-5.4 → gpt-5.5`; default sandbox `read-only → workspace-write`; approval policy expressed via `-c approval_policy='"on-request"'` + `-c approvals_reviewer='"auto_review"'` (`-a` flag dropped per Codex 0.132 removal); new overrides `sandbox:*` and `approvals:*`. Runner result schema extended with `review_quality` + `triage` object (back-compat: legacy results treated as `unknown` / skipped). New operation-aware dispatch table in §SKILL.md Step 4. Mandatory pre/post `git status --porcelain` + `sha256sum` snapshots on every dispatch; tracked-file or `/tmp` review-input mutation is a hard stop. Step 6 rewritten with evaluation matrix + structural operator gate (batch-pause rule). Step 7 resume body restructured into Applied / Re-scoped / Rejected with reasoning / Specific asks. Step 8 adds final operator summary in `OPERATOR_LANGUAGE` at every terminal state. New rationale entries §4.14–§4.21. README adds "Safety considerations" and "Linux sandbox prerequisites" sections. Smoke protocol §7 updated for the new flags. Verification: dogfood plan + code review against this refresh (see §7.8 once added). |
|
||||
| 2026-05-21 | 0.132.0 | reference env (WSL2) | dogfood rounds 1 & 2 (`/adversarial-review code` against this branch) | Round 1: applied #2 (worktree recipe loses unstaged), #3 (stale Full example permissions), #4 (bwrap diagnostic recommends `sandbox:read-only`); re-scoped #1 (porcelain misses content drift on already-dirty tracked files) to docs + runtime warning. Round 2 caught two structural issues: (a) the round-1 warning was placed in `SKILL.md` Step 1 before `REPO_ROOT` capture, making it non-functional — moved to Step 2 after capture; (b) the round-1 re-scope rationale cited `approval_policy=on-request` + `auto_review` as a protection layer for the residual gap, which is incorrect because workspace writes are sandbox-permitted and never generate approval requests — rationale withdrawn in §4.20, README "Safety considerations" rewritten to document the gap honestly. Already-dirty tracked-file content drift remains an accepted residual risk; mitigations are the `<reviewer_permissions>` prompt contract, the runtime warning, and the `sandbox:read-only` operator opt-out. |
|
||||
|
||||
When you re-verify (either during routine maintenance or when
|
||||
triggered by §7.7), add a row. Keep the log chronological.
|
||||
@@ -1484,7 +1919,7 @@ The attempt-scoped `ADVERSARIAL-REVIEW-SESSION` marker (round-7 finding) and pos
|
||||
**Hypothesis to test:** Plan Mode restrictions propagate from main to any subagent main dispatches; the subagent inherits limitations on Write/Edit/Bash. If this holds:
|
||||
- Main's Write to `/tmp/codex-body-*.md` may trigger a permission prompt or exit Plan Mode.
|
||||
- The runner's Writes to `/tmp/codex-prompt-*.md` (Step R2, including the mtime-bump repeat Write) may also trigger prompts.
|
||||
- The runner's `codex exec` (read-only sandbox) should be unaffected since it writes nothing to the user's repo.
|
||||
- The runner's `codex exec` (default `workspace-write` sandbox, governed by the `<reviewer_permissions>` prompt contract per §4.15) writes to `/tmp` for review artifacts but is bound by main's mutation snapshot (§4.20) against unintended `REPO_ROOT` writes.
|
||||
|
||||
**What Task 7 Step 4 must determine:**
|
||||
1. Does dispatching the Agent tool from Plan Mode work (is it blocked, does it prompt, does it just work)?
|
||||
|
||||
Reference in New Issue
Block a user