docs(brainstorm-with-docs): итоги первого живого теста — R1/F1/SEAM + план v2
- Зачем:
- первый живой тест скилла вскрыл правки, которые надо закрепить, и идеи для v2.
- Что:
- SKILL.md: R1 (рекомендация к вопросам-решениям), F1 (2-3 внутри рамки + ≥1
дивергентный сверх, без филлера), inline/confirm для терминов, read-only
инспекция данных в шаге 1, нейтральная SEAM-метка.
- ROADMAP.md: статус первого теста, Landed R1/F1/SEAM, вердикты R2/F4/F5a, план v2.
- Проверка:
- прочитать SKILL.md (core_rules, шаги 1/4/5) и ROADMAP.md.
This commit is contained in:
ddadmin committed
1 parent
f3a41e2790
commit
a1fe4ce319
2 files changed
+110
-45
No files matched your search
@@ -10,37 +10,94 @@ versioned so it survives across sessions and tools (not just an agent's memory).
|
||||
discipline, active grilling, option comparison, durable artifacts, and a
|
||||
stop-before-implementation gate that allows investigation but forbids state
|
||||
changes.
|
||||
- **Next process step:** live test on real tasks (started ~2026-06-03). Findings
|
||||
feed the decisions below.
|
||||
- **First live test — 2026-06-06.** Real task: Superset dashboard redesign in the
|
||||
clickstream repo (KPI semantics → CONTEXT.md glossary → spec + ADRs). Findings
|
||||
landed below; the test report drives the v2 section.
|
||||
|
||||
## Pending decisions (small, near-term)
|
||||
## Landed (2026-06-06, from first live test)
|
||||
|
||||
- **R1 — restore "recommend an answer per question."** `grill-with-docs` and
|
||||
`grill-me` both require giving a recommended answer with each question; v1
|
||||
dropped it (likely accidental). Cheap to restore in `SKILL.md` step 4.
|
||||
*Decision pending: restore now, or after the first live test.*
|
||||
- **R2 — interview relentlessness.** v1 softened grill's "interview relentlessly,
|
||||
walk every branch" into "don't interview forever, move to synthesis" (from
|
||||
superpowers brainstorming). Decide after the live test whether it feels
|
||||
under-rigorous and needs hardening — or whether that's what `grill-with-docs`
|
||||
stays for (see R4).
|
||||
- **F1 — divergence one-liner.** Add one `<core_rule>`: "among the options, at
|
||||
least one must be something the user did not already imply." Deliberately
|
||||
deferred to keep the v1 baseline test clean. Marked as `TODO(F1)` at the SEAM
|
||||
in `SKILL.md` step 5. Reconsider right after the first live test.
|
||||
- **F4 — relationship to `grill-with-docs`.** This skill overlaps it (grill +
|
||||
CONTEXT/ADR). Decide: document when-to-use-which in `README.md`, or supersede
|
||||
`grill-with-docs`. Not a strict superset (we lack full multi-context, R2).
|
||||
- **F5a — read `docs/agents/*.md` in step 1.** On repos using Matt Pocock's
|
||||
skill-config convention, the spec/layout path is more reliable from there than
|
||||
guessing. Cheap addition to the step-1 read list.
|
||||
- **R1 — recommend-an-answer-per-question — RESTORED** (`SKILL.md` step 4).
|
||||
Evidence: per-question recommendations drove nearly every decision in the test;
|
||||
bare questions would have been flat. The user confirmed the recommendation was
|
||||
what he leaned on ("often hits the mark").
|
||||
- **F1 — divergence — ADDED** as a `<core_rule>` (and echoed in step 5). Final
|
||||
form after discussion: offer 2-3 approaches *within* the framing **plus** at
|
||||
least one (more if useful) that *challenges* the framing — the divergent option
|
||||
is additive, not a stolen slot — with an explicit "don't pad to a number" guard.
|
||||
Evidence: in the test, 2 of 3 options were variations of one idea; only the
|
||||
consciously-pushed third was a different *kind* of answer. Rationale for keeping
|
||||
it mandatory (not "preferably"): the user is the final filter, so a weak
|
||||
divergent option costs ~nothing, but a *missing* one is invisible and can't be
|
||||
picked or rewritten.
|
||||
- **SEAM-bias — FIXED.** The verbose `SEAM/TODO` comment in `SKILL.md` step 5
|
||||
primed the running agent with the expected outcome ("we want to see how badly
|
||||
pure convergence collapses") — contaminating the very observation it wanted.
|
||||
Rationale moved here; `SKILL.md` now carries only a neutral marker. *Principle:
|
||||
the shipped skill must not tell a running agent what result the author expects.*
|
||||
(New finding, not previously in backlog.)
|
||||
|
||||
## v2 (larger)
|
||||
## Pending decisions
|
||||
|
||||
- **R2 — interview relentlessness. Verdict: leave SOFT.** In the live test the
|
||||
interview ran long because the *user* kept opening new forks, not from
|
||||
over-asking. The real value came from data/code cross-referencing ("never ask
|
||||
what the repo can answer — search first"), not relentlessness. Keep
|
||||
`grill-with-docs` for hard pressure-testing (see F4).
|
||||
- **F4 — relationship to `grill-with-docs`.** Live-test boundary that worked:
|
||||
**brainstorm** = when you need *options* and a direction (divergence→convergence);
|
||||
**grill** = when you have *one* plan to pressure-test against docs. Action:
|
||||
document this when-to-use-which in `README.md` (not supersede — not a strict
|
||||
superset; lacks full multi-context).
|
||||
- **F5a — read `docs/agents/*.md` in step 1.** Confirmed by the test: `AGENTS.md`
|
||||
/ ADRs were the key map (the doc-layout policy lived there, and reversing it
|
||||
needed that context). Add to the step-1 read list.
|
||||
|
||||
## v2 (larger) — divergence, informed by the first live test
|
||||
|
||||
**Motivating failure from the test.** The dashboard brainstorm converged on "fix
|
||||
the KPI strip" and never generated the option "maybe a KPI dashboard isn't the
|
||||
best teaching vehicle at all" (a guided SQL notebook? a before/after comparison? a
|
||||
deliberately-broken dashboard to fix?). The miss was at the **frame** level, not
|
||||
the option level — F1 helps within the frame but doesn't question the frame
|
||||
itself. v2 must address both levels.
|
||||
|
||||
**Core tension (raised by the user).** Full divergence is time-expensive but
|
||||
high-value. So v2 must be **tiered, not always-on** — spend divergence where it
|
||||
pays.
|
||||
|
||||
Proposals (priority order):
|
||||
|
||||
1. **Tiered divergence (cost control).** Calibrate divergence effort to decision
|
||||
weight at step 3 (scope). Trivial/reversible → F1 only (one non-implied
|
||||
option). Heavy/irreversible/expensive-to-unwind → full toolkit pass. Directly
|
||||
answers the cost worry: divergence proportional to rollback cost.
|
||||
2. **Frame-level divergence (the test gap).** Before listing options, state the
|
||||
user's implicit frame in one sentence, then generate ≥1 option that rejects
|
||||
that frame. "Why a KPI dashboard at all?" is the poster example. Near-zero cost
|
||||
(one sentence), catches frame-lock that F1 misses.
|
||||
3. **Divergence-then-convergence as separate phases (the SEAM intent).** Generate
|
||||
diverse options FIRST without evaluating, THEN judge/recommend. Mixing
|
||||
generation and evaluation self-censors weird options before they're voiced.
|
||||
Structural change, not just a rule.
|
||||
4. **Idea-generation toolkit**, each technique with a WHEN-trigger so the agent
|
||||
knows when to reach for it:
|
||||
- *axis of variation* — name the 1–2 dimensions options vary along; ensure they
|
||||
don't all sit on one axis.
|
||||
- *archetype stances* — force an MVP / a robust / a radical option.
|
||||
- *inversion* — "what if we did the opposite of the obvious?"
|
||||
- *steal-from-adjacent-domain* — how does another field solve this?
|
||||
- *pre-mortem* — "assume the chosen option failed in 6 months; what option would
|
||||
we wish we'd considered?"
|
||||
- *null option* — "what if we don't build it / do nothing?"
|
||||
5. **Cheap wins (80/20, can land before the full toolkit):** always include the
|
||||
null option + the one-sentence frame restatement. Near-zero cost, catch most
|
||||
frame/convergence failures.
|
||||
6. **Diversity self-check before presenting:** "Are these genuinely different kinds
|
||||
of answer, or variations of one? If variations, generate one more from a
|
||||
different axis." A cheap critic that enforces F1 harder.
|
||||
|
||||
Also kept from before:
|
||||
|
||||
- **Divergence phase** at the SEAM (`SKILL.md` step 5): full idea-generation
|
||||
toolkit — axis of variation, archetype stances (MVP / robust / radical),
|
||||
inversion, steal-from-adjacent-domain, cost-shifting, pre-mortem, constraint
|
||||
relaxation, null option. F1 is the cheap precursor to this.
|
||||
- **"Design for isolation / clarity" lens** (from superpowers brainstorming):
|
||||
evaluate options by whether each unit has one clear purpose and a clean
|
||||
interface.
|
||||
|
||||
Reference in new issue
Block a user