diff --git a/agents/.agents/skills/brainstorm-with-docs/ROADMAP.md b/agents/.agents/skills/brainstorm-with-docs/ROADMAP.md index 9c18db1..542e635 100644 --- a/agents/.agents/skills/brainstorm-with-docs/ROADMAP.md +++ b/agents/.agents/skills/brainstorm-with-docs/ROADMAP.md @@ -10,37 +10,94 @@ versioned so it survives across sessions and tools (not just an agent's memory). discipline, active grilling, option comparison, durable artifacts, and a stop-before-implementation gate that allows investigation but forbids state changes. -- **Next process step:** live test on real tasks (started ~2026-06-03). Findings - feed the decisions below. +- **First live test — 2026-06-06.** Real task: Superset dashboard redesign in the + clickstream repo (KPI semantics → CONTEXT.md glossary → spec + ADRs). Findings + landed below; the test report drives the v2 section. -## Pending decisions (small, near-term) +## Landed (2026-06-06, from first live test) -- **R1 — restore "recommend an answer per question."** `grill-with-docs` and - `grill-me` both require giving a recommended answer with each question; v1 - dropped it (likely accidental). Cheap to restore in `SKILL.md` step 4. - *Decision pending: restore now, or after the first live test.* -- **R2 — interview relentlessness.** v1 softened grill's "interview relentlessly, - walk every branch" into "don't interview forever, move to synthesis" (from - superpowers brainstorming). Decide after the live test whether it feels - under-rigorous and needs hardening — or whether that's what `grill-with-docs` - stays for (see R4). -- **F1 — divergence one-liner.** Add one ``: "among the options, at - least one must be something the user did not already imply." Deliberately - deferred to keep the v1 baseline test clean. Marked as `TODO(F1)` at the SEAM - in `SKILL.md` step 5. Reconsider right after the first live test. -- **F4 — relationship to `grill-with-docs`.** This skill overlaps it (grill + - CONTEXT/ADR). Decide: document when-to-use-which in `README.md`, or supersede - `grill-with-docs`. Not a strict superset (we lack full multi-context, R2). -- **F5a — read `docs/agents/*.md` in step 1.** On repos using Matt Pocock's - skill-config convention, the spec/layout path is more reliable from there than - guessing. Cheap addition to the step-1 read list. +- **R1 — recommend-an-answer-per-question — RESTORED** (`SKILL.md` step 4). + Evidence: per-question recommendations drove nearly every decision in the test; + bare questions would have been flat. The user confirmed the recommendation was + what he leaned on ("often hits the mark"). +- **F1 — divergence — ADDED** as a `` (and echoed in step 5). Final + form after discussion: offer 2-3 approaches *within* the framing **plus** at + least one (more if useful) that *challenges* the framing — the divergent option + is additive, not a stolen slot — with an explicit "don't pad to a number" guard. + Evidence: in the test, 2 of 3 options were variations of one idea; only the + consciously-pushed third was a different *kind* of answer. Rationale for keeping + it mandatory (not "preferably"): the user is the final filter, so a weak + divergent option costs ~nothing, but a *missing* one is invisible and can't be + picked or rewritten. +- **SEAM-bias — FIXED.** The verbose `SEAM/TODO` comment in `SKILL.md` step 5 + primed the running agent with the expected outcome ("we want to see how badly + pure convergence collapses") — contaminating the very observation it wanted. + Rationale moved here; `SKILL.md` now carries only a neutral marker. *Principle: + the shipped skill must not tell a running agent what result the author expects.* + (New finding, not previously in backlog.) -## v2 (larger) +## Pending decisions + +- **R2 — interview relentlessness. Verdict: leave SOFT.** In the live test the + interview ran long because the *user* kept opening new forks, not from + over-asking. The real value came from data/code cross-referencing ("never ask + what the repo can answer — search first"), not relentlessness. Keep + `grill-with-docs` for hard pressure-testing (see F4). +- **F4 — relationship to `grill-with-docs`.** Live-test boundary that worked: + **brainstorm** = when you need *options* and a direction (divergence→convergence); + **grill** = when you have *one* plan to pressure-test against docs. Action: + document this when-to-use-which in `README.md` (not supersede — not a strict + superset; lacks full multi-context). +- **F5a — read `docs/agents/*.md` in step 1.** Confirmed by the test: `AGENTS.md` + / ADRs were the key map (the doc-layout policy lived there, and reversing it + needed that context). Add to the step-1 read list. + +## v2 (larger) — divergence, informed by the first live test + +**Motivating failure from the test.** The dashboard brainstorm converged on "fix +the KPI strip" and never generated the option "maybe a KPI dashboard isn't the +best teaching vehicle at all" (a guided SQL notebook? a before/after comparison? a +deliberately-broken dashboard to fix?). The miss was at the **frame** level, not +the option level — F1 helps within the frame but doesn't question the frame +itself. v2 must address both levels. + +**Core tension (raised by the user).** Full divergence is time-expensive but +high-value. So v2 must be **tiered, not always-on** — spend divergence where it +pays. + +Proposals (priority order): + +1. **Tiered divergence (cost control).** Calibrate divergence effort to decision + weight at step 3 (scope). Trivial/reversible → F1 only (one non-implied + option). Heavy/irreversible/expensive-to-unwind → full toolkit pass. Directly + answers the cost worry: divergence proportional to rollback cost. +2. **Frame-level divergence (the test gap).** Before listing options, state the + user's implicit frame in one sentence, then generate ≥1 option that rejects + that frame. "Why a KPI dashboard at all?" is the poster example. Near-zero cost + (one sentence), catches frame-lock that F1 misses. +3. **Divergence-then-convergence as separate phases (the SEAM intent).** Generate + diverse options FIRST without evaluating, THEN judge/recommend. Mixing + generation and evaluation self-censors weird options before they're voiced. + Structural change, not just a rule. +4. **Idea-generation toolkit**, each technique with a WHEN-trigger so the agent + knows when to reach for it: + - *axis of variation* — name the 1–2 dimensions options vary along; ensure they + don't all sit on one axis. + - *archetype stances* — force an MVP / a robust / a radical option. + - *inversion* — "what if we did the opposite of the obvious?" + - *steal-from-adjacent-domain* — how does another field solve this? + - *pre-mortem* — "assume the chosen option failed in 6 months; what option would + we wish we'd considered?" + - *null option* — "what if we don't build it / do nothing?" +5. **Cheap wins (80/20, can land before the full toolkit):** always include the + null option + the one-sentence frame restatement. Near-zero cost, catch most + frame/convergence failures. +6. **Diversity self-check before presenting:** "Are these genuinely different kinds + of answer, or variations of one? If variations, generate one more from a + different axis." A cheap critic that enforces F1 harder. + +Also kept from before: -- **Divergence phase** at the SEAM (`SKILL.md` step 5): full idea-generation - toolkit — axis of variation, archetype stances (MVP / robust / radical), - inversion, steal-from-adjacent-domain, cost-shifting, pre-mortem, constraint - relaxation, null option. F1 is the cheap precursor to this. - **"Design for isolation / clarity" lens** (from superpowers brainstorming): evaluate options by whether each unit has one clear purpose and a clean interface. diff --git a/agents/.agents/skills/brainstorm-with-docs/SKILL.md b/agents/.agents/skills/brainstorm-with-docs/SKILL.md index 5f0a928..f8199a9 100644 --- a/agents/.agents/skills/brainstorm-with-docs/SKILL.md +++ b/agents/.agents/skills/brainstorm-with-docs/SKILL.md @@ -68,7 +68,11 @@ If invoked with arguments, treat `$ARGUMENTS` as the initial problem statement. * Surface assumptions, constraints, risks, trade-offs, and non-goals. * Don't interview forever — when the problem, constraints, and success criteria are clear enough, stop asking and synthesize. -* Always consider 2-3 viable approaches before recommending one. +* Offer 2-3 viable approaches within the user's framing, plus at least one (more + if useful) that challenges the framing itself — the null option, a reframe, an + adjacent-domain steal. Without the framing-challenger, options collapse into + variations of one idea. Don't pad to hit a number; every option must be + genuinely distinct. * Be direct. Give a real recommendation; don't hide behind "it depends" when the context supports a call. * Explicitly state what should NOT be done now. @@ -95,6 +99,11 @@ previously rejected options, deployment model, testing/validation conventions, and patterns the solution should follow. Respect a documented structure if one exists. If no context is available, say so and continue with what the user gives. +Investigation often needs read-only commands beyond the pre-approved git set — +listing files, inspecting data, querying a dev datastore. That's expected: run +them (they may prompt for approval), since grounding options in real data is this +skill's main strength. Never run state-changing commands (see the gate). + ## 2. Classify the work and pick the likely artifact Decide what kind of work this is, then which artifact is likely to fit: @@ -120,8 +129,11 @@ decisions. Ask which part to handle first. ## 4. Interview / grill Interview the user until the problem is clear enough to synthesize. One question -at a time. When the reason for a question isn't obvious, briefly say why it -matters. Prefer questions that clarify: the actual problem and why it matters, +at a time. For a decision or choice question, offer your recommended answer and a +one-line why — a concrete recommendation to react to moves the user further than a +bare question; for pure fact-gathering a recommendation may not apply. When the +reason for a question isn't obvious, briefly say why it matters. Prefer questions +that clarify: the actual problem and why it matters, expected outcome, who is affected, constraints, non-goals, success criteria, failure modes, operational and security risks, reversibility, deadlines or sequencing, what's been tried, and what must not change. @@ -138,25 +150,21 @@ Grill actively — don't just collect answers, pressure-test them: the code agrees; surface contradictions: "Your code cancels whole Orders, but you said partial cancellation is possible — which is right?" -Capture resolved terms in `CONTEXT.md` inline, as they happen, using the format -in [artifacts/context.md](artifacts/context.md) — don't batch them. +Capture a NEW resolved term in `CONTEXT.md` inline, as it happens (additive, low +risk), using the format in [artifacts/context.md](artifacts/context.md) — don't +batch them. Redefining an already-documented term is a vocabulary change: propose +it inline, but confirm before writing (see step 6). ## 5. Synthesize options - + -Present 2-3 viable approaches. For each: short description, when it fits, -benefits, trade-offs, risks, operational implications, documentation impact, and -what it avoids or postpones. Then recommend one, explaining why it fits the +Present 2-3 viable approaches within the user's framing, plus at least one that +challenges it. For each: short description, when it fits, benefits, trade-offs, +risks, operational implications, documentation impact, and what it avoids or +postpones. Then recommend one, explaining why it fits the constraints, what risks remain, what to validate first, and what to explicitly not do now.