feat: review-loop-run — detached MiniMax review launcher for /review-loop artefacts
launch (per-kind prompt templates: verify/plan/spec/tests/code; best-practice + gotcha reading sets; setsid nohup launcher; round N+1 feeds back round N), status, counts (exit 4 on CRITICAL/HIGH). Spec + 29 dryrun tests. Claude-Session: https://claude.ai/code/session_018LRh1S2bHJYhEgrMu3UJx8
This commit is contained in:
Paul O'Reilly
@@ -22,6 +22,7 @@ This project uses **spec-driven development** (OpenSpec) as a testbed for agent-
|
||||
| `git-status-report` | Recursive git repo status with diff stats | Done |
|
||||
| `md-to-docx` | Markdown to DOCX conversion | Done |
|
||||
| `mm-worktree-task` | Launch a MiniMax M3 implementation sub-session in an isolated git worktree, prompt assembled from a planner work item | Done |
|
||||
| `review-loop-run` | Launch/track detached MiniMax reviews of `/review-loop` artefacts (verify, plan, spec, tests, code) from per-kind prompt templates; `counts` gates on CRITICAL/HIGH | Done |
|
||||
| `mp3-to-mp4` | Convert MP3 to MP4 with a static title card | Done |
|
||||
| `split-wezterm` | Split WezTerm pane into a rows × cols grid | Done |
|
||||
| `sync-repos` | Mirror git repos across remotes | Done |
|
||||
|
||||
260
scripts/review-loop-run
Executable file
260
scripts/review-loop-run
Executable file
@@ -0,0 +1,260 @@
|
||||
#!/usr/bin/env bash
|
||||
# review-loop-run — launch/track detached MiniMax reviews of review-loop artefacts.
|
||||
# Spec: specs/review-loop-run.spec.md
|
||||
set -euo pipefail
|
||||
|
||||
BLUE='\033[0;34m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; RED='\033[0;31m'; NC='\033[0m'
|
||||
info() { printf "${BLUE}==>${NC} %s\n" "$*"; }
|
||||
warn() { printf "${YELLOW}warning:${NC} %s\n" "$*" >&2; }
|
||||
die() { printf "${RED}error:${NC} %s\n" "$*" >&2; exit "${2:-1}"; }
|
||||
|
||||
BP_DIR="${REVIEW_LOOP_BP_DIR:-$HOME/dev/claude/projects/best-practices}"
|
||||
KINDS="plan spec tests code verify"
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
Usage:
|
||||
review-loop-run launch --kind <plan|spec|tests|code|verify> --name <tag> [--round N]
|
||||
--file <path> [--file <path> ...] [--context "<text>"]
|
||||
[--prev <result.md>] [--tools "<extra>"] [--timeout <secs>]
|
||||
[--project <repo>] [--dryrun|-n]
|
||||
review-loop-run status [--name <tag>] [--round N] [--project <repo>]
|
||||
review-loop-run counts <result.md>
|
||||
review-loop-run --help
|
||||
|
||||
Launches ask-minimax-run detached (setsid nohup) with a kind-specific review
|
||||
prompt; files land in ${XDG_RUNTIME_DIR:-/tmp}/review-loop/<repo>/<tag>-r<N>-{prompt.txt,launch.sh,stdout.txt,result.md}.
|
||||
counts exits 4 when CRITICAL+HIGH > 0.
|
||||
EOF
|
||||
}
|
||||
|
||||
resolve_repo() {
|
||||
local p="${1:-}"
|
||||
if [[ -n "$p" ]]; then cd "$p"; fi
|
||||
git rev-parse --show-toplevel 2>/dev/null || pwd
|
||||
}
|
||||
|
||||
report_dir_for() {
|
||||
local repo="$1"
|
||||
echo "${REVIEW_LOOP_RUN_DIR:-${XDG_RUNTIME_DIR:-/tmp}/review-loop/$(basename "$repo")}"
|
||||
}
|
||||
|
||||
# counts <file> → prints CRITICAL=n HIGH=n MEDIUM=n LOW=n ; exit 4 if C+H>0
|
||||
counts() {
|
||||
local f="$1"
|
||||
[[ -f "$f" ]] || die "result file not found: $f" 2
|
||||
local out="" total=0
|
||||
for sev in CRITICAL HIGH MEDIUM LOW; do
|
||||
local n
|
||||
n=$(grep -iE "^\| *${sev} *\| *[0-9]+" "$f" | head -1 | sed -E 's/^\| *[A-Za-z]+ *\| *([0-9]+).*/\1/') || true
|
||||
if [[ -z "$n" ]]; then
|
||||
n=$(grep -iE "^[-*]* *\**${sev}:?\**:? *\**([0-9]+)" "$f" | head -1 | sed -E 's/.*[^0-9]([0-9]+)[^0-9]*$/\1/') || true
|
||||
fi
|
||||
if [[ -z "$n" ]]; then
|
||||
n=$(grep -cE "^#{2,4} .*\b${sev}\b" "$f") || true
|
||||
fi
|
||||
n="${n:-0}"; out+="${sev}=${n} "
|
||||
if [[ "$sev" == "CRITICAL" || "$sev" == "HIGH" ]]; then total=$((total + n)); fi
|
||||
done
|
||||
echo "$out"
|
||||
if [[ "$total" -gt 0 ]]; then return 4; fi
|
||||
return 0
|
||||
}
|
||||
|
||||
kind_task() {
|
||||
case "$1" in
|
||||
verify) cat <<'EOF'
|
||||
Verify whether the planner work item described in the first listed file is STILL an open, unresolved issue in the current code on main, or whether it has already been fixed/implemented and the record is merely stale. Extract the concrete claim, inspect the current source with Grep/Glob/Read and `git log -S`/`git show`, and decide from the CODE, not the item's own text.
|
||||
|
||||
Output format:
|
||||
- **Item:** <uuid> — <title>
|
||||
- **Verdict:** RESOLVED / STILL OPEN / PARTIALLY RESOLVED / CANNOT DETERMINE
|
||||
- **Evidence:** 3-8 bullets, each citing file:line or a commit sha
|
||||
- **Remaining work (if any):** concise
|
||||
- **Confidence:** high/medium/low
|
||||
EOF
|
||||
;;
|
||||
plan) cat <<'EOF'
|
||||
Independent, adversarial critical review of a program/implementation plan. For every finding give: severity (CRITICAL/HIGH/MEDIUM/LOW), the plan section, what is wrong, which best-practice/gotcha it violates (file + heading), and a concrete fix. Look for missing requirements, unsafe sequencing, wrong assumptions about the code (verify against source with Grep/Read), untestable statements, security gaps, ID collisions with existing requirement IDs, and anything the gotcha files say has bitten before. Verify every cited file:line and requirement ID exists. Do not pad with praise. End with a count by severity.
|
||||
EOF
|
||||
;;
|
||||
spec) cat <<'EOF'
|
||||
Independent critical spec review. The amendment under review is the unified diff (or spec section) in the listed file(s); the working tree already contains the amended text. Check: ambiguity; untestable statements; missing error/edge cases; contradictions with existing requirements in the same spec; test handles that cannot be derived mechanically (mechanical-test-generation.md); requirement-ID collisions; security gaps; cross-module seams pinned to real signatures and attribute names (verify against the source). OUT OF SCOPE: reporting that a symbol/module/field named by the spec "does not exist yet" — this is a spec-ahead artefact and new symbols are marked NEW in its module-layout rows. For each finding: severity, requirement ID, what is wrong, the best-practice/gotcha violated (file + heading), concrete fix. No praise. End with a count by severity.
|
||||
EOF
|
||||
;;
|
||||
tests) cat <<'EOF'
|
||||
Independent critical review of spec-ahead TEST files. Tests for not-yet-implemented behaviour are @pytest.mark.xfail(strict=True); that is intended. Check: (1) every spec test handle exists with the exact name and asserts the requirement's normative behaviour (not merely something already true today); (2) fixtures match the spec's pinned shapes; (3) assertions would pass against a CORRECT implementation — flag wrong attribute names, wrong status codes/envelopes, mocking the wrong target, invented private helper names, fixtures built with object.__new__ that skip __init__ attributes, assertions that xfail today for an unrelated reason (permanent false negatives); (4) test isolation and mock hygiene per test-driven-development.md and gotchas-python.md (AsyncMock on sync calls, TestClient lifespan, shared state); (5) xfail reasons name the right requirement. Severity CRITICAL/HIGH/MEDIUM/LOW; cite file:line; concrete fix per finding; no praise; end with a count by severity.
|
||||
EOF
|
||||
;;
|
||||
code) cat <<'EOF'
|
||||
Independent critical CODE review of an implementation branch. The diff is in the listed diff file (branch vs main); read source in the worktree/branch named in the context, not the main checkout. Review the implementation against the normative spec text and the tests (the acceptance criteria; the implementer was forbidden from editing tests/ or spec/ — confirm the diff touches neither). Check: (1) every requirement clause is implemented as written (cite spec sentence vs code); (2) correctness bugs, edge cases, fail-open paths, exception swallowing, async misuse, mutation of shared state, ordering relative to the application lifespan (memoised state created before its dependencies are wired); (3) security (input validation, RBAC/identity handling, error leakage per security-architecture.md and llm-code-security.md); (4) gotchas-python.md patterns (Pydantic extra=ignore drops, mock-target divergence, logging reserved attrs, sync/async); (5) unintended behaviour changes to existing call sites and any wire-shape change (request/response fields, error codes, log formats); (6) code quality: duplication, dead code, docstrings referencing spec IDs. Severity CRITICAL/HIGH/MEDIUM/LOW; cite file:line; concrete fix per finding; no praise; end with a count by severity.
|
||||
EOF
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
kind_defaults() {
|
||||
local repo="$1"
|
||||
case "$2" in
|
||||
verify) : ;;
|
||||
plan) printf '%s\n' "$BP_DIR/spec-driven-development.md" "$BP_DIR/test-driven-development.md" "$BP_DIR/security-architecture.md" "$BP_DIR/api-design.md" "$BP_DIR/validation.md" "$BP_DIR/kubernetes.md" "$repo"/memory/gotchas-*.md "$repo/memory/process-lessons.md" ;;
|
||||
spec) printf '%s\n' "$BP_DIR/spec-driven-development.md" "$BP_DIR/test-driven-development.md" "$BP_DIR/mechanical-test-generation.md" "$BP_DIR/api-design.md" "$BP_DIR/security-architecture.md" "$repo/memory/gotchas-workflow.md" "$repo/memory/gotchas-specs.md" "$repo/memory/gotchas-python.md" ;;
|
||||
tests) printf '%s\n' "$BP_DIR/test-driven-development.md" "$BP_DIR/mechanical-test-generation.md" "$repo/memory/gotchas-python.md" ;;
|
||||
code) printf '%s\n' "$BP_DIR/security-architecture.md" "$BP_DIR/llm-code-security.md" "$BP_DIR/python-patterns.md" "$repo/memory/gotchas-python.md" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
kind_tools() {
|
||||
local base="Read,Write,Glob,Grep,Bash(git log *),Bash(git grep *)"
|
||||
case "$1" in
|
||||
verify) echo "Read,Write,Glob,Grep,Bash(git log *),Bash(git show *),Bash(git grep *),Bash(git diff *)" ;;
|
||||
tests) echo "$base,Bash(python3 -m pytest *)" ;;
|
||||
code) echo "$base,Bash(git diff *)" ;;
|
||||
*) echo "$base" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
cmd_launch() {
|
||||
local kind="" name="" round=1 context="" prev="" tools="" timeout=1500 project="" dryrun=0
|
||||
local -a files=()
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--kind) kind="$2"; shift 2 ;;
|
||||
--name) name="$2"; shift 2 ;;
|
||||
--round) round="$2"; shift 2 ;;
|
||||
--file) files+=("$2"); shift 2 ;;
|
||||
--context) context="$2"; shift 2 ;;
|
||||
--prev) prev="$2"; shift 2 ;;
|
||||
--tools) tools="$2"; shift 2 ;;
|
||||
--timeout) timeout="$2"; shift 2 ;;
|
||||
--project) project="$2"; shift 2 ;;
|
||||
--dryrun|-n) dryrun=1; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) die "unknown option: $1" 2 ;;
|
||||
esac
|
||||
done
|
||||
[[ " $KINDS " == *" $kind "* ]] || die "--kind must be one of: $KINDS" 2
|
||||
[[ "$name" =~ ^[A-Za-z0-9._-]+$ ]] || die "--name must match ^[A-Za-z0-9._-]+\$" 2
|
||||
[[ "$round" =~ ^[0-9]+$ ]] || die "--round must be an integer" 2
|
||||
[[ ${#files[@]} -ge 1 ]] || die "at least one --file is required" 2
|
||||
for f in "${files[@]}"; do [[ -f "$f" ]] || die "--file not found: $f" 2; done
|
||||
command -v ask-minimax-run >/dev/null || die "ask-minimax-run not on PATH"
|
||||
[[ -f "$HOME/.claude-oreillyit-minimax/provider.env" ]] || die "MiniMax profile missing (~/.claude-oreillyit-minimax/provider.env)"
|
||||
|
||||
local repo dir tag
|
||||
repo=$(resolve_repo "$project"); dir=$(report_dir_for "$repo"); tag="${name}-r${round}"
|
||||
local prompt="$dir/$tag-prompt.txt" launch="$dir/$tag-launch.sh" stdout="$dir/$tag-stdout.txt" result="$dir/$tag-result.md"
|
||||
|
||||
# previous round
|
||||
local prev_para=""
|
||||
if [[ -z "$prev" && "$round" -ge 2 ]]; then
|
||||
local cand="$dir/${name}-r$((round-1))-result.md"
|
||||
[[ -f "$cand" ]] && prev="$cand"
|
||||
fi
|
||||
if [[ "$round" -ge 2 ]]; then
|
||||
if [[ -n "$prev" && -f "$prev" ]]; then
|
||||
prev_para="This is ROUND ${round}. The previous round's findings are in ${prev} and were adjudicated by the supervisor (accepted findings were applied; rejected findings carry a source-cited reason on the artefact). Read that file; do not re-raise resolved or rejected findings unless you have NEW evidence; report only CRITICAL/HIGH plus any MEDIUM that blocks mechanical test derivation."
|
||||
else
|
||||
warn "round $round but no previous result found; proceeding without the adjudication paragraph"
|
||||
fi
|
||||
fi
|
||||
|
||||
# reading list
|
||||
local -a reading=("${files[@]}")
|
||||
[[ -n "$prev" && -f "$prev" ]] && reading+=("$prev")
|
||||
local -a defaults=() missing=()
|
||||
while IFS= read -r d; do
|
||||
[[ -z "$d" ]] && continue
|
||||
if [[ -f "$d" ]]; then defaults+=("$d"); else missing+=("$d"); fi
|
||||
done < <(kind_defaults "$repo" "$kind")
|
||||
reading+=("${defaults[@]}")
|
||||
|
||||
local allow; allow=$(kind_tools "$kind"); [[ -n "$tools" ]] && allow="$allow,$tools"
|
||||
|
||||
local body
|
||||
body=$(cat <<EOF
|
||||
You are running in non-interactive (-p) mode. Complete the task and write your full response to the output file below using the Write tool. Do not rely on stdout for structured output.
|
||||
|
||||
## Task
|
||||
$(kind_task "$kind")
|
||||
${context:+
|
||||
Context: $context}
|
||||
${prev_para:+
|
||||
$prev_para}
|
||||
|
||||
## Files to read
|
||||
$(printf '%s\n' "${reading[@]}")
|
||||
|
||||
## Output file
|
||||
Write to: $result
|
||||
|
||||
Use the Write tool to create that file. Overwrite if it already exists.
|
||||
|
||||
## Constraints
|
||||
- Read every listed file fully before writing.
|
||||
- Cite file:line or commit sha for every claim.
|
||||
- Do not modify any repository files. Only write the output file.
|
||||
EOF
|
||||
)
|
||||
local launch_body
|
||||
launch_body=$(cat <<EOF
|
||||
#!/usr/bin/env bash
|
||||
cd "$repo" || exit 1
|
||||
timeout ${timeout} ask-minimax-run --model MiniMax-M3 --allowedTools "$allow" --permission-mode acceptEdits -p "\$(cat "$prompt")" > "$stdout" 2>&1
|
||||
echo "exit=\$?" >> "$stdout"
|
||||
EOF
|
||||
)
|
||||
if [[ "$dryrun" -eq 1 ]]; then
|
||||
printf "${YELLOW}[dryrun]${NC} review-loop-run — nothing will be launched\n"
|
||||
echo " Repo: $repo"; echo " Report dir: $dir"; echo " Kind/round: $kind r$round"; echo " Allowlist: $allow"
|
||||
for m in "${missing[@]:-}"; do [[ -n "$m" ]] && echo " default file absent, skipped: $m"; done
|
||||
echo "--- prompt ---"; echo "$body"; echo "--- launch command ---"; echo "$launch_body"
|
||||
echo "[dryrun] nothing launched"; return 0
|
||||
fi
|
||||
mkdir -p "$dir"
|
||||
printf '%s\n' "$body" > "$prompt"
|
||||
printf '%s\n' "$launch_body" > "$launch"; chmod +x "$launch"
|
||||
rm -f "$stdout" "$result"
|
||||
setsid nohup "$launch" >/dev/null 2>&1 &
|
||||
info "launched $kind review $tag"
|
||||
echo " prompt: $prompt"; echo " stdout: $stdout"; echo " result: $result"
|
||||
echo " poll: review-loop-run status --name $name --round $round"
|
||||
}
|
||||
|
||||
cmd_status() {
|
||||
local name="" round="" project=""
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--name) name="$2"; shift 2 ;;
|
||||
--round) round="$2"; shift 2 ;;
|
||||
--project) project="$2"; shift 2 ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) die "unknown option: $1" 2 ;;
|
||||
esac
|
||||
done
|
||||
local repo dir; repo=$(resolve_repo "$project"); dir=$(report_dir_for "$repo")
|
||||
local pat="${name:-*}-r${round:-*}-stdout.txt"
|
||||
local any=0
|
||||
for so in "$dir"/$pat; do
|
||||
[[ -f "$so" ]] || continue
|
||||
any=1
|
||||
local tag="${so##*/}"; tag="${tag%-stdout.txt}"
|
||||
local res="$dir/$tag-result.md" state="running" c=""
|
||||
if tail -1 "$so" 2>/dev/null | grep -q '^exit='; then state="done $(tail -1 "$so")"; fi
|
||||
if [[ -f "$res" ]]; then c=$(counts "$res" 2>/dev/null || true); [[ -z "$c" ]] && c=$(counts "$res" 2>/dev/null; true); else c="(no result yet)"; fi
|
||||
printf "%-28s %-14s %s\n" "$tag" "$state" "$c"
|
||||
done
|
||||
[[ "$any" -eq 1 ]] || echo "no runs in $dir"
|
||||
}
|
||||
|
||||
main() {
|
||||
[[ $# -ge 1 ]] || { usage; exit 2; }
|
||||
case "$1" in
|
||||
launch) shift; cmd_launch "$@" ;;
|
||||
status) shift; cmd_status "$@" ;;
|
||||
counts) shift; [[ $# -ge 1 ]] || die "counts <result.md>" 2; counts "$1" ;;
|
||||
-h|--help) usage ;;
|
||||
*) die "unknown subcommand: $1" 2 ;;
|
||||
esac
|
||||
}
|
||||
main "$@"
|
||||
93
specs/review-loop-run.spec.md
Normal file
93
specs/review-loop-run.spec.md
Normal file
@@ -0,0 +1,93 @@
|
||||
# review-loop-run
|
||||
|
||||
## Purpose
|
||||
|
||||
Launch a detached MiniMax M3 review of a review-loop artefact (plan, spec diff, test files, code diff, or a "is this bug still real" verification) with a kind-specific prompt built from a fixed template, and report its status/severity counts — so the `/review-loop` skill never hand-writes prompt files, launcher scripts, or `pkill`s.
|
||||
|
||||
## Usage
|
||||
|
||||
```
|
||||
review-loop-run launch --kind <plan|spec|tests|code|verify> --name <tag> [--round N] \
|
||||
--file <path> [--file <path> ...] [--context "<text>"] [--prev <result.md>] \
|
||||
[--tools "<extra allowedTools>"] [--timeout <secs>] [--dryrun|-n]
|
||||
review-loop-run status [--name <tag>] [--round N]
|
||||
review-loop-run counts <result.md>
|
||||
review-loop-run --help
|
||||
```
|
||||
|
||||
### Subcommands
|
||||
|
||||
| Subcommand | Description |
|
||||
|---|---|
|
||||
| `launch` | Build the prompt, write it under the report dir, start `ask-minimax-run` detached via `setsid nohup`, print the prompt/stdout/result paths, exit 0 immediately. |
|
||||
| `status` | For every `<tag>-r<N>` in the report dir (or the one named), print `running`/`done exit=N`, the result file's existence, and the CRITICAL/HIGH/MEDIUM/LOW counts parsed from it. Exit 0. |
|
||||
| `counts` | Print `CRITICAL=<n> HIGH=<n> MEDIUM=<n> LOW=<n>` for one result file. Exit 0 when CRITICAL+HIGH == 0, else exit 4. |
|
||||
|
||||
### Options (launch)
|
||||
|
||||
| Flag | Default | Description |
|
||||
|---|---|---|
|
||||
| `--kind` | required | Selects the prompt template and default best-practices set (§Templates). |
|
||||
| `--name` | required | Tag for this artefact (e.g. `spec-BC`, `code-D`, `verify-02a8436d`). |
|
||||
| `--round` | 1 | Round number; appended to file names as `-r<N>`. Rounds ≥2 automatically add the previous round's result file (`<tag>-r<N-1>-result.md`, if present) to the reading list and a "do not re-raise adjudicated findings" instruction. |
|
||||
| `--file` | ≥1 required | Files the reviewer must read fully (diff files, test files, specs, item docs). Repeatable. |
|
||||
| `--context` | — | One paragraph of task context (what the artefact closes, relevant source paths). |
|
||||
| `--prev` | auto | Explicit previous-round result file (overrides the auto lookup). |
|
||||
| `--tools` | kind default | Extra `--allowedTools` entries (comma-separated). |
|
||||
| `--timeout` | 1500 | Seconds before the sub-session is killed. |
|
||||
| `--project` | current repo root | Repo to `cd` into before launching. |
|
||||
| `--dryrun`, `-n` | off | Print the assembled prompt and the exact launch command; write nothing; launch nothing. |
|
||||
|
||||
### Environment
|
||||
|
||||
| Var | Effect |
|
||||
|---|---|
|
||||
| `REVIEW_LOOP_RUN_DIR` | Report dir root. Default `${XDG_RUNTIME_DIR:-/tmp}/review-loop/<repo-basename>`. |
|
||||
| `REVIEW_LOOP_BP_DIR` | Best-practices repo. Default `~/dev/claude/projects/best-practices`. |
|
||||
|
||||
## Behaviour
|
||||
|
||||
1. Resolve the repo root (`git rev-parse --show-toplevel`, or `--project`); derive the report dir; `mkdir -p`.
|
||||
2. Validate: kind in the closed set, name matches `^[A-Za-z0-9._-]+$`, every `--file` exists, `ask-minimax-run` on PATH, MiniMax profile present.
|
||||
3. Assemble the prompt from the kind template (§Templates): header (non-interactive, write to the output file), task text, `--context`, the "Files to read" list = `--file` entries + the kind's default best-practices/gotcha files that exist + the previous-round result (rounds ≥2), the output path, constraints.
|
||||
4. Write `<dir>/<name>-r<N>-prompt.txt`.
|
||||
5. Write `<dir>/<name>-r<N>-launch.sh` (a tiny bash script that `cd`s to the repo, runs `timeout <secs> ask-minimax-run --model MiniMax-M3 --allowedTools "<list>" --permission-mode acceptEdits -p "$(cat prompt)" > stdout 2>&1`, then appends `exit=$?` to the stdout file) and starts it with `setsid nohup <launch.sh> >/dev/null 2>&1 &`.
|
||||
6. Print the three paths and `launched`.
|
||||
|
||||
`status` scans `<dir>/*-r*-stdout.txt`; a file whose last line matches `^exit=` is `done`, else `running`; counts come from `counts`.
|
||||
|
||||
`counts` parses a result file with three fallbacks: a markdown table row `| CRITICAL | <n>`, a list line `- **CRITICAL:** <n>` / `CRITICAL: <n>`, or (failing both) the number of headings containing `CRITICAL`/`HIGH` — the same for HIGH/MEDIUM/LOW.
|
||||
|
||||
## Templates (per `--kind`)
|
||||
|
||||
All prompts share: severity scheme CRITICAL/HIGH/MEDIUM/LOW, `file:line` citations, a concrete fix per finding, "no praise", a count-by-severity table at the end, "read every listed file fully; do not modify repository files".
|
||||
|
||||
| kind | task text | default reading set | allowlist |
|
||||
|---|---|---|---|
|
||||
| `verify` | "Is the planner item in the first file still an open defect on main? Verdict RESOLVED / STILL OPEN / PARTIAL / CANNOT DETERMINE with cited evidence; remaining work." | `memory/gotchas-*.md` not needed; adds `git log` access | `Read,Write,Glob,Grep,Bash(git log *),Bash(git show *),Bash(git grep *),Bash(git diff *)` |
|
||||
| `plan` | adversarial plan critique: missing requirements, unsafe sequencing, wrong code assumptions (verify), untestable statements, security gaps, prior gotchas | spec-driven-development, test-driven-development, security-architecture, api-design, validation, kubernetes + project `memory/gotchas-*.md`, `memory/process-lessons.md` | `Read,Write,Glob,Grep,Bash(git log *),Bash(git grep *)` |
|
||||
| `spec` | spec amendment review: ambiguity, untestable statements, contradictions with existing requirements, non-derivable test handles, ID collisions, security; **the "symbol/module does not exist yet" class is out of scope for spec-ahead specs** | spec-driven-development, test-driven-development, mechanical-test-generation, api-design, security-architecture, gotchas-workflow, gotchas-specs, gotchas-python | same as plan |
|
||||
| `tests` | spec-ahead test review: every handle present and asserting normative behaviour, fixtures match pinned shapes, assertions would pass against a correct implementation, mock hygiene, fixtures that skip `__init__`, xfail reasons | test-driven-development, mechanical-test-generation, gotchas-python | plan set + `Bash(python3 -m pytest *)` |
|
||||
| `code` | implementation review against normative spec text and the tests: correctness, security, gotchas-python, behaviour changes to existing call sites, wire-shape changes, the branch must not touch tests/ or spec/ | security-architecture, llm-code-security, python-patterns, gotchas-python | plan set + `Bash(git diff *)` |
|
||||
|
||||
## Dryrun behaviour
|
||||
|
||||
`--dryrun` prints: resolved repo and report dir, the reading list (marking any default file that is missing as `(absent, skipped)`), the full prompt text, the allowlist, the exact launch command line, and `[dryrun] nothing launched`. It creates no files.
|
||||
|
||||
## Edge cases
|
||||
|
||||
- Unknown `--kind` → exit 2 with the closed set listed.
|
||||
- A `--file` that does not exist → exit 2 naming it (before anything is written).
|
||||
- Round ≥2 with no previous result → warning, proceeds without the "adjudicated" paragraph.
|
||||
- `ask-minimax-run` missing or profile absent → exit 1.
|
||||
- `status` with an empty report dir → prints `no runs` and exits 0.
|
||||
- Never uses `pkill`; the launch script is self-contained so the user can kill by PID.
|
||||
|
||||
## Examples
|
||||
|
||||
```
|
||||
review-loop-run launch --kind spec --name spec-BC --round 2 \
|
||||
--file /tmp/x/spec-BC.diff --context "WIA-17c..f close bug 02a8436d; source: controlplane/api/work_items.py"
|
||||
review-loop-run status --name spec-BC
|
||||
review-loop-run counts /run/user/1000/review-loop/agent-runtimes/spec-BC-r2-result.md
|
||||
```
|
||||
66
tests/test-review-loop-run.sh
Executable file
66
tests/test-review-loop-run.sh
Executable file
@@ -0,0 +1,66 @@
|
||||
#!/usr/bin/env bash
|
||||
# test-review-loop-run.sh — dryrun/behaviour tests for review-loop-run (no launches).
|
||||
set -uo pipefail
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"; REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
SCRIPT="$REPO_ROOT/scripts/review-loop-run"
|
||||
GREEN='\033[0;32m'; RED='\033[0;31m'; NC='\033[0m'; PASS=0; FAIL=0
|
||||
assert_contains() { if [[ "$3" == *"$2"* ]]; then printf " ${GREEN}PASS${NC} %s\n" "$1"; PASS=$((PASS+1)); else printf " ${RED}FAIL${NC} %s\n expected: %s\n" "$1" "$2"; FAIL=$((FAIL+1)); fi; }
|
||||
assert_exit() { if [[ "$2" -eq "$3" ]]; then printf " ${GREEN}PASS${NC} %s\n" "$1"; PASS=$((PASS+1)); else printf " ${RED}FAIL${NC} %s (exit %s, expected %s)\n" "$1" "$3" "$2"; FAIL=$((FAIL+1)); fi; }
|
||||
|
||||
TMP=$(mktemp -d); trap 'rm -rf "$TMP"' EXIT
|
||||
export REVIEW_LOOP_RUN_DIR="$TMP/reports"; export REVIEW_LOOP_BP_DIR="$TMP/bp"; mkdir -p "$TMP/bp"
|
||||
echo "# bp" > "$TMP/bp/spec-driven-development.md"
|
||||
echo "diff" > "$TMP/spec.diff"; echo "# item" > "$TMP/item.md"
|
||||
|
||||
echo "review-loop-run — dryrun tests"
|
||||
out=$("$SCRIPT" launch --kind spec --name spec-X --file "$TMP/spec.diff" --context "closes bug 1234" -n 2>&1); rc=$?
|
||||
assert_exit "spec dryrun exits 0" 0 "$rc"
|
||||
assert_contains "dryrun banner" "[dryrun]" "$out"
|
||||
assert_contains "nothing launched" "nothing launched" "$out"
|
||||
assert_contains "task text is the spec template" "OUT OF SCOPE: reporting that a symbol" "$out"
|
||||
assert_contains "context included" "Context: closes bug 1234" "$out"
|
||||
assert_contains "output path uses tag" "spec-X-r1-result.md" "$out"
|
||||
assert_contains "present default listed" "$TMP/bp/spec-driven-development.md" "$out"
|
||||
assert_contains "absent default skipped" "default file absent, skipped" "$out"
|
||||
assert_contains "launch uses setsid-safe script with timeout" "timeout 1500 ask-minimax-run" "$out"
|
||||
[[ -d "$REVIEW_LOOP_RUN_DIR" ]] && { printf " ${RED}FAIL${NC} dryrun wrote report dir\n"; FAIL=$((FAIL+1)); } || { printf " ${GREEN}PASS${NC} dryrun writes nothing\n"; PASS=$((PASS+1)); }
|
||||
|
||||
out=$("$SCRIPT" launch --kind verify --name verify-1 --file "$TMP/item.md" -n 2>&1); rc=$?
|
||||
assert_exit "verify dryrun exits 0" 0 "$rc"
|
||||
assert_contains "verify allowlist has git show" "Bash(git show *)" "$out"
|
||||
assert_contains "verify verdict format" "Verdict:" "$out"
|
||||
|
||||
out=$("$SCRIPT" launch --kind tests --name t --file "$TMP/item.md" --round 2 -n 2>&1); rc=$?
|
||||
assert_exit "round 2 without prev still exits 0" 0 "$rc"
|
||||
assert_contains "round 2 warning" "no previous result" "$out"
|
||||
assert_contains "tests allowlist has pytest" "Bash(python3 -m pytest *)" "$out"
|
||||
|
||||
mkdir -p "$REVIEW_LOOP_RUN_DIR"; printf '| CRITICAL | 0 |\n| HIGH | 2 |\n' > "$REVIEW_LOOP_RUN_DIR/t-r1-result.md"
|
||||
out=$("$SCRIPT" launch --kind tests --name t --file "$TMP/item.md" --round 2 -n 2>&1)
|
||||
assert_contains "round 2 picks up previous result" "t-r1-result.md" "$out"
|
||||
assert_contains "adjudication paragraph" "do not re-raise" "$out"
|
||||
|
||||
out=$("$SCRIPT" launch --kind bogus --name x --file "$TMP/item.md" -n 2>&1); rc=$?
|
||||
assert_exit "unknown kind exits 2" 2 "$rc"
|
||||
out=$("$SCRIPT" launch --kind spec --name x --file "$TMP/nope" -n 2>&1); rc=$?
|
||||
assert_exit "missing file exits 2" 2 "$rc"
|
||||
out=$("$SCRIPT" launch --kind spec --name "bad name" --file "$TMP/item.md" -n 2>&1); rc=$?
|
||||
assert_exit "bad name exits 2" 2 "$rc"
|
||||
|
||||
echo "review-loop-run — counts"
|
||||
printf '| CRITICAL | 0 |\n| HIGH | 0 |\n| MEDIUM | 3 |\n' > "$TMP/clean.md"
|
||||
out=$("$SCRIPT" counts "$TMP/clean.md"); rc=$?
|
||||
assert_exit "clean result exits 0" 0 "$rc"; assert_contains "clean counts" "CRITICAL=0 HIGH=0 MEDIUM=3" "$out"
|
||||
printf -- '- **CRITICAL:** 1 (C1)\n- **HIGH:** 2\n' > "$TMP/dirty.md"
|
||||
out=$("$SCRIPT" counts "$TMP/dirty.md"); rc=$?
|
||||
assert_exit "dirty result exits 4" 4 "$rc"; assert_contains "list-style counts" "CRITICAL=1 HIGH=2" "$out"
|
||||
printf '## CRITICAL findings\n### C1 CRITICAL x\n### H1 HIGH y\n' > "$TMP/heads.md"
|
||||
out=$("$SCRIPT" counts "$TMP/heads.md"); rc=$?
|
||||
assert_exit "heading fallback exits 4" 4 "$rc"
|
||||
|
||||
echo "review-loop-run — status"
|
||||
printf 'x\nexit=0\n' > "$REVIEW_LOOP_RUN_DIR/t-r1-stdout.txt"
|
||||
out=$("$SCRIPT" status --name t 2>&1); rc=$?
|
||||
assert_exit "status exits 0" 0 "$rc"; assert_contains "status shows done" "done exit=0" "$out"; assert_contains "status shows counts" "HIGH=2" "$out"
|
||||
|
||||
echo; echo "$PASS passed, $FAIL failed"; [[ $FAIL -eq 0 ]]
|
||||
Reference in New Issue
Block a user