8bbf6cbb2f
- pre_test.sh: harness-level test folder revert script, replaces inline
_revert_test_folders() in entrypoint. Writes {"reverted": N} atomically.
- finalize.sh: set -x debug tracing; push retry (AGENT_PUSH_RETRIES, default 1
retry after 5s); output validation (AGENT_EXPECTED_OUTPUT env var); wrong-path
detection moves .agent-output/ files to working dir and writes
.correction-prompt.txt for entrypoint re-invoke
- harness.yaml: add scripts.pre_test
- spec-planning.yaml: all 17 nodes tagged with output:{path,min_bytes}
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1065 lines
42 KiB
YAML
1065 lines
42 KiB
YAML
name: spec-planning
|
|
version: 4
|
|
description: 'Multi-model spec planning with interview, cross-model reviews, human
|
|
gates, and post-synthesis security review with auto-fix. 17-node DAG: 2 interviews,
|
|
1 question consolidation, 2 plans, 6 cross-model reviews (3 disciplines x 2 models
|
|
— each reviews the OTHER''s plan), 1 escalation with human decision points, 1 final
|
|
synthesis by best-scoring model, 2 post-synthesis security reviews (parallel, cross-model),
|
|
1 auto-fix for non-escalated findings, 1 final fix implementation after human review
|
|
of escalations.
|
|
|
|
'
|
|
params:
|
|
required:
|
|
task_description:
|
|
type: string
|
|
description: What to spec — the feature or subsystem to design
|
|
project_id:
|
|
type: string
|
|
description: Target project identifier
|
|
optional:
|
|
model_a:
|
|
type: string
|
|
default: null
|
|
description: Override model A endpoint (auto-selected from registry if omitted)
|
|
model_b:
|
|
type: string
|
|
default: null
|
|
description: Override model B endpoint (auto-selected from registry if omitted)
|
|
best_model:
|
|
type: string
|
|
default: null
|
|
description: Model override for final synthesis — set after reading escalation
|
|
output (the best-scoring model)
|
|
repo:
|
|
type: string
|
|
default: null
|
|
description: Git repo URL to clone for project context (optional)
|
|
existing_specs:
|
|
type: string
|
|
default: ''
|
|
description: Paste existing spec content for context (e.g., current WF-1..WF-22)
|
|
scope_notes:
|
|
type: string
|
|
default: ''
|
|
description: Any constraints, prior decisions, or scope boundaries from the
|
|
human
|
|
nodes:
|
|
interview_a:
|
|
name: Interview Questions ({{ model_a or 'auto' }})
|
|
prompt: "You are a senior software architect preparing to write a detailed specification.\n\
|
|
Before you start, you need to ask clarifying questions to avoid costly assumptions.\n\
|
|
\n## Task to Spec\n{{ task_description }}\n\n{% if scope_notes %}\n## Scope\
|
|
\ Notes from Human\n{{ scope_notes }}\n{% endif %}\n\n{% if existing_specs %}\n\
|
|
## Existing Specifications (for context)\n{{ existing_specs }}\n{% endif %}\n\
|
|
\n## Best Practices\nRead these files from `/opt/harness/context/planning/best-practices/`:\n\
|
|
1. `spec-driven-development.md` — spec structure, requirement numbering, scenarios\n\
|
|
2. `test-driven-development.md` — deriving tests from specs, edge case discovery\n\
|
|
3. `security-architecture.md` — server boundary rule, defense in depth, auth\
|
|
\ patterns\n4. `llm-code-security.md` — injection flaws, input validation, OWASP\
|
|
\ for AI code\n\n{% if repo %}\n## Project Context\nThe project repo has been\
|
|
\ cloned. Read `CLAUDE.md` and any existing `spec/` files\nto understand the\
|
|
\ current architecture, conventions, and design decisions.\n{% endif %}\n\n\
|
|
## Your Task\nGenerate questions that will improve the quality of the spec you'll\
|
|
\ write later.\nThink about what you'd ask a product owner, tech lead, or domain\
|
|
\ expert before\ncommitting to a design.\n\nFocus on:\n- **Ambiguous requirements**\
|
|
\ — what does X mean in this context?\n- **Missing scope boundaries** — is Y\
|
|
\ in or out of scope?\n- **Business logic decisions** — should Z behave as A\
|
|
\ or B?\n- **Technical constraints** — performance targets, compatibility, resource\
|
|
\ limits?\n- **Dependencies and integration** — how does this interact with\
|
|
\ existing subsystems?\n- **Priority and phasing** — which parts are essential\
|
|
\ vs nice-to-have?\n- **Security implications** — who are the threat actors,\
|
|
\ what's the trust boundary?\n- **Testing strategy** — what's the expected test\
|
|
\ infrastructure?\n\n## Output Format\nWrite a numbered list of questions to\
|
|
\ `/workspace/project/output.md`:\n\n```\n## Clarifying Questions\n\n1. **[Question]**\n\
|
|
\ Why it matters: [what design decision hinges on the answer]\n Suggested\
|
|
\ default: [what you'd assume if no answer is given]\n\n2. **[Question]**\n\
|
|
\ ...\n```\n\nAim for 8-20 questions. Prioritize questions whose answers would\
|
|
\ change the most\ndesign decisions. Don't ask about things that are clearly\
|
|
\ stated in the task\ndescription or scope notes.\n"
|
|
harness: planning/v1
|
|
requirements:
|
|
min_scores:
|
|
complexity: 7
|
|
spec_adherence: 8
|
|
model_override: '{{ model_a }}'
|
|
output:
|
|
path: /workspace/project/output.md
|
|
min_bytes: 500
|
|
interview_b:
|
|
name: Interview Questions ({{ model_b or 'auto' }})
|
|
prompt: "You are a senior software architect preparing to write a detailed specification.\n\
|
|
Before you start, you need to ask clarifying questions to avoid costly assumptions.\n\
|
|
\n## Task to Spec\n{{ task_description }}\n\n{% if scope_notes %}\n## Scope\
|
|
\ Notes from Human\n{{ scope_notes }}\n{% endif %}\n\n{% if existing_specs %}\n\
|
|
## Existing Specifications (for context)\n{{ existing_specs }}\n{% endif %}\n\
|
|
\n## Best Practices\nRead these files from `/opt/harness/context/planning/best-practices/`:\n\
|
|
1. `spec-driven-development.md` — spec structure, requirement numbering, scenarios\n\
|
|
2. `test-driven-development.md` — deriving tests from specs, edge case discovery\n\
|
|
3. `security-architecture.md` — server boundary rule, defense in depth, auth\
|
|
\ patterns\n4. `llm-code-security.md` — injection flaws, input validation, OWASP\
|
|
\ for AI code\n\n{% if repo %}\n## Project Context\nThe project repo has been\
|
|
\ cloned. Read `CLAUDE.md` and any existing `spec/` files\nto understand the\
|
|
\ current architecture, conventions, and design decisions.\n{% endif %}\n\n\
|
|
## Your Task\nGenerate questions that will improve the quality of the spec you'll\
|
|
\ write later.\nThink about what you'd ask a product owner, tech lead, or domain\
|
|
\ expert before\ncommitting to a design.\n\nFocus on:\n- **Ambiguous requirements**\
|
|
\ — what does X mean in this context?\n- **Missing scope boundaries** — is Y\
|
|
\ in or out of scope?\n- **Business logic decisions** — should Z behave as A\
|
|
\ or B?\n- **Technical constraints** — performance targets, compatibility, resource\
|
|
\ limits?\n- **Dependencies and integration** — how does this interact with\
|
|
\ existing subsystems?\n- **Priority and phasing** — which parts are essential\
|
|
\ vs nice-to-have?\n- **Security implications** — who are the threat actors,\
|
|
\ what's the trust boundary?\n- **Testing strategy** — what's the expected test\
|
|
\ infrastructure?\n\n## Output Format\nWrite a numbered list of questions to\
|
|
\ `/workspace/project/output.md`:\n\n```\n## Clarifying Questions\n\n1. **[Question]**\n\
|
|
\ Why it matters: [what design decision hinges on the answer]\n Suggested\
|
|
\ default: [what you'd assume if no answer is given]\n\n2. **[Question]**\n\
|
|
\ ...\n```\n\nAim for 8-20 questions. Prioritize questions whose answers would\
|
|
\ change the most\ndesign decisions. Don't ask about things that are clearly\
|
|
\ stated in the task\ndescription or scope notes.\n"
|
|
harness: planning/v1
|
|
requirements:
|
|
min_scores:
|
|
complexity: 7
|
|
spec_adherence: 8
|
|
model_override: '{{ model_b }}'
|
|
output:
|
|
path: /workspace/project/output.md
|
|
min_bytes: 500
|
|
consolidate_questions:
|
|
name: Consolidate Interview Questions
|
|
depends_on:
|
|
- interview_a
|
|
- interview_b
|
|
prompt: "You are consolidating clarifying questions from two independent reviewers\n\
|
|
into a single, organized list for a human to answer.\n\n## Questions from Model\
|
|
\ A\n<<ARTIFACT:interview_a:output>>\n\n## Questions from Model B\n<<ARTIFACT:interview_b:output>>\n\
|
|
\n## Instructions\n1. **Deduplicate** — merge questions that ask the same thing\
|
|
\ in different words\n2. **Flag consensus** — when both models asked the same\
|
|
\ question, note \"(Asked by both)\"\n as this is a strong signal the question\
|
|
\ matters\n3. **Group by theme** — organize into sections (Scope, Architecture,\
|
|
\ Security,\n Testing, Integration, Priority, etc.)\n4. **Preserve context**\
|
|
\ — keep each question's \"why it matters\" and \"suggested default\"\n5. **Order\
|
|
\ by impact** — within each group, questions whose answers change the most\n\
|
|
\ design decisions come first\n\n## Output Format\nWrite to `/workspace/project/output.md`:\n\
|
|
\n```\n## Questions for Human Review\n\n### High Impact (answers change multiple\
|
|
\ design decisions)\n1. **[Question]** (Asked by: A / B / both)\n Why it matters:\
|
|
\ [explanation]\n Suggested default: [what to assume if unanswered]\n\n###\
|
|
\ Medium Impact\n...\n\n### Low Impact / Confirmations\n...\n```\n\nDo not add\
|
|
\ your own questions — only consolidate what the two models asked.\nDo not answer\
|
|
\ the questions — that's the human's job.\n"
|
|
harness: planning/v1
|
|
requirements:
|
|
min_scores:
|
|
spec_adherence: 7
|
|
output:
|
|
path: /workspace/project/output.md
|
|
min_bytes: 1000
|
|
plan_a:
|
|
name: Spec Draft A ({{ model_a or 'auto' }})
|
|
depends_on:
|
|
- consolidate_questions
|
|
prompt: 'You are a senior software architect writing a detailed specification.
|
|
|
|
|
|
## Task
|
|
|
|
{{ task_description }}
|
|
|
|
|
|
{% if scope_notes %}
|
|
|
|
## Scope Notes from Human
|
|
|
|
{{ scope_notes }}
|
|
|
|
{% endif %}
|
|
|
|
|
|
{% if existing_specs %}
|
|
|
|
## Existing Specifications (for context — extend, don''t duplicate)
|
|
|
|
{{ existing_specs }}
|
|
|
|
{% endif %}
|
|
|
|
|
|
## Interview Answers
|
|
|
|
The following questions were asked during the interview phase.
|
|
|
|
The human has provided answers — use them to guide your design decisions.
|
|
|
|
Where a question has no answer, use the suggested default.
|
|
|
|
|
|
<<ARTIFACT:consolidate_questions:output>>
|
|
|
|
|
|
## Methodology
|
|
|
|
Read and apply these best-practice documents from `/opt/harness/context/planning/best-practices/`:
|
|
|
|
1. `spec-driven-development.md` — spec structure, requirement numbering, scenarios
|
|
|
|
2. `test-driven-development.md` — deriving tests from specs, edge case discovery
|
|
|
|
3. `security-architecture.md` — server boundary rule, defense in depth, auth
|
|
patterns
|
|
|
|
4. `llm-code-security.md` — injection flaws, input validation, OWASP for AI
|
|
code
|
|
|
|
|
|
{% if repo %}
|
|
|
|
## Project Context
|
|
|
|
The project repo has been cloned. Read `CLAUDE.md` and any existing `spec/`
|
|
files
|
|
|
|
to understand the current architecture and conventions.
|
|
|
|
{% endif %}
|
|
|
|
|
|
## Output Requirements
|
|
|
|
Write a complete spec with these sections:
|
|
|
|
1. **Overview** — 2-3 sentences on what this subsystem does
|
|
|
|
2. **Responsibilities** — what it owns vs delegates
|
|
|
|
3. **Dependencies** — other specs to read
|
|
|
|
4. **Data Model** — types, schemas, with concrete JSON/YAML examples
|
|
|
|
5. **Requirements** — numbered (e.g., XX-1, XX-2), each independently testable,
|
|
each with a "Why:" rationale
|
|
|
|
6. **Scenarios** — given/when/then for every requirement
|
|
|
|
7. **Security Considerations** — input validation, injection risks, access control,
|
|
resource limits
|
|
|
|
8. **Test Strategy** — how each requirement maps to tests, edge cases to cover
|
|
|
|
|
|
Be opinionated. Make concrete design decisions with rationale. Call out trade-offs.
|
|
|
|
Do NOT leave things vague ("handle errors appropriately") — be specific ("return
|
|
HTTP 413 with error body").
|
|
|
|
|
|
Write the complete spec to /workspace/project/output.md
|
|
|
|
'
|
|
harness: planning/v1
|
|
requirements:
|
|
min_scores:
|
|
complexity: 7
|
|
spec_adherence: 8
|
|
model_override: '{{ model_a }}'
|
|
output:
|
|
path: /workspace/project/output.md
|
|
min_bytes: 3000
|
|
plan_b:
|
|
name: Spec Draft B ({{ model_b or 'auto' }})
|
|
depends_on:
|
|
- consolidate_questions
|
|
prompt: 'You are a senior software architect writing a detailed specification.
|
|
|
|
|
|
## Task
|
|
|
|
{{ task_description }}
|
|
|
|
|
|
{% if scope_notes %}
|
|
|
|
## Scope Notes from Human
|
|
|
|
{{ scope_notes }}
|
|
|
|
{% endif %}
|
|
|
|
|
|
{% if existing_specs %}
|
|
|
|
## Existing Specifications (for context — extend, don''t duplicate)
|
|
|
|
{{ existing_specs }}
|
|
|
|
{% endif %}
|
|
|
|
|
|
## Interview Answers
|
|
|
|
The following questions were asked during the interview phase.
|
|
|
|
The human has provided answers — use them to guide your design decisions.
|
|
|
|
Where a question has no answer, use the suggested default.
|
|
|
|
|
|
<<ARTIFACT:consolidate_questions:output>>
|
|
|
|
|
|
## Methodology
|
|
|
|
Read and apply these best-practice documents from `/opt/harness/context/planning/best-practices/`:
|
|
|
|
1. `spec-driven-development.md` — spec structure, requirement numbering, scenarios
|
|
|
|
2. `test-driven-development.md` — deriving tests from specs, edge case discovery
|
|
|
|
3. `security-architecture.md` — server boundary rule, defense in depth, auth
|
|
patterns
|
|
|
|
4. `llm-code-security.md` — injection flaws, input validation, OWASP for AI
|
|
code
|
|
|
|
|
|
{% if repo %}
|
|
|
|
## Project Context
|
|
|
|
The project repo has been cloned. Read `CLAUDE.md` and any existing `spec/`
|
|
files
|
|
|
|
to understand the current architecture and conventions.
|
|
|
|
{% endif %}
|
|
|
|
|
|
## Output Requirements
|
|
|
|
Write a complete spec with these sections:
|
|
|
|
1. **Overview** — 2-3 sentences on what this subsystem does
|
|
|
|
2. **Responsibilities** — what it owns vs delegates
|
|
|
|
3. **Dependencies** — other specs to read
|
|
|
|
4. **Data Model** — types, schemas, with concrete JSON/YAML examples
|
|
|
|
5. **Requirements** — numbered (e.g., XX-1, XX-2), each independently testable,
|
|
each with a "Why:" rationale
|
|
|
|
6. **Scenarios** — given/when/then for every requirement
|
|
|
|
7. **Security Considerations** — input validation, injection risks, access control,
|
|
resource limits
|
|
|
|
8. **Test Strategy** — how each requirement maps to tests, edge cases to cover
|
|
|
|
|
|
Be opinionated. Make concrete design decisions with rationale. Call out trade-offs.
|
|
|
|
Do NOT leave things vague ("handle errors appropriately") — be specific ("return
|
|
HTTP 413 with error body").
|
|
|
|
|
|
Write the complete spec to /workspace/project/output.md
|
|
|
|
'
|
|
harness: planning/v1
|
|
requirements:
|
|
min_scores:
|
|
complexity: 7
|
|
spec_adherence: 8
|
|
model_override: '{{ model_b }}'
|
|
output:
|
|
path: /workspace/project/output.md
|
|
min_bytes: 3000
|
|
spec_review_a:
|
|
name: Spec Review by A (of Plan B)
|
|
depends_on:
|
|
- plan_b
|
|
prompt: 'You are reviewing a specification for adherence to spec-driven development
|
|
practices.
|
|
|
|
You did NOT write this plan. Review it critically.
|
|
|
|
|
|
## Best Practice Reference
|
|
|
|
Read `/opt/harness/context/planning/best-practices/spec-driven-development.md`
|
|
thoroughly before reviewing.
|
|
|
|
|
|
## Spec Under Review (written by another model)
|
|
|
|
<<ARTIFACT:plan_b:output>>
|
|
|
|
|
|
## Review Checklist
|
|
|
|
Evaluate:
|
|
|
|
1. **Structure** — Does it have all required sections (Overview, Responsibilities,
|
|
Dependencies, Data Model, Requirements, Scenarios)?
|
|
|
|
2. **Requirements quality** — Are they numbered, independently testable, unambiguous?
|
|
Does each have a "Why:" rationale?
|
|
|
|
3. **Scenarios** — Does every requirement have at least one given/when/then?
|
|
Are edge cases covered?
|
|
|
|
4. **Cross-references** — Does it reference other specs correctly without duplicating?
|
|
|
|
5. **Concrete examples** — Are data models shown with realistic JSON/code, not
|
|
just abstract schemas?
|
|
|
|
6. **Completeness** — Are there gaps? Requirements that should exist but don''t?
|
|
|
|
|
|
## Output Format
|
|
|
|
List numbered improvement points with severity (CRITICAL/HIGH/MEDIUM/LOW).
|
|
|
|
Then: "## Questions for Human Review" — decisions needing human input (unclear
|
|
requirements, business logic, scope).
|
|
|
|
|
|
Write to /workspace/project/output.md
|
|
|
|
'
|
|
harness: planning/v1
|
|
requirements:
|
|
min_scores:
|
|
spec_adherence: 9
|
|
model_override: '{{ model_a }}'
|
|
output:
|
|
path: /workspace/project/output.md
|
|
min_bytes: 1500
|
|
spec_review_b:
|
|
name: Spec Review by B (of Plan A)
|
|
depends_on:
|
|
- plan_a
|
|
prompt: 'You are reviewing a specification for adherence to spec-driven development
|
|
practices.
|
|
|
|
You did NOT write this plan. Review it critically.
|
|
|
|
|
|
## Best Practice Reference
|
|
|
|
Read `/opt/harness/context/planning/best-practices/spec-driven-development.md`
|
|
thoroughly before reviewing.
|
|
|
|
|
|
## Spec Under Review (written by another model)
|
|
|
|
<<ARTIFACT:plan_a:output>>
|
|
|
|
|
|
## Review Checklist
|
|
|
|
Evaluate:
|
|
|
|
1. **Structure** — Does it have all required sections (Overview, Responsibilities,
|
|
Dependencies, Data Model, Requirements, Scenarios)?
|
|
|
|
2. **Requirements quality** — Are they numbered, independently testable, unambiguous?
|
|
Does each have a "Why:" rationale?
|
|
|
|
3. **Scenarios** — Does every requirement have at least one given/when/then?
|
|
Are edge cases covered?
|
|
|
|
4. **Cross-references** — Does it reference other specs correctly without duplicating?
|
|
|
|
5. **Concrete examples** — Are data models shown with realistic JSON/code, not
|
|
just abstract schemas?
|
|
|
|
6. **Completeness** — Are there gaps? Requirements that should exist but don''t?
|
|
|
|
|
|
## Output Format
|
|
|
|
List numbered improvement points with severity (CRITICAL/HIGH/MEDIUM/LOW).
|
|
|
|
Then: "## Questions for Human Review" — decisions needing human input (unclear
|
|
requirements, business logic, scope).
|
|
|
|
|
|
Write to /workspace/project/output.md
|
|
|
|
'
|
|
harness: planning/v1
|
|
requirements:
|
|
min_scores:
|
|
spec_adherence: 9
|
|
model_override: '{{ model_b }}'
|
|
output:
|
|
path: /workspace/project/output.md
|
|
min_bytes: 1500
|
|
security_review_a:
|
|
name: Security Review by A (of Plan B)
|
|
depends_on:
|
|
- plan_b
|
|
prompt: 'You are performing a security review of a specification.
|
|
|
|
You did NOT write this plan. Review it critically.
|
|
|
|
|
|
## Best Practice References
|
|
|
|
Read these thoroughly before reviewing:
|
|
|
|
- `/opt/harness/context/planning/best-practices/security-architecture.md` —
|
|
server boundary rule, defense in depth, proxy patterns
|
|
|
|
- `/opt/harness/context/planning/best-practices/llm-code-security.md` — injection
|
|
flaws, OWASP for AI code, hallucinated packages
|
|
|
|
|
|
## Spec Under Review (written by another model)
|
|
|
|
<<ARTIFACT:plan_b:output>>
|
|
|
|
|
|
## Review Focus
|
|
|
|
Evaluate:
|
|
|
|
1. **Injection risks** — SQL, command, template, prompt, SSRF, path traversal
|
|
|
|
2. **Input validation** — are all inputs validated at system boundaries? Size
|
|
limits?
|
|
|
|
3. **Authentication/Authorization** — who can access what? Multi-tenancy isolation?
|
|
|
|
4. **Secret handling** — are credentials ever exposed? Server boundary rule
|
|
compliance?
|
|
|
|
5. **Resource exhaustion** — unbounded loops, unlimited sizes, missing timeouts?
|
|
|
|
6. **Data integrity** — race conditions, TOCTOU, atomic operations?
|
|
|
|
7. **LLM-specific risks** — prompt injection via user input, hallucinated dependencies,
|
|
over-permissive defaults
|
|
|
|
|
|
## Output Format
|
|
|
|
For each finding: Severity (CRITICAL/HIGH/MEDIUM/LOW/INFO), attack vector, affected
|
|
requirement ID, concrete fix.
|
|
|
|
Then: "## Questions for Human Review" — security decisions needing human judgement.
|
|
|
|
|
|
Write to /workspace/project/output.md
|
|
|
|
'
|
|
harness: planning/v1
|
|
requirements:
|
|
min_scores:
|
|
spec_adherence: 8
|
|
model_override: '{{ model_a }}'
|
|
output:
|
|
path: /workspace/project/output.md
|
|
min_bytes: 1500
|
|
security_review_b:
|
|
name: Security Review by B (of Plan A)
|
|
depends_on:
|
|
- plan_a
|
|
prompt: 'You are performing a security review of a specification.
|
|
|
|
You did NOT write this plan. Review it critically.
|
|
|
|
|
|
## Best Practice References
|
|
|
|
Read these thoroughly before reviewing:
|
|
|
|
- `/opt/harness/context/planning/best-practices/security-architecture.md` —
|
|
server boundary rule, defense in depth, proxy patterns
|
|
|
|
- `/opt/harness/context/planning/best-practices/llm-code-security.md` — injection
|
|
flaws, OWASP for AI code, hallucinated packages
|
|
|
|
|
|
## Spec Under Review (written by another model)
|
|
|
|
<<ARTIFACT:plan_a:output>>
|
|
|
|
|
|
## Review Focus
|
|
|
|
Evaluate:
|
|
|
|
1. **Injection risks** — SQL, command, template, prompt, SSRF, path traversal
|
|
|
|
2. **Input validation** — are all inputs validated at system boundaries? Size
|
|
limits?
|
|
|
|
3. **Authentication/Authorization** — who can access what? Multi-tenancy isolation?
|
|
|
|
4. **Secret handling** — are credentials ever exposed? Server boundary rule
|
|
compliance?
|
|
|
|
5. **Resource exhaustion** — unbounded loops, unlimited sizes, missing timeouts?
|
|
|
|
6. **Data integrity** — race conditions, TOCTOU, atomic operations?
|
|
|
|
7. **LLM-specific risks** — prompt injection via user input, hallucinated dependencies,
|
|
over-permissive defaults
|
|
|
|
|
|
## Output Format
|
|
|
|
For each finding: Severity (CRITICAL/HIGH/MEDIUM/LOW/INFO), attack vector, affected
|
|
requirement ID, concrete fix.
|
|
|
|
Then: "## Questions for Human Review" — security decisions needing human judgement.
|
|
|
|
|
|
Write to /workspace/project/output.md
|
|
|
|
'
|
|
harness: planning/v1
|
|
requirements:
|
|
min_scores:
|
|
spec_adherence: 8
|
|
model_override: '{{ model_b }}'
|
|
output:
|
|
path: /workspace/project/output.md
|
|
min_bytes: 1500
|
|
tdd_review_a:
|
|
name: TDD Review by A (of Plan B)
|
|
depends_on:
|
|
- plan_b
|
|
prompt: "You are reviewing a specification for testability and test strategy quality.\n\
|
|
You did NOT write this plan. Review it critically.\n\n## Best Practice Reference\n\
|
|
Read `/opt/harness/context/planning/best-practices/test-driven-development.md`\
|
|
\ thoroughly before reviewing.\n\n## Spec Under Review (written by another model)\n\
|
|
<<ARTIFACT:plan_b:output>>\n\n## Review Focus\nEvaluate:\n1. **Requirement testability**\
|
|
\ — Can each requirement be tested without human judgement?\n Flag requirements\
|
|
\ that say \"appropriate\", \"reasonable\", \"as needed\".\n2. **Test coverage\
|
|
\ plan** — Does the test strategy cover all requirements? Any gaps?\n3. **Edge\
|
|
\ cases** — Are boundary conditions, error paths, and concurrency scenarios\
|
|
\ covered?\n4. **Test naming** — Do proposed test names include requirement\
|
|
\ IDs (e.g., test_wf23_artifact_upload)?\n5. **Parameterised tests** — Where\
|
|
\ specs enumerate valid values, are parameterised tests suggested?\n6. **Integration\
|
|
\ test boundary** — Is the line between unit and integration tests clear?\n\
|
|
7. **Property-based testing** — Are there invariants that would benefit from\
|
|
\ hypothesis/property testing?\n8. **Mock boundaries** — What should be mocked\
|
|
\ vs tested with real dependencies?\n\n## Output Format\nList numbered improvement\
|
|
\ points with severity.\nThen: \"## Questions for Human Review\" — test scope\
|
|
\ decisions needing human input.\n\nWrite to /workspace/project/output.md\n"
|
|
harness: planning/v1
|
|
requirements:
|
|
min_scores:
|
|
spec_adherence: 8
|
|
model_override: '{{ model_a }}'
|
|
output:
|
|
path: /workspace/project/output.md
|
|
min_bytes: 1500
|
|
tdd_review_b:
|
|
name: TDD Review by B (of Plan A)
|
|
depends_on:
|
|
- plan_a
|
|
prompt: "You are reviewing a specification for testability and test strategy quality.\n\
|
|
You did NOT write this plan. Review it critically.\n\n## Best Practice Reference\n\
|
|
Read `/opt/harness/context/planning/best-practices/test-driven-development.md`\
|
|
\ thoroughly before reviewing.\n\n## Spec Under Review (written by another model)\n\
|
|
<<ARTIFACT:plan_a:output>>\n\n## Review Focus\nEvaluate:\n1. **Requirement testability**\
|
|
\ — Can each requirement be tested without human judgement?\n Flag requirements\
|
|
\ that say \"appropriate\", \"reasonable\", \"as needed\".\n2. **Test coverage\
|
|
\ plan** — Does the test strategy cover all requirements? Any gaps?\n3. **Edge\
|
|
\ cases** — Are boundary conditions, error paths, and concurrency scenarios\
|
|
\ covered?\n4. **Test naming** — Do proposed test names include requirement\
|
|
\ IDs (e.g., test_wf23_artifact_upload)?\n5. **Parameterised tests** — Where\
|
|
\ specs enumerate valid values, are parameterised tests suggested?\n6. **Integration\
|
|
\ test boundary** — Is the line between unit and integration tests clear?\n\
|
|
7. **Property-based testing** — Are there invariants that would benefit from\
|
|
\ hypothesis/property testing?\n8. **Mock boundaries** — What should be mocked\
|
|
\ vs tested with real dependencies?\n\n## Output Format\nList numbered improvement\
|
|
\ points with severity.\nThen: \"## Questions for Human Review\" — test scope\
|
|
\ decisions needing human input.\n\nWrite to /workspace/project/output.md\n"
|
|
harness: planning/v1
|
|
requirements:
|
|
min_scores:
|
|
spec_adherence: 8
|
|
model_override: '{{ model_b }}'
|
|
output:
|
|
path: /workspace/project/output.md
|
|
min_bytes: 1500
|
|
escalate:
|
|
name: Decision Briefing for Human
|
|
depends_on:
|
|
- spec_review_a
|
|
- spec_review_b
|
|
- security_review_a
|
|
- security_review_b
|
|
- tdd_review_a
|
|
- tdd_review_b
|
|
prompt: 'You are preparing a decision briefing for a human reviewer.
|
|
|
|
You have two spec drafts and SIX cross-model reviews (each model reviewed
|
|
|
|
only the OTHER model''s plan — no self-review). Your job: surface what needs
|
|
|
|
human decisions vs what the reviewers agree on.
|
|
|
|
|
|
## Plan A (written by Model A, reviewed by Model B)
|
|
|
|
<<ARTIFACT:plan_a:output>>
|
|
|
|
|
|
## Plan B (written by Model B, reviewed by Model A)
|
|
|
|
<<ARTIFACT:plan_b:output>>
|
|
|
|
|
|
## Reviews of Plan B (by Model A)
|
|
|
|
### Spec Review
|
|
|
|
<<ARTIFACT:spec_review_a:output>>
|
|
|
|
### Security Review
|
|
|
|
<<ARTIFACT:security_review_a:output>>
|
|
|
|
### TDD Review
|
|
|
|
<<ARTIFACT:tdd_review_a:output>>
|
|
|
|
|
|
## Reviews of Plan A (by Model B)
|
|
|
|
### Spec Review
|
|
|
|
<<ARTIFACT:spec_review_b:output>>
|
|
|
|
### Security Review
|
|
|
|
<<ARTIFACT:security_review_b:output>>
|
|
|
|
### TDD Review
|
|
|
|
<<ARTIFACT:tdd_review_b:output>>
|
|
|
|
|
|
## Output Format
|
|
|
|
|
|
### 1. Design Disagreements
|
|
|
|
For each disagreement between Plan A and Plan B:
|
|
|
|
- **Topic:** (e.g., "Artifact storage: dedicated table vs metadata inline")
|
|
|
|
- **Plan A approach:** (summary)
|
|
|
|
- **Plan B approach:** (summary)
|
|
|
|
- **What A''s reviewer of B said:** (summary)
|
|
|
|
- **What B''s reviewer of A said:** (summary)
|
|
|
|
- **Recommendation:** (your assessment, with rationale)
|
|
|
|
- **Decision needed:** YES / NO (YES if reviewers disagree or reasonable people
|
|
could disagree)
|
|
|
|
|
|
### 2. Correlated Findings
|
|
|
|
Where Model A''s review of Plan B and Model B''s review of Plan A found
|
|
|
|
related issues (same underlying problem manifesting in both plans):
|
|
|
|
- **Issue:** (description)
|
|
|
|
- **In Plan A:** (what B''s reviewer found)
|
|
|
|
- **In Plan B:** (what A''s reviewer found)
|
|
|
|
- **Fix:** (recommended resolution)
|
|
|
|
|
|
### 3. Open Questions
|
|
|
|
Consolidate all "Questions for Human Review" from all six reviews.
|
|
|
|
Deduplicate. For each:
|
|
|
|
- **Question:** (clear, actionable)
|
|
|
|
- **Context:** (why this matters)
|
|
|
|
- **Options:** (concrete choices)
|
|
|
|
- **Default if no answer:** (what the synthesizer would pick)
|
|
|
|
|
|
### 4. Security Decisions
|
|
|
|
All CRITICAL and HIGH findings from both security reviews.
|
|
|
|
Note whether the finding affects Plan A, Plan B, or both.
|
|
|
|
|
|
### 5. Consensus Items
|
|
|
|
Areas where both plans agree AND reviewers found no issues.
|
|
|
|
These don''t need human review.
|
|
|
|
|
|
### 6. Model Scoring Recommendation
|
|
|
|
Based on review findings, recommend which model should write the final synthesis:
|
|
|
|
- **Plan A review score:** (fewer/less severe findings = better)
|
|
|
|
- **Plan B review score:** (fewer/less severe findings = better)
|
|
|
|
- **Recommendation:** Model A or Model B for final synthesis, with rationale
|
|
|
|
|
|
Be concise. Lead with decisions, not context.
|
|
|
|
|
|
Write to /workspace/project/output.md
|
|
|
|
'
|
|
harness: planning/v1
|
|
requirements:
|
|
min_scores:
|
|
complexity: 9
|
|
spec_adherence: 9
|
|
output:
|
|
path: /workspace/project/output.md
|
|
min_bytes: 2000
|
|
synthesize:
|
|
name: Final Spec Synthesis
|
|
depends_on:
|
|
- escalate
|
|
prompt: "You are producing the FINAL specification by combining two competing\n\
|
|
drafts, six cross-model reviews, and an escalation briefing with human decisions.\n\
|
|
\n## Plan A\n<<ARTIFACT:plan_a:output>>\n\n## Plan B\n<<ARTIFACT:plan_b:output>>\n\
|
|
\n## Reviews of Plan B (by Model A)\n<<ARTIFACT:spec_review_a:output>>\n<<ARTIFACT:security_review_a:output>>\n\
|
|
<<ARTIFACT:tdd_review_a:output>>\n\n## Reviews of Plan A (by Model B)\n<<ARTIFACT:spec_review_b:output>>\n\
|
|
<<ARTIFACT:security_review_b:output>>\n<<ARTIFACT:tdd_review_b:output>>\n\n\
|
|
## Escalation Briefing (with human decisions)\n<<ARTIFACT:escalate:output>>\n\
|
|
\n## Best Practice References\nRead ALL of these from `/opt/harness/context/planning/best-practices/`:\n\
|
|
- `spec-driven-development.md`\n- `test-driven-development.md`\n- `security-architecture.md`\n\
|
|
- `llm-code-security.md`\n\n## Instructions\n\n1. **Start from the stronger\
|
|
\ plan** — the one with fewer and less severe review findings.\n Don't average;\
|
|
\ pick the better foundation and incorporate the best from the other.\n\n2.\
|
|
\ **Resolve all disagreements** using the escalation briefing.\n Where the\
|
|
\ briefing says \"Decision needed: YES\" and human decisions are present,\n\
|
|
\ follow the human's choice. Where no human response is present,\n use the\
|
|
\ briefing's \"Default if no answer\" and mark with\n `<!-- HUMAN DECISION\
|
|
\ PENDING: [topic] -->`.\n\n3. **Address correlated findings** — issues found\
|
|
\ in both plans indicate a\n fundamental problem that needs a different approach,\
|
|
\ not just a patch.\n\n4. **Address all CRITICAL and HIGH security findings**\
|
|
\ as concrete requirement changes.\n\n5. **Every requirement must have:** unique\
|
|
\ ID, \"Why:\" rationale, given/when/then scenario, test mapping.\n\n6. **Include\
|
|
\ a Security Requirements section** and a **Test Strategy section**.\n\n7. **Do\
|
|
\ NOT include** review commentary, improvement points, or process notes.\n\n\
|
|
Write the complete final spec to /workspace/project/output.md\n"
|
|
harness: planning/v1
|
|
requirements:
|
|
min_scores:
|
|
complexity: 9
|
|
spec_adherence: 9
|
|
creativity: 7
|
|
model_override: '{{ best_model }}'
|
|
output:
|
|
path: /workspace/project/output.md
|
|
min_bytes: 3000
|
|
post_security_a:
|
|
name: Post-Synthesis Security Review ({{ model_a or 'auto' }})
|
|
depends_on:
|
|
- synthesize
|
|
prompt: "You are performing a security review of a FINAL synthesized specification.\n\
|
|
This spec was produced by merging two competing drafts and applying\nreview\
|
|
\ feedback. Your job: find security issues that survived synthesis\nor were\
|
|
\ introduced by the merge process.\n\n## Synthesized Spec\n<<ARTIFACT:synthesize:output>>\n\
|
|
\n## Best Practice References\nRead these thoroughly before reviewing:\n- `/opt/harness/context/planning/best-practices/security-architecture.md`\n\
|
|
- `/opt/harness/context/planning/best-practices/llm-code-security.md`\n- `/opt/harness/context/planning/best-practices/api-design.md`\
|
|
\ (if it exists)\n\n{% if repo %}\n## Project Context\nThe project repo has\
|
|
\ been cloned. Read `CLAUDE.md` and any existing `spec/` files\nfor architectural\
|
|
\ context (existing auth patterns, data flows, trust boundaries).\n{% endif\
|
|
\ %}\n\n## Review Focus\n1. **Injection risks** — SQL, command, template, prompt,\
|
|
\ SSRF, path traversal\n2. **Input validation** — all inputs validated at system\
|
|
\ boundaries? Size limits?\n3. **Authentication/Authorization** — who can access\
|
|
\ what? Identity spoofing?\n4. **Secret handling** — credentials exposed? Server\
|
|
\ boundary rule?\n5. **Resource exhaustion** — unbounded loops, unlimited sizes,\
|
|
\ missing timeouts?\n6. **Data integrity** — race conditions, TOCTOU, atomic\
|
|
\ operations?\n7. **LLM-specific risks** — prompt injection, hallucinated deps,\
|
|
\ over-permissive defaults\n8. **Synthesis artifacts** — inconsistencies between\
|
|
\ merged sections, conflicting\n requirements, gaps where one plan's approach\
|
|
\ was dropped but the replacement\n was incomplete\n\n## Output Format\nStructure\
|
|
\ your output into two clear sections:\n\n### Fixes (implement directly — no\
|
|
\ human decision needed)\nFor each finding that has an unambiguous fix:\n- **ID:**\
|
|
\ F-NN\n- **Severity:** CRITICAL / HIGH / MEDIUM / LOW / INFO\n- **Requirement:**\
|
|
\ affected requirement ID(s)\n- **Issue:** what's wrong\n- **Current text:**\
|
|
\ quote the problematic spec text\n- **Fixed text:** exact replacement text\n\
|
|
- **Why:** rationale for the fix\n\n### Escalations (need human decision)\n\
|
|
For each finding where reasonable people could disagree on the fix:\n- **ID:**\
|
|
\ E-NN\n- **Severity:** CRITICAL / HIGH / MEDIUM / LOW\n- **Requirement:** affected\
|
|
\ requirement ID(s)\n- **Issue:** what's wrong\n- **Options:** concrete choices\
|
|
\ (A, B, C) with trade-offs\n- **Default recommendation:** what you'd pick and\
|
|
\ why\n\nBe specific. Quote exact spec text. Provide exact replacement text\
|
|
\ for fixes.\nDo NOT suggest vague improvements (\"consider adding validation\"\
|
|
) — specify\nexactly what validation, on what field, with what error code.\n\
|
|
\nWrite to /workspace/project/output.md\n"
|
|
harness: planning/v1
|
|
requirements:
|
|
min_scores:
|
|
spec_adherence: 8
|
|
model_override: '{{ model_a }}'
|
|
output:
|
|
path: /workspace/project/output.md
|
|
min_bytes: 1500
|
|
post_security_b:
|
|
name: Post-Synthesis Security Review ({{ model_b or 'auto' }})
|
|
depends_on:
|
|
- synthesize
|
|
prompt: "You are performing a security review of a FINAL synthesized specification.\n\
|
|
This spec was produced by merging two competing drafts and applying\nreview\
|
|
\ feedback. Your job: find security issues that survived synthesis\nor were\
|
|
\ introduced by the merge process.\n\n## Synthesized Spec\n<<ARTIFACT:synthesize:output>>\n\
|
|
\n## Best Practice References\nRead these thoroughly before reviewing:\n- `/opt/harness/context/planning/best-practices/security-architecture.md`\n\
|
|
- `/opt/harness/context/planning/best-practices/llm-code-security.md`\n- `/opt/harness/context/planning/best-practices/api-design.md`\
|
|
\ (if it exists)\n\n{% if repo %}\n## Project Context\nThe project repo has\
|
|
\ been cloned. Read `CLAUDE.md` and any existing `spec/` files\nfor architectural\
|
|
\ context (existing auth patterns, data flows, trust boundaries).\n{% endif\
|
|
\ %}\n\n## Review Focus\n1. **Injection risks** — SQL, command, template, prompt,\
|
|
\ SSRF, path traversal\n2. **Input validation** — all inputs validated at system\
|
|
\ boundaries? Size limits?\n3. **Authentication/Authorization** — who can access\
|
|
\ what? Identity spoofing?\n4. **Secret handling** — credentials exposed? Server\
|
|
\ boundary rule?\n5. **Resource exhaustion** — unbounded loops, unlimited sizes,\
|
|
\ missing timeouts?\n6. **Data integrity** — race conditions, TOCTOU, atomic\
|
|
\ operations?\n7. **LLM-specific risks** — prompt injection, hallucinated deps,\
|
|
\ over-permissive defaults\n8. **Synthesis artifacts** — inconsistencies between\
|
|
\ merged sections, conflicting\n requirements, gaps where one plan's approach\
|
|
\ was dropped but the replacement\n was incomplete\n\n## Output Format\nStructure\
|
|
\ your output into two clear sections:\n\n### Fixes (implement directly — no\
|
|
\ human decision needed)\nFor each finding that has an unambiguous fix:\n- **ID:**\
|
|
\ F-NN\n- **Severity:** CRITICAL / HIGH / MEDIUM / LOW / INFO\n- **Requirement:**\
|
|
\ affected requirement ID(s)\n- **Issue:** what's wrong\n- **Current text:**\
|
|
\ quote the problematic spec text\n- **Fixed text:** exact replacement text\n\
|
|
- **Why:** rationale for the fix\n\n### Escalations (need human decision)\n\
|
|
For each finding where reasonable people could disagree on the fix:\n- **ID:**\
|
|
\ E-NN\n- **Severity:** CRITICAL / HIGH / MEDIUM / LOW\n- **Requirement:** affected\
|
|
\ requirement ID(s)\n- **Issue:** what's wrong\n- **Options:** concrete choices\
|
|
\ (A, B, C) with trade-offs\n- **Default recommendation:** what you'd pick and\
|
|
\ why\n\nBe specific. Quote exact spec text. Provide exact replacement text\
|
|
\ for fixes.\nDo NOT suggest vague improvements (\"consider adding validation\"\
|
|
) — specify\nexactly what validation, on what field, with what error code.\n\
|
|
\nWrite to /workspace/project/output.md\n"
|
|
harness: planning/v1
|
|
requirements:
|
|
min_scores:
|
|
spec_adherence: 8
|
|
model_override: '{{ model_b }}'
|
|
output:
|
|
path: /workspace/project/output.md
|
|
min_bytes: 1500
|
|
auto_fix:
|
|
name: Apply Security Fixes
|
|
depends_on:
|
|
- post_security_a
|
|
- post_security_b
|
|
prompt: "You are implementing security fixes on a specification.\nTwo independent\
|
|
\ security reviews have been performed on the synthesized spec.\nYour job: apply\
|
|
\ all non-controversial fixes and prepare escalations for human review.\n\n\
|
|
## Synthesized Spec (the document to modify)\n<<ARTIFACT:synthesize:output>>\n\
|
|
\n## Security Review A\n<<ARTIFACT:post_security_a:output>>\n\n## Security Review\
|
|
\ B\n<<ARTIFACT:post_security_b:output>>\n\n## Instructions\n\n### Step 1: Triage\
|
|
\ and Deduplicate\nCompare both reviews. Many findings will overlap (same issue\
|
|
\ found by both).\nCreate a single merged list with:\n- Deduplicated findings\
|
|
\ (note when both reviewers found the same issue)\n- Severity from the stricter\
|
|
\ reviewer (if A says HIGH and B says MEDIUM, use HIGH)\n\n### Step 2: Categorize\n\
|
|
Split findings into:\n- **Fixes** — clear, unambiguous improvements. Apply these\
|
|
\ directly.\n- **Escalations** — findings where the fix involves a design trade-off,\n\
|
|
\ changes the external API, or where the two reviewers disagree on the approach.\n\
|
|
\n### Step 3: Apply Fixes\nFor each fix, modify the spec text directly. Track\
|
|
\ what you changed:\n- Requirement ID\n- What changed (old text → new text,\
|
|
\ summarized)\n- Which review(s) identified the issue\n\n### Step 4: Write Escalation\
|
|
\ Summary\nFor each escalation, include:\n- Finding description\n- Which reviewer(s)\
|
|
\ raised it\n- Options with trade-offs\n- Default recommendation\n\n## Output\n\
|
|
Write TWO files:\n\n**`/workspace/project/output.md`** — The complete updated\
|
|
\ spec with all\nnon-escalated fixes applied. This should be the full spec text,\
|
|
\ ready to save\nas the final spec file. Include a comment at the top:\n`<!--\
|
|
\ Security fixes applied: N fixes from post-synthesis review. M escalations\
|
|
\ pending human review. -->`\n\n**`/workspace/project/escalations.md`** — Escalation\
|
|
\ summary for human\nreview. Include:\n- Total findings: N (X fixes applied,\
|
|
\ Y escalations)\n- Deduplication stats (how many found by both reviewers)\n\
|
|
- Each escalation with options and default recommendation\n- \"Accepted risks\"\
|
|
\ section for INFO-level findings that don't need action\n"
|
|
harness: planning/v1
|
|
requirements:
|
|
min_scores:
|
|
complexity: 9
|
|
spec_adherence: 9
|
|
output:
|
|
path: /workspace/project/output.md
|
|
min_bytes: 500
|
|
final_fix:
|
|
name: Apply Escalation Decisions
|
|
depends_on:
|
|
- auto_fix
|
|
prompt: "You are applying human decisions to a specification that has already\
|
|
\ had\nnon-controversial security fixes applied.\n\n## Spec with Auto-Fixes\
|
|
\ Applied\n<<ARTIFACT:auto_fix:output>>\n\n## Escalation Summary (with human\
|
|
\ decisions)\n<<ARTIFACT:auto_fix:escalations>>\n\n## Instructions\n\n1. Read\
|
|
\ the escalation summary. The human has annotated each escalation\n with their\
|
|
\ decision (which option to implement, or a custom approach).\n\n2. For each\
|
|
\ escalation where the human provided a decision:\n - Apply the chosen fix\
|
|
\ to the spec text\n - If the human chose a custom approach, implement it\
|
|
\ faithfully\n\n3. For escalations where the human did NOT provide a decision:\n\
|
|
\ - Apply the default recommendation noted in the escalation\n - Mark with\
|
|
\ `<!-- DEFAULT APPLIED: [topic] — human did not override -->`\n\n4. For any\
|
|
\ \"Accepted risks\" the human flagged as needing action after all:\n - Implement\
|
|
\ the fix\n\n5. Verify requirement ID uniqueness — no duplicate IDs after all\
|
|
\ changes.\n\n6. Verify all scenarios still match their requirements after text\
|
|
\ changes.\n\n## Output\nWrite the FINAL spec to `/workspace/project/output.md`.\n\
|
|
This is the production-ready spec. No review commentary, no TODOs,\nno pending\
|
|
\ decisions. Every requirement has an ID, rationale, and scenario.\n\nAt the\
|
|
\ top, include a summary comment:\n`<!-- Final spec: N escalations resolved,\
|
|
\ M defaults applied. Ready for merge. -->`\n"
|
|
harness: planning/v1
|
|
requirements:
|
|
min_scores:
|
|
complexity: 9
|
|
spec_adherence: 9
|
|
model_override: '{{ best_model }}'
|
|
output:
|
|
path: /workspace/project/output.md
|
|
min_bytes: 500
|
|
scoring:
|
|
type: comparative_review
|
|
reviews:
|
|
- review_node: spec_review_a
|
|
scored_model_node: plan_b
|
|
dimension: spec_adherence
|
|
- review_node: security_review_a
|
|
scored_model_node: plan_b
|
|
dimension: spec_adherence
|
|
- review_node: tdd_review_a
|
|
scored_model_node: plan_b
|
|
dimension: test_pass_rate
|
|
- review_node: spec_review_b
|
|
scored_model_node: plan_a
|
|
dimension: spec_adherence
|
|
- review_node: security_review_b
|
|
scored_model_node: plan_a
|
|
dimension: spec_adherence
|
|
- review_node: tdd_review_b
|
|
scored_model_node: plan_a
|
|
dimension: test_pass_rate
|
|
method: fewer_improvement_points_wins
|