small-scripts/specs/score-paragraphs.spec.md

score-paragraphs

Purpose

Score markdown paragraphs against a style guide via claude -p, inserting a compact per-paragraph score block after each scored paragraph and, on request, emitting the same results as machine-readable JSON for downstream tooling.

This is a generalised port of ~/dev/claude/octopus/goes/scripts/score-paragraphs.py (the GOES book's voice-scoring engine). The scoring engine — chunking, hashing, the five-dimension rubric, and the claude -p prompt — is unchanged. What changed is the coupling to the GOES repo layout: the style guide is now an explicit --style argument instead of a repo-relative default, and --book (GOES chapter/appendix discovery) is gone.

Usage

score-paragraphs --style FILE [OPTIONS] FILE [FILE ...]

Required

Argument	Description
FILE ... (positional)	One or more markdown files to score. At least one required.
--style FILE	Style guide markdown file to score paragraphs against. No default — the caller always states which voice the paragraphs are graded against.

Options

Flag	Default	Description
-o, --output FILE	<input>-scored.md next to the input	Output file. Only valid with a single input FILE.
--json FILE|-	(none)	Also emit machine-readable results (see JSON output). - writes to stdout.
--parallel N	4	Number of parallel claude -p calls per file.
--force	off	Re-score all eligible paragraphs, ignoring existing hash matches.
--dryrun, -n	off	Preview which paragraphs would be scored/skipped and which claude -p calls would run. Makes no claude calls and writes no files (see Dryrun behaviour).
--help, -h	—	Show usage and exit 0.

Requires the claude CLI on PATH for any non-dryrun run — the script exits 1 immediately if it is missing, before touching any input.

Scoring dimensions (each 0–3, unchanged from the source engine)

Dim	Name	What it measures
E	Evidence	Specific claims backed by named sources / research links
J	Judgment	Direct language; no hedging where evidence supports a claim
V	Voice	Free of LLM tells, buzzwords, corporate softening
R	Rhythm	Sentence and paragraph length variety; not uniform blocks
G	Register	Appropriate to the target genre (not flat framework, not academic, etc.)

Total is E+J+V+R+G, out of 15.

Score block format

Inserted after each scored paragraph, unchanged from the source engine:

> `◈` E:3 · J:2 · V:3 · R:2 · G:3 = **13/15** `¶a3f5b2`
> ⚑ *"potentially" (s2) — evidence supports direct claim*

The ¶xxxxxxx suffix is the first 7 hex characters of the SHA-256 hash of the paragraph's stripped text. The engine is designed so that, on re-runs, a paragraph whose hash matches the hash embedded in its existing score block is skipped (no claude call) and the block is left in place, while a paragraph whose text changed gets re-scored and its stale block replaced; --force is designed to re-score every eligible paragraph regardless of hash. In practice this hash-comparison path never executes — see Known limitation below. What re-runs actually do: a paragraph that already has a score block attached (in the exact adjacent format this script itself writes — no blank line between the paragraph and its `◈` line) is not re-parsed as a scoreable paragraph at all on the next run, with or without --force. Only paragraphs with no score block attached get (re-)scored.

Behaviour

1. Unless --dryrun, verify claude is on PATH; exit 1 with an error if not.
2. Resolve input paths (relative paths resolve against the current working directory). Exit 1 if --output is given with more than one input file. Exit 1 if any input file does not exist.
3. Resolve each input's output path: --output if given (single-file only), else <input-stem>-scored<suffix> next to the input.
4. Load --style and exit 1 if it does not exist.
5. For each input file: a. Split the markdown into chunks tagged content (substantive prose paragraphs) or skip (YAML frontmatter, fenced code blocks, headings, tables, horizontal rules, blank lines, existing score blocks). b. Paragraphs under MIN_WORDS (25) words are never scored. Paragraphs that already have a score block attached in the script's own output format are not classified as content at all by the chunker (see Known limitation) — they fall out of scoring consideration entirely, silently, forever. c. For each remaining content paragraph, compute its hash. If --force is not set and the immediately following chunk is an existing score block whose embedded hash matches, skip it (unchanged) — this branch is designed-for but, per the limitation above, structurally unreachable. Otherwise queue it for scoring; if a stale score block immediately follows, mark it for removal once the new block is written (also unreachable for the same reason). d. Score all queued paragraphs in parallel (--parallel workers) by calling claude -p <prompt> per paragraph, where the prompt embeds the style guide text and the paragraph text, and asks for a JSON object {"evidence":0-3,"judgment":0-3,"voice":0-3,"rhythm":0-3,"register":0-3,"flags":[...],"note":"..."}. A timeout, a JSON parse failure, or any other subprocess error produces an all-zero score with a scoring error: ... flag rather than aborting the run. e. Write the output file: original text with a score block inserted after each newly-scored or unchanged-and-already-scored paragraph; stale blocks for changed paragraphs are dropped.
6. Print a per-file summary line and a run-total summary line.
7. If --json was given (and this is not a dryrun — see below), assemble and write the JSON payload.

JSON output

--json FILE (or --json - for stdout) emits, after scoring completes:

{
  "style": "/abs/path/to/style-guide.md",
  "dryrun": false,
  "files": [
    {
      "input": "/abs/path/to/chapter-01.md",
      "output": "/abs/path/to/chapter-01-scored.md",
      "paragraphs": [
        {
          "index": 1,
          "hash": "a3f5b2c",
          "action": "scored",
          "scores": {"evidence": 3, "judgment": 2, "voice": 3, "rhythm": 2, "register": 3},
          "total": 13,
          "flags": ["hedge: potentially"],
          "note": "evidence supports a direct claim here"
        },
        {
          "index": 2,
          "hash": "9c1d0ef",
          "action": "reused",
          "scores": {"evidence": 2, "judgment": 2, "voice": 2, "rhythm": 2, "register": 2},
          "total": 10,
          "flags": [],
          "note": ""
        }
      ],
      "summary": {
        "mean": 11.5,
        "min": 10,
        "paragraphs_scored": 2,
        "paragraphs_skipped": 1
      }
    }
  ]
}

• paragraphs lists every paragraph the chunker still recognises as content and that clears MIN_WORDS, in document order, 1-indexed via index. action is "scored" (freshly scored this run) or "reused" (hash matched an existing block; the score is parsed back out of that block with no claude call). "reused" is defined for completeness but is not reachable through this script's own output format — see Known limitation. Paragraphs below MIN_WORDS, and paragraphs that already carry an attached score block, never appear in this list at all — the latter are not "reused with a null diff", they are simply absent.
• summary.paragraphs_scored is len(paragraphs); summary.paragraphs_skipped is the count of too-short content paragraphs excluded from paragraphs. It does not count already-scored paragraphs, since those are not content chunks at all by the time this runs (they contribute to neither paragraphs_scored nor paragraphs_skipped — they are invisible to this accounting, not merely uncounted). summary.mean/summary.min are computed over paragraphs[*].total; both are null if the list is empty.
• This is the interface a future review work type would consume to find paragraphs below a threshold on the /15 scale (see IDLE-DRAFT-PLAN.md, review_score_threshold) — but only on a first, from-scratch scoring pass. On any re-run of an already-scored file, paragraphs (and therefore the JSON) covers only paragraphs that had no score block at all going in; it is not a complete, current picture of every paragraph's score. A caller that wants a complete /15 picture of a fully-scored file must keep its own copy of prior JSON output and merge it with each incremental run's output, or parse the score blocks out of the -scored.md file directly.

Dryrun behaviour

--dryrun/-n makes no claude -p calls and writes no files. For each content-classified paragraph it reports one of:

• would score — eligible, no matching existing block found for it at parse time; shows the hash it would be scored under and that a claude -p call would be made.
• skip (unchanged) — eligible, hash matches an existing score block found immediately after it; no call would be made. Defined for completeness — see Known limitation; in practice this line is never printed by a real run.
• skip (too short) — under MIN_WORDS words; never scored.

Paragraphs that already have a score block attached are not content at all by dryrun time, so they produce no line whatsoever — they are not "unchanged", they are invisible to the paragraph walk, exactly as in a real run.

Followed by a per-file and run-total count, mirroring the real-run summary line shape.

[dryrun] score-paragraphs — no claude calls will be made, no files will be written
Style:    /home/user/style/voice.md
Parallel: 4 workers

Scoring: /home/user/book/chapter-01.md → /home/user/book/chapter-01-scored.md
  1 would be scored, 0 unchanged (hash match, would skip), 1 too short (skipped)
    [would score]      ¶7f0e412 (#1, 33 words) — would run: claude -p <scoring prompt>
    [skip too-short]   paragraph ~6 words

Total: 1 would be scored, 0 unchanged.

(Two other paragraphs in chapter-01.md already carry score blocks from a prior run and simply do not appear above at all — see Known limitation.)

Dryrun + --json

--json - (stdout) is honoured under --dryrun: no filesystem writes occur either way, so the preview JSON is printed. Progress lines above move to stderr in this case, keeping stdout pure JSON.

--json FILE under --dryrun is not written (writing a file is a change, which --dryrun promises not to make); instead the script logs [dryrun] would write JSON results to FILE.

In both cases the payload shape matches the real-run schema with "dryrun": true and one difference: paragraphs with action: "would_score" have scores: null, total: null, flags: [], note: "" (the score is not known without calling claude). Any action: "reused" entries (see the Known limitation — not reachable via this script's own output format, but defined in case a hand-edited file makes it reachable) still carry real scores, parsed back out of the existing score block — no claude call needed. summary.mean/ summary.min are computed only over paragraphs with a non-null total.

Edge cases

Case	Handling
claude not on PATH, not --dryrun	Error to stderr, exit 1, before touching any input file
--output with multiple input files	Error: --output can only be used with a single input file, exit 1
Input file does not exist	Error: file not found: <path>, exit 1
--style file does not exist	Error: style guide not found: <path>, exit 1
No positional files given	argparse usage error, exit 2
Paragraph scoring subprocess errors (timeout, bad JSON, non-zero exit)	All-zero score, scoring error: <reason> flag; run continues
A file with zero eligible paragraphs	Output is a byte-identical copy (skip chunks only); JSON paragraphs: [], summary.mean/min: null
--force with --json	Every content-classified paragraph gets action: "scored"; already-scored paragraphs are still invisible regardless — --force does not resurrect them (see Known limitation)
Re-running on an already-scored file	Paragraphs with an attached score block are silently excluded from processing and from paragraphs_scored/paragraphs_skipped/JSON entirely — the run only touches paragraphs with no block yet. This is the practical, verified behaviour, not a hash-based decision
--json - (stdout) without --dryrun	Progress lines still go to stdout normally before the final JSON print — the caller is expected to take the last JSON blob, or redirect and parse only after the run-total line. --json - combined with multiple files is fine (one JSON payload with a files array).

Known limitation: hash comparison is unreachable

Verified during the port (not introduced by it — present identically in the source score-paragraphs.py, confirmed against real GOES production output, book/chapter-01-the-amplifier-test-scored.md):

The chunker (split_into_chunks) and the writer (format_score_block / process_file) disagree about spacing. The writer inserts a score block with zero blank-line separation from its paragraph. The chunker's score-block branch calls flush("skip"), which flushes whatever text is currently accumulating — not just the score-block line. Because there is no blank line, the paragraph itself is still accumulating when that branch fires, so the paragraph is absorbed into the same "skip" chunk as its score block. It is no longer kind == "content" on any subsequent parse.

Consequence: _jobs_for_file's hash-comparison branch (compare an existing block's embedded hash against the paragraph's current hash to decide skip-vs-stale) requires a content chunk immediately followed by a skip chunk whose first line is a score block. That combination cannot occur for any file this script wrote — inserting a blank line to avoid the swallow doesn't help either, because then the chunk immediately following the paragraph is the blank line, not the score block (off by one). Checked exhaustively against a fresh two-paragraph file, a hand-edited blank-line-separated file, and the real GOES chapter file: 0 paragraphs out of 15+ already-scored ones were ever re-examined, with or without --force.

Net effect, stated plainly: this script cannot currently detect that an already-scored paragraph's text changed, and cannot force a re-score of an already-scored paragraph. What it can do reliably: score paragraphs that have never been scored before, leaving previously-scored paragraphs untouched (which happens to look like "hash-skip working correctly" for the common "add new paragraphs, don't touch old ones" case — the two are behaviourally indistinguishable until someone edits an already-scored paragraph and expects a re-score).

This is a scoring-engine bug, not a generalisation concern, so it is out of scope for this port to fix (scripts/score-paragraphs keeps the chunker and writer exactly as ported, with a code comment pointing here). Anyone relying on this script for edit-detection — notably IDLE-DRAFT-PLAN.md's review work type, which assumes --json gives a complete, current /15 picture on every run — should design around this rather than assume the docstring's aspirational description.

Examples

# Single file, default output path
score-paragraphs --style style/voice.md book/chapter-01.md

# Explicit output path
score-paragraphs --style style/voice.md book/chapter-01.md -o /tmp/ch01-scored.md

# Multiple files, plus a JSON results file for a downstream review pass
score-paragraphs --style style/voice.md book/chapter-*.md --json /tmp/scores.json

# Preview only — no claude calls, no files written
score-paragraphs --style style/voice.md book/chapter-01.md --dryrun

# Force re-score everything, parallelism of 8
score-paragraphs --style style/voice.md book/chapter-01.md --force --parallel 8