small-scripts/specs/score-paragraphs.spec.md

# score-paragraphs

## Purpose

Score markdown paragraphs against a style guide via `claude -p`, inserting a
compact per-paragraph score block after each scored paragraph and, on request,
emitting the same results as machine-readable JSON for downstream tooling.

This is a generalised port of `~/dev/claude/octopus/goes/scripts/score-paragraphs.py`
(the GOES book's voice-scoring engine). The scoring engine — chunking, hashing,
the five-dimension rubric, and the `claude -p` prompt — is unchanged. What
changed is the coupling to the GOES repo layout: the style guide is now an
explicit `--style` argument instead of a repo-relative default, and `--book`
(GOES chapter/appendix discovery) is gone.

## Usage

```
score-paragraphs --style FILE [OPTIONS] FILE [FILE ...]
```

### Required

| Argument | Description |
|---|---|
| `FILE ...` (positional) | One or more markdown files to score. At least one required. |
| `--style FILE` | Style guide markdown file to score paragraphs against. No default — the caller always states which voice the paragraphs are graded against. |

### Options

| Flag | Default | Description |
|---|---|---|
| `-o, --output FILE` | `<input>-scored.md` next to the input | Output file. Only valid with a single input `FILE`. |
| `--json FILE\|-` | (none) | Also emit machine-readable results (see [JSON output](#json-output)). `-` writes to stdout. |
| `--parallel N` | `4` | Number of parallel `claude -p` calls per file. |
| `--force` | off | Re-score all eligible paragraphs, ignoring existing hash matches. |
| `--dryrun`, `-n` | off | Preview which paragraphs would be scored/skipped and which `claude -p` calls would run. Makes no `claude` calls and writes no files (see [Dryrun behaviour](#dryrun-behaviour)). |
| `--help`, `-h` | — | Show usage and exit 0. |

Requires the `claude` CLI on PATH for any non-dryrun run — the script exits 1
immediately if it is missing, before touching any input.

## Scoring dimensions (each 0–3, unchanged from the source engine)

| Dim | Name | What it measures |
|---|---|---|
| E | Evidence | Specific claims backed by named sources / research links |
| J | Judgment | Direct language; no hedging where evidence supports a claim |
| V | Voice | Free of LLM tells, buzzwords, corporate softening |
| R | Rhythm | Sentence and paragraph length variety; not uniform blocks |
| G | Register | Appropriate to the target genre (not flat framework, not academic, etc.) |

Total is `E+J+V+R+G`, out of 15.

## Score block format

Inserted after each scored paragraph, unchanged from the source engine:

```
> `◈` E:3 · J:2 · V:3 · R:2 · G:3 = **13/15** `¶a3f5b2`
> ⚑ *"potentially" (s2) — evidence supports direct claim*
```

The `¶xxxxxxx` suffix is the first 7 hex characters of the SHA-256 hash of the
paragraph's stripped text. The engine is designed so that, on re-runs, a
paragraph whose hash matches the hash embedded in its existing score block is
skipped (no `claude` call) and the block is left in place, while a paragraph
whose text changed gets re-scored and its stale block replaced; `--force` is
designed to re-score every eligible paragraph regardless of hash. **In
practice this hash-comparison path never executes** — see
[Known limitation](#known-limitation-hash-comparison-is-unreachable) below.
What re-runs actually do: a paragraph that already has a score block attached
(in the exact adjacent format this script itself writes — no blank line
between the paragraph and its `` `◈` `` line) is not re-parsed as a scoreable
paragraph at all on the next run, with or without `--force`. Only paragraphs
with **no** score block attached get (re-)scored.

## Behaviour

1. Unless `--dryrun`, verify `claude` is on PATH; exit 1 with an error if not.
2. Resolve input paths (relative paths resolve against the current working
   directory). Exit 1 if `--output` is given with more than one input file.
   Exit 1 if any input file does not exist.
3. Resolve each input's output path: `--output` if given (single-file only),
   else `<input-stem>-scored<suffix>` next to the input.
4. Load `--style` and exit 1 if it does not exist.
5. For each input file:
   a. Split the markdown into chunks tagged `content` (substantive prose
      paragraphs) or `skip` (YAML frontmatter, fenced code blocks, headings,
      tables, horizontal rules, blank lines, existing score blocks).
   b. Paragraphs under `MIN_WORDS` (25) words are never scored. Paragraphs
      that already have a score block attached in the script's own output
      format are **not classified as `content` at all** by the chunker (see
      [Known limitation](#known-limitation-hash-comparison-is-unreachable)) —
      they fall out of scoring consideration entirely, silently, forever.
   c. For each remaining `content` paragraph, compute its hash. If `--force`
      is not set and the immediately following chunk is an existing score
      block whose embedded hash matches, skip it (unchanged) — this branch
      is designed-for but, per the limitation above, structurally
      unreachable. Otherwise queue it for scoring; if a stale score block
      immediately follows, mark it for removal once the new block is written
      (also unreachable for the same reason).
   d. Score all queued paragraphs in parallel (`--parallel` workers) by
      calling `claude -p <prompt>` per paragraph, where the prompt embeds the
      style guide text and the paragraph text, and asks for a JSON object
      `{"evidence":0-3,"judgment":0-3,"voice":0-3,"rhythm":0-3,"register":0-3,"flags":[...],"note":"..."}`.
      A timeout, a JSON parse failure, or any other subprocess error produces
      an all-zero score with a `scoring error: ...` flag rather than aborting
      the run.
   e. Write the output file: original text with a score block inserted after
      each newly-scored or unchanged-and-already-scored paragraph; stale
      blocks for changed paragraphs are dropped.
6. Print a per-file summary line and a run-total summary line.
7. If `--json` was given (and this is not a dryrun — see below), assemble and
   write the JSON payload.

## JSON output

`--json FILE` (or `--json -` for stdout) emits, after scoring completes:

```json
{
  "style": "/abs/path/to/style-guide.md",
  "dryrun": false,
  "files": [
    {
      "input": "/abs/path/to/chapter-01.md",
      "output": "/abs/path/to/chapter-01-scored.md",
      "paragraphs": [
        {
          "index": 1,
          "hash": "a3f5b2c",
          "action": "scored",
          "scores": {"evidence": 3, "judgment": 2, "voice": 3, "rhythm": 2, "register": 3},
          "total": 13,
          "flags": ["hedge: potentially"],
          "note": "evidence supports a direct claim here"
        },
        {
          "index": 2,
          "hash": "9c1d0ef",
          "action": "reused",
          "scores": {"evidence": 2, "judgment": 2, "voice": 2, "rhythm": 2, "register": 2},
          "total": 10,
          "flags": [],
          "note": ""
        }
      ],
      "summary": {
        "mean": 11.5,
        "min": 10,
        "paragraphs_scored": 2,
        "paragraphs_skipped": 1
      }
    }
  ]
}
```

- `paragraphs` lists every paragraph the chunker still recognises as
  `content` and that clears `MIN_WORDS`, in document order, 1-indexed via
  `index`. `action` is `"scored"` (freshly scored this run) or `"reused"`
  (hash matched an existing block; the score is parsed back out of that
  block with no `claude` call). **`"reused"` is defined for completeness but
  is not reachable through this script's own output format** — see
  [Known limitation](#known-limitation-hash-comparison-is-unreachable).
  Paragraphs below `MIN_WORDS`, and paragraphs that already carry an
  attached score block, never appear in this list at all — the latter are
  not "reused with a null diff", they are simply absent.
- `summary.paragraphs_scored` is `len(paragraphs)`; `summary.paragraphs_skipped`
  is the count of too-short `content` paragraphs excluded from `paragraphs`.
  It does **not** count already-scored paragraphs, since those are not
  `content` chunks at all by the time this runs (they contribute to neither
  `paragraphs_scored` nor `paragraphs_skipped` — they are invisible to this
  accounting, not merely uncounted). `summary.mean`/`summary.min` are
  computed over `paragraphs[*].total`; both are `null` if the list is empty.
- This is the interface a future `review` work type would consume to find
  paragraphs below a threshold on the `/15` scale (see `IDLE-DRAFT-PLAN.md`,
  `review_score_threshold`) — **but only on a first, from-scratch scoring
  pass.** On any re-run of an already-scored file, `paragraphs` (and
  therefore the JSON) covers only paragraphs that had no score block at all
  going in; it is not a complete, current picture of every paragraph's score.
  A caller that wants a complete `/15` picture of a fully-scored file must
  keep its own copy of prior JSON output and merge it with each incremental
  run's output, or parse the score blocks out of the `-scored.md` file
  directly.

## Dryrun behaviour

`--dryrun`/`-n` makes no `claude -p` calls and writes no files. For each
`content`-classified paragraph it reports one of:

- **would score** — eligible, no matching existing block found for it at
  parse time; shows the hash it would be scored under and that a
  `claude -p` call would be made.
- **skip (unchanged)** — eligible, hash matches an existing score block
  found immediately after it; no call would be made. Defined for
  completeness — see [Known limitation](#known-limitation-hash-comparison-is-unreachable);
  in practice this line is never printed by a real run.
- **skip (too short)** — under `MIN_WORDS` words; never scored.

Paragraphs that already have a score block attached are not `content` at
all by dryrun time, so they produce **no line whatsoever** — they are not
"unchanged", they are invisible to the paragraph walk, exactly as in a real
run.

Followed by a per-file and run-total count, mirroring the real-run summary
line shape.

```
[dryrun] score-paragraphs — no claude calls will be made, no files will be written
Style:    /home/user/style/voice.md
Parallel: 4 workers

Scoring: /home/user/book/chapter-01.md → /home/user/book/chapter-01-scored.md
  1 would be scored, 0 unchanged (hash match, would skip), 1 too short (skipped)
    [would score]      ¶7f0e412 (#1, 33 words) — would run: claude -p <scoring prompt>
    [skip too-short]   paragraph ~6 words

Total: 1 would be scored, 0 unchanged.
```

(Two other paragraphs in `chapter-01.md` already carry score blocks from a
prior run and simply do not appear above at all — see
[Known limitation](#known-limitation-hash-comparison-is-unreachable).)

### Dryrun + `--json`

`--json -` (stdout) is honoured under `--dryrun`: no filesystem writes occur
either way, so the preview JSON is printed. Progress lines above move to
stderr in this case, keeping stdout pure JSON.

`--json FILE` under `--dryrun` is **not** written (writing a file is a change,
which `--dryrun` promises not to make); instead the script logs
`[dryrun] would write JSON results to FILE`.

In both cases the payload shape matches the real-run schema with `"dryrun":
true` and one difference: paragraphs with `action: "would_score"` have
`scores: null`, `total: null`, `flags: []`, `note: ""` (the score is not
known without calling `claude`). Any `action: "reused"` entries (see the
[Known limitation](#known-limitation-hash-comparison-is-unreachable) — not
reachable via this script's own output format, but defined in case a
hand-edited file makes it reachable) still carry real scores, parsed back
out of the existing score block — no `claude` call needed. `summary.mean`/
`summary.min` are computed only over paragraphs with a non-null `total`.

## Edge cases

| Case | Handling |
|---|---|
| `claude` not on PATH, not `--dryrun` | Error to stderr, exit 1, before touching any input file |
| `--output` with multiple input files | Error: `--output can only be used with a single input file`, exit 1 |
| Input file does not exist | Error: `file not found: <path>`, exit 1 |
| `--style` file does not exist | Error: `style guide not found: <path>`, exit 1 |
| No positional files given | argparse usage error, exit 2 |
| Paragraph scoring subprocess errors (timeout, bad JSON, non-zero exit) | All-zero score, `scoring error: <reason>` flag; run continues |
| A file with zero eligible paragraphs | Output is a byte-identical copy (skip chunks only); JSON `paragraphs: []`, `summary.mean/min: null` |
| `--force` with `--json` | Every *`content`-classified* paragraph gets `action: "scored"`; already-scored paragraphs are still invisible regardless — `--force` does not resurrect them (see [Known limitation](#known-limitation-hash-comparison-is-unreachable)) |
| Re-running on an already-scored file | Paragraphs with an attached score block are silently excluded from processing and from `paragraphs_scored`/`paragraphs_skipped`/JSON entirely — the run only touches paragraphs with no block yet. This is the practical, verified behaviour, not a hash-based decision |
| `--json -` (stdout) without `--dryrun` | Progress lines still go to stdout normally *before* the final JSON print — the caller is expected to take the last JSON blob, or redirect and parse only after the run-total line. `--json -` combined with multiple files is fine (one JSON payload with a `files` array). |

## Known limitation: hash comparison is unreachable

Verified during the port (not introduced by it — present identically in the
source `score-paragraphs.py`, confirmed against real GOES production output,
`book/chapter-01-the-amplifier-test-scored.md`):

The chunker (`split_into_chunks`) and the writer (`format_score_block` /
`process_file`) disagree about spacing. The writer inserts a score block with
**zero** blank-line separation from its paragraph. The chunker's score-block
branch calls `flush("skip")`, which flushes *whatever text is currently
accumulating* — not just the score-block line. Because there is no blank
line, the paragraph itself is still accumulating when that branch fires, so
the paragraph is absorbed into the same `"skip"` chunk as its score block.
It is no longer `kind == "content"` on any subsequent parse.

Consequence: `_jobs_for_file`'s hash-comparison branch (compare an existing
block's embedded hash against the paragraph's current hash to decide
skip-vs-stale) requires a `content` chunk immediately followed by a `skip`
chunk whose first line is a score block. That combination cannot occur for
any file this script wrote — inserting a blank line to *avoid* the swallow
doesn't help either, because then the chunk immediately following the
paragraph is the blank line, not the score block (off by one). Checked
exhaustively against a fresh two-paragraph file, a hand-edited
blank-line-separated file, and the real GOES chapter file: **0 paragraphs
out of 15+ already-scored ones were ever re-examined, with or without
`--force`.**

Net effect, stated plainly: this script cannot currently detect that an
already-scored paragraph's text changed, and cannot force a re-score of an
already-scored paragraph. What it *can* do reliably: score paragraphs that
have never been scored before, leaving previously-scored paragraphs
untouched (which happens to look like "hash-skip working correctly" for the
common "add new paragraphs, don't touch old ones" case — the two are
behaviourally indistinguishable until someone edits an already-scored
paragraph and expects a re-score).

This is a scoring-engine bug, not a generalisation concern, so it is out of
scope for this port to fix (`scripts/score-paragraphs` keeps the chunker and
writer exactly as ported, with a code comment pointing here). Anyone relying
on this script for edit-detection — notably `IDLE-DRAFT-PLAN.md`'s `review`
work type, which assumes `--json` gives a complete, current `/15` picture on
every run — should design around this rather than assume the docstring's
aspirational description.

## Examples

```bash
# Single file, default output path
score-paragraphs --style style/voice.md book/chapter-01.md

# Explicit output path
score-paragraphs --style style/voice.md book/chapter-01.md -o /tmp/ch01-scored.md

# Multiple files, plus a JSON results file for a downstream review pass
score-paragraphs --style style/voice.md book/chapter-*.md --json /tmp/scores.json

# Preview only — no claude calls, no files written
score-paragraphs --style style/voice.md book/chapter-01.md --dryrun

# Force re-score everything, parallelism of 8
score-paragraphs --style style/voice.md book/chapter-01.md --force --parallel 8
```