skynet/small-scripts
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

score-paragraphs: port GOES paragraph scorer as general-purpose script

Browse Source 
Required --style, --json machine-readable output, mandatory --dryrun.
Engine unchanged; known limitation documented: already-scored paragraphs
are never re-examined (present in the original).

Claude-Session: https://claude.ai/code/session_01YQDoWNM7XPPii28khFWoMc
This commit is contained in:
Paul O'Reilly
2026-08-02 21:18:02 +12:00
parent 52eb484ef7
commit f78d292f05
3 changed files with 1409 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

323
specs/score-paragraphs.spec.md Normal file
View File

@@ -0,0 +1,323 @@
# score-paragraphs
## Purpose
Score markdown paragraphs against a style guide via `claude -p`, inserting a
compact per-paragraph score block after each scored paragraph and, on request,
emitting the same results as machine-readable JSON for downstream tooling.
This is a generalised port of `~/dev/claude/octopus/goes/scripts/score-paragraphs.py`
(the GOES book's voice-scoring engine). The scoring engine — chunking, hashing,
the five-dimension rubric, and the `claude -p` prompt — is unchanged. What
changed is the coupling to the GOES repo layout: the style guide is now an
explicit `--style` argument instead of a repo-relative default, and `--book`
(GOES chapter/appendix discovery) is gone.
## Usage
```
score-paragraphs --style FILE [OPTIONS] FILE [FILE ...]
```
### Required
| Argument | Description |
|---|---|
| `FILE ...` (positional) | One or more markdown files to score. At least one required. |
| `--style FILE` | Style guide markdown file to score paragraphs against. No default — the caller always states which voice the paragraphs are graded against. |
### Options
| Flag | Default | Description |
|---|---|---|
| `-o, --output FILE` | `<input>-scored.md` next to the input | Output file. Only valid with a single input `FILE`. |
| `--json FILE\|-` | (none) | Also emit machine-readable results (see [JSON output](#json-output)). `-` writes to stdout. |
| `--parallel N` | `4` | Number of parallel `claude -p` calls per file. |
| `--force` | off | Re-score all eligible paragraphs, ignoring existing hash matches. |
| `--dryrun`, `-n` | off | Preview which paragraphs would be scored/skipped and which `claude -p` calls would run. Makes no `claude` calls and writes no files (see [Dryrun behaviour](#dryrun-behaviour)). |
| `--help`, `-h` | — | Show usage and exit 0. |
Requires the `claude` CLI on PATH for any non-dryrun run — the script exits 1
immediately if it is missing, before touching any input.
## Scoring dimensions (each 0–3, unchanged from the source engine)
| Dim | Name | What it measures |
|---|---|---|
| E | Evidence | Specific claims backed by named sources / research links |
| J | Judgment | Direct language; no hedging where evidence supports a claim |
| V | Voice | Free of LLM tells, buzzwords, corporate softening |
| R | Rhythm | Sentence and paragraph length variety; not uniform blocks |
| G | Register | Appropriate to the target genre (not flat framework, not academic, etc.) |
Total is `E+J+V+R+G`, out of 15.
## Score block format
Inserted after each scored paragraph, unchanged from the source engine:
```
> `◈` E:3 · J:2 · V:3 · R:2 · G:3 = **13/15** `¶a3f5b2`
> ⚑ *"potentially" (s2) — evidence supports direct claim*
```
The `¶xxxxxxx` suffix is the first 7 hex characters of the SHA-256 hash of the
paragraph's stripped text. The engine is designed so that, on re-runs, a
paragraph whose hash matches the hash embedded in its existing score block is
skipped (no `claude` call) and the block is left in place, while a paragraph
whose text changed gets re-scored and its stale block replaced; `--force` is
designed to re-score every eligible paragraph regardless of hash. **In
practice this hash-comparison path never executes** — see
[Known limitation](#known-limitation-hash-comparison-is-unreachable) below.
What re-runs actually do: a paragraph that already has a score block attached
(in the exact adjacent format this script itself writes — no blank line
between the paragraph and its `` `◈` `` line) is not re-parsed as a scoreable
paragraph at all on the next run, with or without `--force`. Only paragraphs
with **no** score block attached get (re-)scored.
## Behaviour
1. Unless `--dryrun`, verify `claude` is on PATH; exit 1 with an error if not.
2. Resolve input paths (relative paths resolve against the current working
directory). Exit 1 if `--output` is given with more than one input file.
Exit 1 if any input file does not exist.
3. Resolve each input's output path: `--output` if given (single-file only),
else `<input-stem>-scored<suffix>` next to the input.
4. Load `--style` and exit 1 if it does not exist.
5. For each input file:
a. Split the markdown into chunks tagged `content` (substantive prose
paragraphs) or `skip` (YAML frontmatter, fenced code blocks, headings,
tables, horizontal rules, blank lines, existing score blocks).
b. Paragraphs under `MIN_WORDS` (25) words are never scored. Paragraphs
that already have a score block attached in the script's own output
format are **not classified as `content` at all** by the chunker (see
[Known limitation](#known-limitation-hash-comparison-is-unreachable)) —
they fall out of scoring consideration entirely, silently, forever.
c. For each remaining `content` paragraph, compute its hash. If `--force`
is not set and the immediately following chunk is an existing score
block whose embedded hash matches, skip it (unchanged) — this branch
is designed-for but, per the limitation above, structurally
unreachable. Otherwise queue it for scoring; if a stale score block
immediately follows, mark it for removal once the new block is written
(also unreachable for the same reason).
d. Score all queued paragraphs in parallel (`--parallel` workers) by
calling `claude -p <prompt>` per paragraph, where the prompt embeds the
style guide text and the paragraph text, and asks for a JSON object
`{"evidence":0-3,"judgment":0-3,"voice":0-3,"rhythm":0-3,"register":0-3,"flags":[...],"note":"..."}`.
A timeout, a JSON parse failure, or any other subprocess error produces
an all-zero score with a `scoring error: ...` flag rather than aborting
the run.
e. Write the output file: original text with a score block inserted after
each newly-scored or unchanged-and-already-scored paragraph; stale
blocks for changed paragraphs are dropped.
6. Print a per-file summary line and a run-total summary line.
7. If `--json` was given (and this is not a dryrun — see below), assemble and
write the JSON payload.
## JSON output
`--json FILE` (or `--json -` for stdout) emits, after scoring completes:
```json
{
"style": "/abs/path/to/style-guide.md",
"dryrun": false,
"files": [
{
"input": "/abs/path/to/chapter-01.md",
"output": "/abs/path/to/chapter-01-scored.md",
"paragraphs": [
{
"index": 1,
"hash": "a3f5b2c",
"action": "scored",
"scores": {"evidence": 3, "judgment": 2, "voice": 3, "rhythm": 2, "register": 3},
"total": 13,
"flags": ["hedge: potentially"],
"note": "evidence supports a direct claim here"
},
{
"index": 2,
"hash": "9c1d0ef",
"action": "reused",
"scores": {"evidence": 2, "judgment": 2, "voice": 2, "rhythm": 2, "register": 2},
"total": 10,
"flags": [],
"note": ""
}
],
"summary": {
"mean": 11.5,
"min": 10,
"paragraphs_scored": 2,
"paragraphs_skipped": 1
}
}
]
}
```
- `paragraphs` lists every paragraph the chunker still recognises as
`content` and that clears `MIN_WORDS`, in document order, 1-indexed via
`index`. `action` is `"scored"` (freshly scored this run) or `"reused"`
(hash matched an existing block; the score is parsed back out of that
block with no `claude` call). **`"reused"` is defined for completeness but
is not reachable through this script's own output format** — see
[Known limitation](#known-limitation-hash-comparison-is-unreachable).
Paragraphs below `MIN_WORDS`, and paragraphs that already carry an
attached score block, never appear in this list at all — the latter are
not "reused with a null diff", they are simply absent.
- `summary.paragraphs_scored` is `len(paragraphs)`; `summary.paragraphs_skipped`
is the count of too-short `content` paragraphs excluded from `paragraphs`.
It does **not** count already-scored paragraphs, since those are not
`content` chunks at all by the time this runs (they contribute to neither
`paragraphs_scored` nor `paragraphs_skipped` — they are invisible to this
accounting, not merely uncounted). `summary.mean`/`summary.min` are
computed over `paragraphs[*].total`; both are `null` if the list is empty.
- This is the interface a future `review` work type would consume to find
paragraphs below a threshold on the `/15` scale (see `IDLE-DRAFT-PLAN.md`,
`review_score_threshold`) — **but only on a first, from-scratch scoring
pass.** On any re-run of an already-scored file, `paragraphs` (and
therefore the JSON) covers only paragraphs that had no score block at all
going in; it is not a complete, current picture of every paragraph's score.
A caller that wants a complete `/15` picture of a fully-scored file must
keep its own copy of prior JSON output and merge it with each incremental
run's output, or parse the score blocks out of the `-scored.md` file
directly.
## Dryrun behaviour
`--dryrun`/`-n` makes no `claude -p` calls and writes no files. For each
`content`-classified paragraph it reports one of:
- **would score** — eligible, no matching existing block found for it at
parse time; shows the hash it would be scored under and that a
`claude -p` call would be made.
- **skip (unchanged)** — eligible, hash matches an existing score block
found immediately after it; no call would be made. Defined for
completeness — see [Known limitation](#known-limitation-hash-comparison-is-unreachable);
in practice this line is never printed by a real run.
- **skip (too short)** — under `MIN_WORDS` words; never scored.
Paragraphs that already have a score block attached are not `content` at
all by dryrun time, so they produce **no line whatsoever** — they are not
"unchanged", they are invisible to the paragraph walk, exactly as in a real
run.
Followed by a per-file and run-total count, mirroring the real-run summary
line shape.
```
[dryrun] score-paragraphs — no claude calls will be made, no files will be written
Style: /home/user/style/voice.md
Parallel: 4 workers
Scoring: /home/user/book/chapter-01.md → /home/user/book/chapter-01-scored.md
1 would be scored, 0 unchanged (hash match, would skip), 1 too short (skipped)
[would score] ¶7f0e412 (#1, 33 words) — would run: claude -p <scoring prompt>
[skip too-short] paragraph ~6 words
Total: 1 would be scored, 0 unchanged.
```
(Two other paragraphs in `chapter-01.md` already carry score blocks from a
prior run and simply do not appear above at all — see
[Known limitation](#known-limitation-hash-comparison-is-unreachable).)
### Dryrun + `--json`
`--json -` (stdout) is honoured under `--dryrun`: no filesystem writes occur
either way, so the preview JSON is printed. Progress lines above move to
stderr in this case, keeping stdout pure JSON.
`--json FILE` under `--dryrun` is **not** written (writing a file is a change,
which `--dryrun` promises not to make); instead the script logs
`[dryrun] would write JSON results to FILE`.
In both cases the payload shape matches the real-run schema with `"dryrun":
true` and one difference: paragraphs with `action: "would_score"` have
`scores: null`, `total: null`, `flags: []`, `note: ""` (the score is not
known without calling `claude`). Any `action: "reused"` entries (see the
[Known limitation](#known-limitation-hash-comparison-is-unreachable) — not
reachable via this script's own output format, but defined in case a
hand-edited file makes it reachable) still carry real scores, parsed back
out of the existing score block — no `claude` call needed. `summary.mean`/
`summary.min` are computed only over paragraphs with a non-null `total`.
## Edge cases
| Case | Handling |
|---|---|
| `claude` not on PATH, not `--dryrun` | Error to stderr, exit 1, before touching any input file |
| `--output` with multiple input files | Error: `--output can only be used with a single input file`, exit 1 |
| Input file does not exist | Error: `file not found: <path>`, exit 1 |
| `--style` file does not exist | Error: `style guide not found: <path>`, exit 1 |
| No positional files given | argparse usage error, exit 2 |
| Paragraph scoring subprocess errors (timeout, bad JSON, non-zero exit) | All-zero score, `scoring error: <reason>` flag; run continues |
| A file with zero eligible paragraphs | Output is a byte-identical copy (skip chunks only); JSON `paragraphs: []`, `summary.mean/min: null` |
| `--force` with `--json` | Every *`content`-classified* paragraph gets `action: "scored"`; already-scored paragraphs are still invisible regardless — `--force` does not resurrect them (see [Known limitation](#known-limitation-hash-comparison-is-unreachable)) |
| Re-running on an already-scored file | Paragraphs with an attached score block are silently excluded from processing and from `paragraphs_scored`/`paragraphs_skipped`/JSON entirely — the run only touches paragraphs with no block yet. This is the practical, verified behaviour, not a hash-based decision |
| `--json -` (stdout) without `--dryrun` | Progress lines still go to stdout normally *before* the final JSON print — the caller is expected to take the last JSON blob, or redirect and parse only after the run-total line. `--json -` combined with multiple files is fine (one JSON payload with a `files` array). |
## Known limitation: hash comparison is unreachable
Verified during the port (not introduced by it — present identically in the
source `score-paragraphs.py`, confirmed against real GOES production output,
`book/chapter-01-the-amplifier-test-scored.md`):
The chunker (`split_into_chunks`) and the writer (`format_score_block` /
`process_file`) disagree about spacing. The writer inserts a score block with
**zero** blank-line separation from its paragraph. The chunker's score-block
branch calls `flush("skip")`, which flushes *whatever text is currently
accumulating* — not just the score-block line. Because there is no blank
line, the paragraph itself is still accumulating when that branch fires, so
the paragraph is absorbed into the same `"skip"` chunk as its score block.
It is no longer `kind == "content"` on any subsequent parse.
Consequence: `_jobs_for_file`'s hash-comparison branch (compare an existing
block's embedded hash against the paragraph's current hash to decide
skip-vs-stale) requires a `content` chunk immediately followed by a `skip`
chunk whose first line is a score block. That combination cannot occur for
any file this script wrote — inserting a blank line to *avoid* the swallow
doesn't help either, because then the chunk immediately following the
paragraph is the blank line, not the score block (off by one). Checked
exhaustively against a fresh two-paragraph file, a hand-edited
blank-line-separated file, and the real GOES chapter file: **0 paragraphs
out of 15+ already-scored ones were ever re-examined, with or without
`--force`.**
Net effect, stated plainly: this script cannot currently detect that an
already-scored paragraph's text changed, and cannot force a re-score of an
already-scored paragraph. What it *can* do reliably: score paragraphs that
have never been scored before, leaving previously-scored paragraphs
untouched (which happens to look like "hash-skip working correctly" for the
common "add new paragraphs, don't touch old ones" case — the two are
behaviourally indistinguishable until someone edits an already-scored
paragraph and expects a re-score).
This is a scoring-engine bug, not a generalisation concern, so it is out of
scope for this port to fix (`scripts/score-paragraphs` keeps the chunker and
writer exactly as ported, with a code comment pointing here). Anyone relying
on this script for edit-detection — notably `IDLE-DRAFT-PLAN.md`'s `review`
work type, which assumes `--json` gives a complete, current `/15` picture on
every run — should design around this rather than assume the docstring's
aspirational description.
## Examples
```bash
# Single file, default output path
score-paragraphs --style style/voice.md book/chapter-01.md
# Explicit output path
score-paragraphs --style style/voice.md book/chapter-01.md -o /tmp/ch01-scored.md
# Multiple files, plus a JSON results file for a downstream review pass
score-paragraphs --style style/voice.md book/chapter-*.md --json /tmp/scores.json
# Preview only — no claude calls, no files written
score-paragraphs --style style/voice.md book/chapter-01.md --dryrun
# Force re-score everything, parallelism of 8
score-paragraphs --style style/voice.md book/chapter-01.md --force --parallel 8
```