idle-draft: idle-subscription dispatcher for the writing pipeline

Elapsed-paced gates, bounded-parallel event-driven dispatch, filesystem- derived stages with thin human-gate state, citation validation with quarantine, credential parity check, prompt templates as data. Per IDLE-DRAFT-PLAN.md in writing/oreillyconsulting. Claude-Session: https://claude.ai/code/session_01YQDoWNM7XPPii28khFWoMc
2026-08-02 21:18:02 +12:00
parent f78d292f05
commit 6c4bf449aa
8 changed files with 2398 additions and 0 deletions
--- a/specs/idle-draft.spec.md
+++ b/specs/idle-draft.spec.md
@@ -0,0 +1,355 @@
+# idle-draft
+
+## Purpose
+
+Cron-invoked dispatcher that consumes idle Anthropic/MiniMax subscription capacity to
+advance the O'Reilly Consulting writing pipeline (`~/dev/claude/writing/oreillyconsulting`)
+unattended: probes usage via `agent-subscriptions`, computes idle capacity per provider,
+dispatches ready work items to headless `claude -p` with bounded parallelism, re-probes
+on every completion, validates output, and commits results.
+
+Design contract: `~/dev/claude/writing/oreillyconsulting/IDLE-DRAFT-PLAN.md`. This spec
+translates that plan into the repo's implementation contract; where the two disagree,
+the plan wins and this file should be corrected.
+
+## Usage
+
+```
+idle-draft [OPTIONS]
+idle-draft mark <dossier/NN-slug> edited|sampled|approved|unblock [OPTIONS]
+idle-draft status [OPTIONS]
+```
+
+### Options (dispatch mode and `status`)
+
+| Flag | Default | Description |
+|---|---|---|
+| `--config FILE` | `./idle-draft.config.json` | Path to the config file |
+| `--repo DIR` | config file's directory | Writing repo root (contains the dossiers) |
+| `--once` | off | Dispatch at most one task, then exit (pilot mode) |
+| `--parallel N` | from config, or `2` | Max concurrent tasks (overrides config) |
+| `--dryrun`, `-n` | off | Probe + gate + select, print what would be dispatched, execute and mutate nothing |
+| `--probe-json FILE` | none | Read probe output from FILE instead of running `agent-subscriptions` (test hook / pilot aid) |
+| `--help`, `-h` | — | Show usage and exit 0 |
+
+### `mark` subcommand
+
+```
+idle-draft mark <dossier/NN-slug> edited|sampled|approved|unblock [--config FILE] [--repo DIR]
+```
+
+The only human mutation path into `idle-draft.state.json`. Validates the item resolves
+to an existing `<repo>/<dossier>/NN-slug.overview.md`, loads and schema-validates the
+state file, applies exactly one field mutation, writes atomically.
+
+| Action | Effect |
+|---|---|
+| `edited` | `human_edit_done = true` |
+| `sampled` | `research_sampled = true` |
+| `approved` | `approved = true` |
+| `unblock` | `blocked = null` |
+
+### `status` subcommand
+
+Human-readable report: per-item derived stage, human-gate flags, next eligible action
+(or the reason it is not eligible), and per-provider gate values (`idle_points`,
+`five_hour_pct`, eligibility) from a live probe (or `--probe-json`). Read-only; takes no
+lock, mutates nothing.
+
+## Behaviour
+
+### 1. Locking
+
+Before doing anything else (dispatch mode only — `mark` and `status` do not take the
+lock), `flock` (`fcntl.flock`, `LOCK_EX | LOCK_NB`) on `<repo>/.idle-draft.lock`. If
+already locked by another invocation, log one line and exit **0** — a concurrent cron
+tick is not a failure.
+
+### 2. Log rotation
+
+At startup, if `<repo>/idle-draft.log` exceeds 5 MB, rename it to `idle-draft.log.1`
+(overwriting any existing `.1`) before appending further.
+
+### 3. Config load
+
+Parse `--config` as JSON. Required top-level keys: `parallel`, `providers`, `work_types`,
+`dossiers`, `review_score_threshold`, `max_unreviewed_research_per_dossier`,
+`max_open_topic_proposals`, `evidence_dirs` — matching `data/idle-draft/config.example.json`
+verbatim. Missing required keys or malformed structure → log loudly, exit **2**.
+`--parallel` on the CLI overrides the config value.
+
+### 4. State load and validation
+
+Read `<repo>/idle-draft.state.json`. Missing file is not an error — treat as
+`{"items": {}}`. If present:
+
+- Must parse as JSON.
+- Top level must be an object with **only** the key `items` (unknown top-level keys
+  rejected).
+- `items` must be an object. Each key must be of the form `<dossier>/<NN-slug>` and
+  must resolve to an existing `<repo>/<dossier>/<NN-slug>.overview.md` — an item entry
+  for a file that doesn't exist is rejected.
+- Each item value must be an object containing **only** these keys (all optional,
+  defaults shown): `human_edit_done` (bool, default `false`), `research_sampled` (bool,
+  default `false`), `approved` (bool, default `false`), `blocked` (string or `null`,
+  default `null`), `attempts` (object mapping `research`/`draft`/`review` → non-negative
+  int, default `{}`). Any other key, or a wrong-typed value, is rejected.
+
+Any validation failure aborts the run loudly (message to stderr and the log) with
+**exit 2** — never "best effort," never silently ignored. This applies to `mark` and
+`status` too (both load and validate the state file before proceeding).
+
+### 5. Probe
+
+Run `agent-subscriptions --output json` as a subprocess and parse stdout as JSON,
+**unless** `--probe-json FILE` is given, in which case that file's contents are used
+verbatim instead (no subprocess call — the test hook / pilot aid). A subprocess
+failure (non-zero exit, unparseable stdout, timeout) is logged loudly; the run exits
+**1** if no usable probe data was obtained at all (dispatch loop never starts).
+
+### 6. Gates (evaluated per provider, every cycle — including on re-probe after each
+completion)
+
+For each provider in `config["providers"]`, using that provider's `seven_day` and
+`five_hour` window records from the probe report:
+
+```
+idle_points = threshold_pct × elapsed_pct − usage_pct        (on the seven_day window)
+```
+
+A provider is **eligible** for a new dispatch iff **all** of:
+
+- the probe marks it `available: true`
+- `seven_day.elapsed_pct` is non-null (null → "cannot pace", fail closed)
+- `seven_day.utilization_pct` is non-null
+- `five_hour.utilization_pct` is non-null
+- `idle_points > min_idle`
+- `five_hour.utilization_pct < five_hour_ceiling`
+- the credential-parity check (§7) passes for that provider
+
+All three gate values (`idle_points`, `five_hour.utilization_pct`, eligibility) are
+logged for every provider on every cycle, whether or not the provider ends up used.
+
+### 7. Credential parity
+
+Before a provider can be selected, idle-draft resolves the credential its profile would
+export at launch and compares it (by content, via SHA-256) against the credential
+`agent-subscriptions` used to probe that same provider:
+
+- **Profile side:** if `<profile>/provider.env` defines `ANTHROPIC_API_KEY_FILE`, read
+  that file's content. If the profile has no `provider.env` (the default Anthropic
+  profile), use the same token file `agent-subscriptions` reads for Anthropic
+  (`~/dev/claude/secrets/anthropic/api_key`).
+- **Probe side:** for `anthropic`, the same `~/dev/claude/secrets/anthropic/api_key`
+  file content. For `minimax`, the SOPS-decrypted `ANTHROPIC_AUTH_TOKEN` value from
+  `agent-subscriptions`' MiniMax path (same SOPS file, same key file, same dotenv key
+  — these constants are replicated locally and must be kept in sync with
+  `scripts/agent-subscriptions` if that script's paths change).
+
+A mismatch (or either side unreadable) makes the provider ineligible for this cycle and
+logs a loud line: `credential mismatch: <provider> profile≠probe (<reason>)`. The gate
+must meter the account that actually spends, not the account the profile file merely
+names.
+
+### 8. Work-item discovery and stage derivation
+
+For each dossier in `config["dossiers"]`, glob `<repo>/<dossier>/*.overview.md` matching
+`^(\d+)-(.+)\.overview\.md$`. For each `NN-slug`, derive the next eligible work type
+(or "not eligible, because...") purely from which sibling files exist plus the item's
+state entry:
+
+| Condition | Result |
+|---|---|
+| `state.blocked` is set | not eligible: blocked |
+| `state.approved` is true | not eligible: approved (terminal) |
+| `NN-slug.agent.md` missing | not eligible: no commissioning brief |
+| `NN-slug.research.md` missing | next: `research` |
+| `NN-slug.draft.md` missing | next: `draft` |
+| `NN-slug.review.md` exists | not eligible: awaiting human revise/approve |
+| `state.human_edit_done` is not true | not eligible: waiting on human edit |
+| (all of the above pass) | next: `review` |
+
+In-flight items (a task currently dispatched for that item in this run) are excluded
+from consideration for further dispatch until the in-flight task completes.
+
+**Cold-start throttle:** per dossier, count `NN-slug.research.md` files that exist and
+whose state entry does **not** have `research_sampled: true`. Once that count reaches
+`max_unreviewed_research_per_dossier`, no further `research` candidates are offered for
+that dossier this cycle (draft/review candidates in that dossier are unaffected).
+
+**`topic_ideas`:** dossier-level (not tied to an `NN-slug`), considered only when the
+combined candidate list above (across all dossiers) is empty. For each dossier, eligible
+iff the count of proposals already recorded in `<dossier>/TOPIC-PROPOSALS.md` (one `##`
+heading per proposal; file absent counts as 0) is below `max_open_topic_proposals`.
+
+### 9. Prioritisation
+
+Ready `(item, work_type)` candidates (plus, only when the list would otherwise be
+empty, `(dossier, topic_ideas)` candidates) are sorted by:
+
+1. Stage rank, descending: `review` (3) > `draft` (2) > `research` (1) > `topic_ideas` (0).
+2. Numeric filename prefix, ascending (`topic_ideas` sorts as `0`, always last within
+   its own rank tier — moot since it only appears when nothing else is ready).
+3. Dossier config order (`config["dossiers"]` index), ascending, as the tiebreak.
+
+For each candidate in this order, the provider is the **first** entry in
+`config["work_types"][work_type]["providers"]` that is currently eligible (§6, §7). A
+candidate with no eligible provider is skipped (not dispatched this cycle, tried again
+next cycle); the walk continues to the next candidate.
+
+### 10. Dispatch loop (worker pool, event-driven)
+
+Up to `parallel` tasks run concurrently (`concurrent.futures.ThreadPoolExecutor`).
+Each completion is handled **serially** in the main thread (state updates and git
+commits never race). On every completion — and before the very first dispatch — the
+gates (§6) are recomputed from a fresh probe (§5); a task is submitted to fill a free
+worker slot only while an eligible `(item, work_type, provider)` triple remains. The
+loop exits (dispatch mode, non-dryrun) when no eligible candidate remains. `--once`
+dispatches at most one task total, then exits without waiting for further slots.
+
+### 11. Task execution
+
+For the selected `(item_or_dossier, work_type, provider)`:
+
+1. Resolve the prompt template: `data/idle-draft/prompts/{research,draft,review-suggest,topic-ideas}.md`
+   (`review` work type uses `review-suggest.md`).
+2. Render the template (`string.Template`, `$placeholder` substitution) with the
+   resolved paths for that item (overview, agent, research, draft as applicable),
+   dossier name, slug, style directory, source register path, and a temp output path.
+3. Resolve the profile directory from `config["providers"][provider]["profile"]`
+   (`~` expanded). Build the child environment: `CLAUDE_CONFIG_DIR=<profile>`, plus —
+   if `<profile>/provider.env` exists — `ANTHROPIC_BASE_URL` (if set),
+   `ANTHROPIC_API_KEY` (read from `ANTHROPIC_API_KEY_FILE`, if set), and any other
+   `KEY=value` line verbatim. A profile with no `provider.env` (plain Anthropic) gets
+   only `CLAUDE_CONFIG_DIR`. This logic is a local re-implementation of
+   `scripts/claude-profile` lines ~320–440 — **`claude-profile` itself is never
+   invoked** (it has interactive pickers and terminal theming unsuitable for headless
+   cron use).
+4. Build the argv: `claude -p --max-turns <N>` (`N` = 25 unless overridden), plus
+   `--model <id>` if `provider.env` sets `MODEL_ID`, plus `--add-dir <dir>` for each
+   entry in `config["evidence_dirs"]` **only** for `research` and `topic_ideas` work
+   types (the only ones that cite external evidence).
+5. Run the subprocess: `cwd=<repo>`, `env=<built env>`, prompt piped via **stdin**
+   (not as an argv element — avoids `ARG_MAX` on large rendered prompts, same lesson
+   `claude-profile` already applies to its system-prompt injection), timeout = per-task
+   timeout (1800s unless overridden).
+6. Classify the result:
+   - **Timeout** → transient failure. Do not increment `attempts`.
+   - **Non-zero exit** whose stderr matches a retryable signature (`429`, `5xx`,
+     `rate limit`, `overloaded`, `timeout`, `temporarily unavailable`, connection-reset
+     markers) → transient failure. Do not increment `attempts`.
+   - **Non-zero exit**, no retryable signature, or **exit 0 but validation fails below**
+     → content failure. Increment `state.items[item].attempts[work_type]`. At
+     `max_attempts` (2) content failures for that work type, set
+     `state.items[item].blocked` with a reason (the item is then excluded from all
+     further dispatch until a human runs `mark unblock`).
+   - **Exit 0 and validation passes** → success.
+7. Validation (on exit 0, before promotion): the temp output file must be non-empty and
+   its first non-blank line must be a top-level Markdown heading (`# ...`). For
+   `research` work type specifically: every absolute path matching `/home/\S+` cited in
+   the file must exist on disk (`Path.exists()`); any dead path fails validation.
+8. On validation pass: `os.replace()` the temp file to the canonical path
+   (`<repo>/<dossier>/<NN-slug>.<work_type>.md`, or append to
+   `<dossier>/TOPIC-PROPOSALS.md` for `topic_ideas`) — atomic, never a partial file
+   visible under the canonical name.
+9. On content-failure: `os.replace()` the temp file to
+   `<repo>/<dossier>/<NN-slug>.<work_type>.md.rejected` (kept for human inspection,
+   never promoted, never committed to the canonical name; `topic_ideas` content
+   failures are simply discarded — nothing is appended, nothing is blocked, since
+   `topic_ideas` has no per-item state entry to carry an attempt counter).
+10. On transient failure: temp file is discarded; no state change; no commit; the
+    candidate may be retried on a later cycle.
+11. On success: write `idle-draft.state.json` atomically (temp file + `os.replace()` in
+    the same directory), then:
+    ```
+    git -C <repo> add -- <produced-file> idle-draft.state.json
+    git -C <repo> commit -m "<message>" -- <produced-file> idle-draft.state.json
+    ```
+    Never `git add -A`, never `git commit -a`. Commit message:
+    `idle-draft: <work_type> <item> via <provider> (7d <before>%→<after>%)`, where
+    before/after are that provider's `seven_day.utilization_pct` immediately before
+    dispatch and immediately after re-probe on completion. No push.
+
+### 12. Logging
+
+One line per event (gate decision, dispatch, completion, failure, commit) appended to
+`<repo>/idle-draft.log` with an ISO-8601 timestamp, mirrored to stderr.
+
+## Dryrun behaviour
+
+`--dryrun` runs the full probe (or reads `--probe-json`), computes all gates, builds
+the ready queue, and selects up to `parallel` (or `1`, under `--once`) `(item,
+work_type, provider)` triples exactly as the real dispatch loop's first wave would —
+**without** simulating gate depletion across a re-probe (a real run only knows that
+after actually dispatching; the preview shows the initial wave only, noted as such).
+For each selected triple it prints, and executes nothing:
+
+- item or dossier identifier, work type, chosen provider
+- resolved prompt template path
+- the exact `claude` argv list that would run
+- the child env deltas (`CLAUDE_CONFIG_DIR`, `ANTHROPIC_BASE_URL` if set, whether
+  `ANTHROPIC_API_KEY` would be exported)
+- the temp output path and the canonical destination path
+
+No subprocess is run, no file is written, no state is mutated, no lock is required to
+be free for the preview to work (the lock is still attempted and its outcome reported,
+but a held lock does not block the dryrun preview from computing and printing — real
+dispatch mode is what respects the lock as a hard gate). Exit 0.
+
+## State schema
+
+`idle-draft.state.json`, repo root — see §4. Humans never hand-edit this file; the
+`mark` subcommand is the only mutation path. Derived state (anything the filesystem
+already says) is never written here.
+
+## Config schema
+
+`idle-draft.config.json` — see `data/idle-draft/config.example.json` for the exact
+structure. All top-level keys listed in §3 are required.
+
+## Failure classes
+
+| Class | Examples | `attempts` effect | Item outcome |
+|---|---|---|---|
+| Transient | timeout, HTTP 429/5xx, network reset | not incremented | retried next eligible cycle |
+| Content | empty output, missing heading, dead citation path, non-retryable non-zero exit | incremented | `.rejected` kept; `blocked` at `max_attempts` (2) |
+
+## Exit codes
+
+| Code | Meaning |
+|---|---|
+| 0 | Nothing to do, or all dispatched tasks completed without a fatal problem (individual task failures are logged, not fatal) |
+| 1 | Transient/probe problem prevented the run from proceeding (no usable probe data, lock held is exit 0 not 1 — see §1) |
+| 2 | Config or state validation failure |
+
+## Edge cases
+
+| Scenario | Handling |
+|---|---|
+| Lock already held | Log one line, exit 0 (not an error — another cron tick is running) |
+| `idle-draft.state.json` missing | Treated as `{"items": {}}`, not an error |
+| `idle-draft.state.json` present but invalid | Exit 2, loud message, run never starts |
+| Item's `.agent.md` missing | Item excluded from all dispatch (commissioning briefs exist today for every current item; this guards future additions) |
+| `seven_day.elapsed_pct` null | Provider ineligible this cycle, never assumed idle |
+| Provider `available: false` | Provider ineligible this cycle |
+| Credential mismatch | Provider ineligible this cycle, loud log line |
+| `research` output cites a path that doesn't exist | Content failure, quarantined as `.rejected`, attempts incremented |
+| `max_attempts` reached | Item `blocked`, excluded until `mark unblock` |
+| `topic_ideas` content failure | Discarded silently (logged, not blocked — no per-item state key exists for a dossier-level work type) |
+| Two work types both ready in the same dossier | Higher stage rank wins (review > draft > research); `topic_ideas` never competes (last-resort only) |
+| `--probe-json` combined with `--dryrun` | Composes normally — no subprocess call either way |
+| `--once` with no eligible work | Exits 0 immediately, no task dispatched |
+| Config missing a required key | Exit 2 |
+| `agent-subscriptions` subprocess fails entirely | Exit 1, loud log line, no dispatch attempted |
+
+## Examples
+
+```sh
+idle-draft --dryrun                                    # preview one dispatch wave
+idle-draft --once                                       # pilot: one real task, then exit
+idle-draft --probe-json /tmp/fake-probe.json --dryrun    # preview against fixture data
+idle-draft --config ~/dev/claude/writing/oreillyconsulting/idle-draft.config.json
+idle-draft mark ai/03-shadow-agents-are-the-new-shadow-it edited
+idle-draft mark ai/03-shadow-agents-are-the-new-shadow-it sampled
+idle-draft status
+```