small-scripts/specs/idle-draft.spec.md

idle-draft

Purpose

Cron-invoked dispatcher that consumes idle Anthropic/MiniMax subscription capacity to advance the O'Reilly Consulting writing pipeline (~/dev/claude/writing/oreillyconsulting) unattended: probes usage via agent-subscriptions, computes idle capacity per provider, dispatches ready work items to headless claude -p with bounded parallelism, re-probes on every completion, validates output, and commits results.

Design contract: ~/dev/claude/writing/oreillyconsulting/IDLE-DRAFT-PLAN.md. This spec translates that plan into the repo's implementation contract; where the two disagree, the plan wins and this file should be corrected.

Usage

idle-draft [OPTIONS]
idle-draft mark <dossier/NN-slug> edited|sampled|approved|unblock [OPTIONS]
idle-draft status [OPTIONS]

Options (dispatch mode and status)

Flag	Default	Description
--config FILE	./idle-draft.config.json	Path to the config file
--repo DIR	config file's directory	Writing repo root (contains the dossiers)
--once	off	Dispatch at most one task, then exit (pilot mode)
--parallel N	from config, or 2	Max concurrent tasks (overrides config)
--dryrun, -n	off	Probe + gate + select, print what would be dispatched, execute and mutate nothing
--probe-json FILE	none	Read probe output from FILE instead of running agent-subscriptions (test hook / pilot aid)
--help, -h	—	Show usage and exit 0

mark subcommand

idle-draft mark <dossier/NN-slug> edited|sampled|approved|unblock [--config FILE] [--repo DIR]

The only human mutation path into idle-draft.state.json. Validates the item resolves to an existing <repo>/<dossier>/NN-slug.overview.md, loads and schema-validates the state file, applies exactly one field mutation, writes atomically.

Action	Effect
edited	human_edit_done = true
sampled	research_sampled = true
approved	approved = true
unblock	blocked = null

status subcommand

Human-readable report: per-item derived stage, human-gate flags, next eligible action (or the reason it is not eligible), and per-provider gate values (idle_points, five_hour_pct, eligibility) from a live probe (or --probe-json). Read-only; takes no lock, mutates nothing.

Behaviour

1. Locking

Before doing anything else (dispatch mode only — mark and status do not take the lock), flock (fcntl.flock, LOCK_EX | LOCK_NB) on <repo>/.idle-draft.lock. If already locked by another invocation, log one line and exit 0 — a concurrent cron tick is not a failure.

2. Log rotation

At startup, if <repo>/idle-draft.log exceeds 5 MB, rename it to idle-draft.log.1 (overwriting any existing .1) before appending further.

3. Config load

Parse --config as JSON. Required top-level keys: parallel, providers, work_types, dossiers, review_score_threshold, max_unreviewed_research_per_dossier, max_open_topic_proposals, evidence_dirs — matching data/idle-draft/config.example.json verbatim. Missing required keys or malformed structure → log loudly, exit 2. --parallel on the CLI overrides the config value.

4. State load and validation

Read <repo>/idle-draft.state.json. Missing file is not an error — treat as {"items": {}}. If present:

• Must parse as JSON.
• Top level must be an object with only the key items (unknown top-level keys rejected).
• items must be an object. Each key must be of the form <dossier>/<NN-slug> and must resolve to an existing <repo>/<dossier>/<NN-slug>.overview.md — an item entry for a file that doesn't exist is rejected.
• Each item value must be an object containing only these keys (all optional, defaults shown): human_edit_done (bool, default false), research_sampled (bool, default false), approved (bool, default false), blocked (string or null, default null), attempts (object mapping research/draft/review → non-negative int, default {}). Any other key, or a wrong-typed value, is rejected.

Any validation failure aborts the run loudly (message to stderr and the log) with exit 2 — never "best effort," never silently ignored. This applies to mark and status too (both load and validate the state file before proceeding).

5. Probe

Run agent-subscriptions --output json as a subprocess and parse stdout as JSON, unless --probe-json FILE is given, in which case that file's contents are used verbatim instead (no subprocess call — the test hook / pilot aid). A subprocess failure (non-zero exit, unparseable stdout, timeout) is logged loudly; the run exits 1 if no usable probe data was obtained at all (dispatch loop never starts).

6. Gates (evaluated per provider, every cycle — including on re-probe after each

completion)

For each provider in config["providers"], using that provider's seven_day and five_hour window records from the probe report:

idle_points = threshold_pct × elapsed_pct − usage_pct        (on the seven_day window)

A provider is eligible for a new dispatch iff all of:

• the probe marks it available: true
• seven_day.elapsed_pct is non-null (null → "cannot pace", fail closed)
• seven_day.utilization_pct is non-null
• five_hour.utilization_pct is non-null
• idle_points > min_idle
• five_hour.utilization_pct < five_hour_ceiling
• the credential-parity check (§7) passes for that provider

All three gate values (idle_points, five_hour.utilization_pct, eligibility) are logged for every provider on every cycle, whether or not the provider ends up used.

7. Credential parity

Before a provider can be selected, idle-draft resolves the credential its profile would export at launch and compares it (by content, via SHA-256) against the credential agent-subscriptions used to probe that same provider:

• Profile side: if <profile>/provider.env defines ANTHROPIC_API_KEY_FILE, read that file's content. If the profile has no provider.env (the default Anthropic profile), use the same token file agent-subscriptions reads for Anthropic (~/dev/claude/secrets/anthropic/api_key).
• Probe side: for anthropic, the same ~/dev/claude/secrets/anthropic/api_key file content. For minimax, the SOPS-decrypted ANTHROPIC_AUTH_TOKEN value from agent-subscriptions' MiniMax path (same SOPS file, same key file, same dotenv key — these constants are replicated locally and must be kept in sync with scripts/agent-subscriptions if that script's paths change).

A mismatch (or either side unreadable) makes the provider ineligible for this cycle and logs a loud line: credential mismatch: <provider> profile≠probe (<reason>). The gate must meter the account that actually spends, not the account the profile file merely names.

8. Work-item discovery and stage derivation

For each dossier in config["dossiers"], glob <repo>/<dossier>/*.overview.md matching ^(\d+)-(.+)\.overview\.md$. For each NN-slug, derive the next eligible work type (or "not eligible, because...") purely from which sibling files exist plus the item's state entry:

Condition	Result
state.blocked is set	not eligible: blocked
state.approved is true	not eligible: approved (terminal)
NN-slug.agent.md missing	not eligible: no commissioning brief
NN-slug.research.md missing	next: research
NN-slug.draft.md missing	next: draft
NN-slug.review.md exists	not eligible: awaiting human revise/approve
state.human_edit_done is not true	not eligible: waiting on human edit
(all of the above pass)	next: review

In-flight items (a task currently dispatched for that item in this run) are excluded from consideration for further dispatch until the in-flight task completes.

Cold-start throttle: per dossier, count NN-slug.research.md files that exist and whose state entry does not have research_sampled: true. Once that count reaches max_unreviewed_research_per_dossier, no further research candidates are offered for that dossier this cycle (draft/review candidates in that dossier are unaffected).

topic_ideas: dossier-level (not tied to an NN-slug), considered only when the combined candidate list above (across all dossiers) is empty. For each dossier, eligible iff the count of proposals already recorded in <dossier>/TOPIC-PROPOSALS.md (one ## heading per proposal; file absent counts as 0) is below max_open_topic_proposals.

9. Prioritisation

Ready (item, work_type) candidates (plus, only when the list would otherwise be empty, (dossier, topic_ideas) candidates) are sorted by:

1. Stage rank, descending: review (3) > draft (2) > research (1) > topic_ideas (0).
2. Numeric filename prefix, ascending (topic_ideas sorts as 0, always last within its own rank tier — moot since it only appears when nothing else is ready).
3. Dossier config order (config["dossiers"] index), ascending, as the tiebreak.

For each candidate in this order, the provider is the first entry in config["work_types"][work_type]["providers"] that is currently eligible (§6, §7). A candidate with no eligible provider is skipped (not dispatched this cycle, tried again next cycle); the walk continues to the next candidate.

10. Dispatch loop (worker pool, event-driven)

Up to parallel tasks run concurrently (concurrent.futures.ThreadPoolExecutor). Each completion is handled serially in the main thread (state updates and git commits never race). On every completion — and before the very first dispatch — the gates (§6) are recomputed from a fresh probe (§5); a task is submitted to fill a free worker slot only while an eligible (item, work_type, provider) triple remains. The loop exits (dispatch mode, non-dryrun) when no eligible candidate remains. --once dispatches at most one task total, then exits without waiting for further slots.

11. Task execution

For the selected (item_or_dossier, work_type, provider):

1. Resolve the prompt template: data/idle-draft/prompts/{research,draft,review-suggest,topic-ideas}.md (review work type uses review-suggest.md).
2. Render the template (string.Template, $placeholder substitution) with the resolved paths for that item (overview, agent, research, draft as applicable), dossier name, slug, style directory, source register path, and a temp output path.
3. Resolve the profile directory from config["providers"][provider]["profile"] (~ expanded). Build the child environment: CLAUDE_CONFIG_DIR=<profile>, plus — if <profile>/provider.env exists — ANTHROPIC_BASE_URL (if set), ANTHROPIC_API_KEY (read from ANTHROPIC_API_KEY_FILE, if set), and any other KEY=value line verbatim. A profile with no provider.env (plain Anthropic) gets only CLAUDE_CONFIG_DIR. This logic is a local re-implementation of scripts/claude-profile lines ~320–440 — claude-profile itself is never invoked (it has interactive pickers and terminal theming unsuitable for headless cron use).
4. Build the argv: claude -p --max-turns <N> (N = 25 unless overridden), plus --model <id> if provider.env sets MODEL_ID, plus --add-dir <dir> for each entry in config["evidence_dirs"] only for research and topic_ideas work types (the only ones that cite external evidence), plus --allowedTools "<comma-joined list>" if config["work_types"][work_type]["allowed_tools"] is present and non-empty (see Config schema below). Headless claude -p runs cannot answer permission prompts, so any tool beyond the CLI's defaults must be granted this way or the child run stalls/fails; when the key is absent or an empty list, the flag is omitted entirely (unchanged pre-existing behaviour).
5. Run the subprocess: cwd=<repo>, env=<built env>, prompt piped via stdin (not as an argv element — avoids ARG_MAX on large rendered prompts, same lesson claude-profile already applies to its system-prompt injection), timeout = per-task timeout (1800s unless overridden).
6. Classify the result:
   • Timeout → transient failure. Do not increment attempts.
   • Non-zero exit whose stderr matches a retryable signature (429, 5xx, rate limit, overloaded, timeout, temporarily unavailable, connection-reset markers) → transient failure. Do not increment attempts.
   • Non-zero exit, no retryable signature, or exit 0 but validation fails below → content failure. Increment state.items[item].attempts[work_type]. At max_attempts (2) content failures for that work type, set state.items[item].blocked with a reason (the item is then excluded from all further dispatch until a human runs mark unblock).
   • Exit 0 and validation passes → success.
7. Validation (on exit 0, before promotion): the temp output file must be non-empty and its first non-blank line must be a top-level Markdown heading (# ...). For research work type specifically: every absolute path matching /home/\S+ cited in the file must exist on disk (Path.exists()); any dead path fails validation. Citations may carry a trailing line-range suffix in path:120-145 or path:120 form (per the research prompt template's required citation format); this suffix is stripped — via re.sub(r":\d+(-\d+)?$", ...), only when the remainder still looks like a path — before the existence check, so a valid ranged citation is not flagged as a dead path.
8. On validation pass: os.replace() the temp file to the canonical path (<repo>/<dossier>/<NN-slug>.<work_type>.md, or append to <dossier>/TOPIC-PROPOSALS.md for topic_ideas) — atomic, never a partial file visible under the canonical name.
9. On content-failure: os.replace() the temp file to <repo>/<dossier>/<NN-slug>.<work_type>.md.rejected (kept for human inspection, never promoted, never committed to the canonical name; topic_ideas content failures are simply discarded — nothing is appended, nothing is blocked, since topic_ideas has no per-item state entry to carry an attempt counter).
10. On transient failure: temp file is discarded; no state change; no commit; the candidate may be retried on a later cycle.
11. On success: write idle-draft.state.json atomically (temp file + os.replace() in the same directory), then:

    git -C <repo> add -- <produced-file> idle-draft.state.json
    git -C <repo> commit -m "<message>" -- <produced-file> idle-draft.state.json
    

    Never git add -A, never git commit -a. Commit message: idle-draft: <work_type> <item> via <provider> (7d <before>%→<after>%), where before/after are that provider's seven_day.utilization_pct immediately before dispatch and immediately after re-probe on completion. No push.

12. Logging

One line per event (gate decision, dispatch, completion, failure, commit) appended to <repo>/idle-draft.log with an ISO-8601 timestamp, mirrored to stderr.

Dryrun behaviour

--dryrun runs the full probe (or reads --probe-json), computes all gates, builds the ready queue, and selects up to parallel (or 1, under --once) (item, work_type, provider) triples exactly as the real dispatch loop's first wave would — without simulating gate depletion across a re-probe (a real run only knows that after actually dispatching; the preview shows the initial wave only, noted as such). For each selected triple it prints, and executes nothing:

• item or dossier identifier, work type, chosen provider
• resolved prompt template path
• the exact claude argv list that would run
• the child env deltas (CLAUDE_CONFIG_DIR, ANTHROPIC_BASE_URL if set, whether ANTHROPIC_API_KEY would be exported)
• the temp output path and the canonical destination path

No subprocess is run, no file is written, no state is mutated, no lock is required to be free for the preview to work (the lock is still attempted and its outcome reported, but a held lock does not block the dryrun preview from computing and printing — real dispatch mode is what respects the lock as a hard gate). Exit 0.

State schema

idle-draft.state.json, repo root — see §4. Humans never hand-edit this file; the mark subcommand is the only mutation path. Derived state (anything the filesystem already says) is never written here.

Config schema

idle-draft.config.json — see data/idle-draft/config.example.json for the exact structure. All top-level keys listed in §3 are required.

Each entry in work_types may carry an optional allowed_tools key: a list of Claude Code tool names (e.g. ["WebSearch", "WebFetch"]) granted to that work type's headless child via --allowedTools (§11 step 4). Present-but-empty and absent are both treated as "no extra tools" — the flag is omitted. If present, it must be a JSON array of strings; any other shape (a string, a number, a list containing a non-string) is a config validation failure — exit 2 at config load, same as a missing required key. config.example.json sets allowed_tools: ["WebSearch", "WebFetch"] on research and topic_ideas (the research prompt template requires at least one WebSearch call before writing its Subjective stories section; the topic-ideas template requires one as a lighter-touch check); draft and review carry no allowed_tools key.

Failure classes

Class	Examples	attempts effect	Item outcome
Transient	timeout, HTTP 429/5xx, network reset	not incremented	retried next eligible cycle
Content	empty output, missing heading, dead citation path, non-retryable non-zero exit	incremented	.rejected kept; blocked at max_attempts (2)

Exit codes

Code	Meaning
0	Nothing to do, or all dispatched tasks completed without a fatal problem (individual task failures are logged, not fatal)
1	Transient/probe problem prevented the run from proceeding (no usable probe data, lock held is exit 0 not 1 — see §1)
2	Config or state validation failure

Edge cases

Scenario	Handling
Lock already held	Log one line, exit 0 (not an error — another cron tick is running)
idle-draft.state.json missing	Treated as {"items": {}}, not an error
idle-draft.state.json present but invalid	Exit 2, loud message, run never starts
Item's .agent.md missing	Item excluded from all dispatch (commissioning briefs exist today for every current item; this guards future additions)
seven_day.elapsed_pct null	Provider ineligible this cycle, never assumed idle
Provider available: false	Provider ineligible this cycle
Credential mismatch	Provider ineligible this cycle, loud log line
research output cites a path that doesn't exist	Content failure, quarantined as .rejected, attempts incremented
max_attempts reached	Item blocked, excluded until mark unblock
topic_ideas content failure	Discarded silently (logged, not blocked — no per-item state key exists for a dossier-level work type)
Two work types both ready in the same dossier	Higher stage rank wins (review > draft > research); topic_ideas never competes (last-resort only)
--probe-json combined with --dryrun	Composes normally — no subprocess call either way
--once with no eligible work	Exits 0 immediately, no task dispatched
Config missing a required key	Exit 2
work_types[*].allowed_tools present but not a list of strings	Exit 2
agent-subscriptions subprocess fails entirely	Exit 1, loud log line, no dispatch attempted

Examples

idle-draft --dryrun                                    # preview one dispatch wave
idle-draft --once                                       # pilot: one real task, then exit
idle-draft --probe-json /tmp/fake-probe.json --dryrun    # preview against fixture data
idle-draft --config ~/dev/claude/writing/oreillyconsulting/idle-draft.config.json
idle-draft mark ai/03-shadow-agents-are-the-new-shadow-it edited
idle-draft mark ai/03-shadow-agents-are-the-new-shadow-it sampled
idle-draft status