75add58246
Research and topic_ideas runs get --allowedTools WebSearch,WebFetch; research template mandates >=1 web search for related stories with full-URL + access-date citations under the dossier's status labels. Claude-Session: https://claude.ai/code/session_01YQDoWNM7XPPii28khFWoMc
377 lines
20 KiB
Markdown
377 lines
20 KiB
Markdown
# idle-draft
|
||
|
||
## Purpose
|
||
|
||
Cron-invoked dispatcher that consumes idle Anthropic/MiniMax subscription capacity to
|
||
advance the O'Reilly Consulting writing pipeline (`~/dev/claude/writing/oreillyconsulting`)
|
||
unattended: probes usage via `agent-subscriptions`, computes idle capacity per provider,
|
||
dispatches ready work items to headless `claude -p` with bounded parallelism, re-probes
|
||
on every completion, validates output, and commits results.
|
||
|
||
Design contract: `~/dev/claude/writing/oreillyconsulting/IDLE-DRAFT-PLAN.md`. This spec
|
||
translates that plan into the repo's implementation contract; where the two disagree,
|
||
the plan wins and this file should be corrected.
|
||
|
||
## Usage
|
||
|
||
```
|
||
idle-draft [OPTIONS]
|
||
idle-draft mark <dossier/NN-slug> edited|sampled|approved|unblock [OPTIONS]
|
||
idle-draft status [OPTIONS]
|
||
```
|
||
|
||
### Options (dispatch mode and `status`)
|
||
|
||
| Flag | Default | Description |
|
||
|---|---|---|
|
||
| `--config FILE` | `./idle-draft.config.json` | Path to the config file |
|
||
| `--repo DIR` | config file's directory | Writing repo root (contains the dossiers) |
|
||
| `--once` | off | Dispatch at most one task, then exit (pilot mode) |
|
||
| `--parallel N` | from config, or `2` | Max concurrent tasks (overrides config) |
|
||
| `--dryrun`, `-n` | off | Probe + gate + select, print what would be dispatched, execute and mutate nothing |
|
||
| `--probe-json FILE` | none | Read probe output from FILE instead of running `agent-subscriptions` (test hook / pilot aid) |
|
||
| `--help`, `-h` | — | Show usage and exit 0 |
|
||
|
||
### `mark` subcommand
|
||
|
||
```
|
||
idle-draft mark <dossier/NN-slug> edited|sampled|approved|unblock [--config FILE] [--repo DIR]
|
||
```
|
||
|
||
The only human mutation path into `idle-draft.state.json`. Validates the item resolves
|
||
to an existing `<repo>/<dossier>/NN-slug.overview.md`, loads and schema-validates the
|
||
state file, applies exactly one field mutation, writes atomically.
|
||
|
||
| Action | Effect |
|
||
|---|---|
|
||
| `edited` | `human_edit_done = true` |
|
||
| `sampled` | `research_sampled = true` |
|
||
| `approved` | `approved = true` |
|
||
| `unblock` | `blocked = null` |
|
||
|
||
### `status` subcommand
|
||
|
||
Human-readable report: per-item derived stage, human-gate flags, next eligible action
|
||
(or the reason it is not eligible), and per-provider gate values (`idle_points`,
|
||
`five_hour_pct`, eligibility) from a live probe (or `--probe-json`). Read-only; takes no
|
||
lock, mutates nothing.
|
||
|
||
## Behaviour
|
||
|
||
### 1. Locking
|
||
|
||
Before doing anything else (dispatch mode only — `mark` and `status` do not take the
|
||
lock), `flock` (`fcntl.flock`, `LOCK_EX | LOCK_NB`) on `<repo>/.idle-draft.lock`. If
|
||
already locked by another invocation, log one line and exit **0** — a concurrent cron
|
||
tick is not a failure.
|
||
|
||
### 2. Log rotation
|
||
|
||
At startup, if `<repo>/idle-draft.log` exceeds 5 MB, rename it to `idle-draft.log.1`
|
||
(overwriting any existing `.1`) before appending further.
|
||
|
||
### 3. Config load
|
||
|
||
Parse `--config` as JSON. Required top-level keys: `parallel`, `providers`, `work_types`,
|
||
`dossiers`, `review_score_threshold`, `max_unreviewed_research_per_dossier`,
|
||
`max_open_topic_proposals`, `evidence_dirs` — matching `data/idle-draft/config.example.json`
|
||
verbatim. Missing required keys or malformed structure → log loudly, exit **2**.
|
||
`--parallel` on the CLI overrides the config value.
|
||
|
||
### 4. State load and validation
|
||
|
||
Read `<repo>/idle-draft.state.json`. Missing file is not an error — treat as
|
||
`{"items": {}}`. If present:
|
||
|
||
- Must parse as JSON.
|
||
- Top level must be an object with **only** the key `items` (unknown top-level keys
|
||
rejected).
|
||
- `items` must be an object. Each key must be of the form `<dossier>/<NN-slug>` and
|
||
must resolve to an existing `<repo>/<dossier>/<NN-slug>.overview.md` — an item entry
|
||
for a file that doesn't exist is rejected.
|
||
- Each item value must be an object containing **only** these keys (all optional,
|
||
defaults shown): `human_edit_done` (bool, default `false`), `research_sampled` (bool,
|
||
default `false`), `approved` (bool, default `false`), `blocked` (string or `null`,
|
||
default `null`), `attempts` (object mapping `research`/`draft`/`review` → non-negative
|
||
int, default `{}`). Any other key, or a wrong-typed value, is rejected.
|
||
|
||
Any validation failure aborts the run loudly (message to stderr and the log) with
|
||
**exit 2** — never "best effort," never silently ignored. This applies to `mark` and
|
||
`status` too (both load and validate the state file before proceeding).
|
||
|
||
### 5. Probe
|
||
|
||
Run `agent-subscriptions --output json` as a subprocess and parse stdout as JSON,
|
||
**unless** `--probe-json FILE` is given, in which case that file's contents are used
|
||
verbatim instead (no subprocess call — the test hook / pilot aid). A subprocess
|
||
failure (non-zero exit, unparseable stdout, timeout) is logged loudly; the run exits
|
||
**1** if no usable probe data was obtained at all (dispatch loop never starts).
|
||
|
||
### 6. Gates (evaluated per provider, every cycle — including on re-probe after each
|
||
completion)
|
||
|
||
For each provider in `config["providers"]`, using that provider's `seven_day` and
|
||
`five_hour` window records from the probe report:
|
||
|
||
```
|
||
idle_points = threshold_pct × elapsed_pct − usage_pct (on the seven_day window)
|
||
```
|
||
|
||
A provider is **eligible** for a new dispatch iff **all** of:
|
||
|
||
- the probe marks it `available: true`
|
||
- `seven_day.elapsed_pct` is non-null (null → "cannot pace", fail closed)
|
||
- `seven_day.utilization_pct` is non-null
|
||
- `five_hour.utilization_pct` is non-null
|
||
- `idle_points > min_idle`
|
||
- `five_hour.utilization_pct < five_hour_ceiling`
|
||
- the credential-parity check (§7) passes for that provider
|
||
|
||
All three gate values (`idle_points`, `five_hour.utilization_pct`, eligibility) are
|
||
logged for every provider on every cycle, whether or not the provider ends up used.
|
||
|
||
### 7. Credential parity
|
||
|
||
Before a provider can be selected, idle-draft resolves the credential its profile would
|
||
export at launch and compares it (by content, via SHA-256) against the credential
|
||
`agent-subscriptions` used to probe that same provider:
|
||
|
||
- **Profile side:** if `<profile>/provider.env` defines `ANTHROPIC_API_KEY_FILE`, read
|
||
that file's content. If the profile has no `provider.env` (the default Anthropic
|
||
profile), use the same token file `agent-subscriptions` reads for Anthropic
|
||
(`~/dev/claude/secrets/anthropic/api_key`).
|
||
- **Probe side:** for `anthropic`, the same `~/dev/claude/secrets/anthropic/api_key`
|
||
file content. For `minimax`, the SOPS-decrypted `ANTHROPIC_AUTH_TOKEN` value from
|
||
`agent-subscriptions`' MiniMax path (same SOPS file, same key file, same dotenv key
|
||
— these constants are replicated locally and must be kept in sync with
|
||
`scripts/agent-subscriptions` if that script's paths change).
|
||
|
||
A mismatch (or either side unreadable) makes the provider ineligible for this cycle and
|
||
logs a loud line: `credential mismatch: <provider> profile≠probe (<reason>)`. The gate
|
||
must meter the account that actually spends, not the account the profile file merely
|
||
names.
|
||
|
||
### 8. Work-item discovery and stage derivation
|
||
|
||
For each dossier in `config["dossiers"]`, glob `<repo>/<dossier>/*.overview.md` matching
|
||
`^(\d+)-(.+)\.overview\.md$`. For each `NN-slug`, derive the next eligible work type
|
||
(or "not eligible, because...") purely from which sibling files exist plus the item's
|
||
state entry:
|
||
|
||
| Condition | Result |
|
||
|---|---|
|
||
| `state.blocked` is set | not eligible: blocked |
|
||
| `state.approved` is true | not eligible: approved (terminal) |
|
||
| `NN-slug.agent.md` missing | not eligible: no commissioning brief |
|
||
| `NN-slug.research.md` missing | next: `research` |
|
||
| `NN-slug.draft.md` missing | next: `draft` |
|
||
| `NN-slug.review.md` exists | not eligible: awaiting human revise/approve |
|
||
| `state.human_edit_done` is not true | not eligible: waiting on human edit |
|
||
| (all of the above pass) | next: `review` |
|
||
|
||
In-flight items (a task currently dispatched for that item in this run) are excluded
|
||
from consideration for further dispatch until the in-flight task completes.
|
||
|
||
**Cold-start throttle:** per dossier, count `NN-slug.research.md` files that exist and
|
||
whose state entry does **not** have `research_sampled: true`. Once that count reaches
|
||
`max_unreviewed_research_per_dossier`, no further `research` candidates are offered for
|
||
that dossier this cycle (draft/review candidates in that dossier are unaffected).
|
||
|
||
**`topic_ideas`:** dossier-level (not tied to an `NN-slug`), considered only when the
|
||
combined candidate list above (across all dossiers) is empty. For each dossier, eligible
|
||
iff the count of proposals already recorded in `<dossier>/TOPIC-PROPOSALS.md` (one `##`
|
||
heading per proposal; file absent counts as 0) is below `max_open_topic_proposals`.
|
||
|
||
### 9. Prioritisation
|
||
|
||
Ready `(item, work_type)` candidates (plus, only when the list would otherwise be
|
||
empty, `(dossier, topic_ideas)` candidates) are sorted by:
|
||
|
||
1. Stage rank, descending: `review` (3) > `draft` (2) > `research` (1) > `topic_ideas` (0).
|
||
2. Numeric filename prefix, ascending (`topic_ideas` sorts as `0`, always last within
|
||
its own rank tier — moot since it only appears when nothing else is ready).
|
||
3. Dossier config order (`config["dossiers"]` index), ascending, as the tiebreak.
|
||
|
||
For each candidate in this order, the provider is the **first** entry in
|
||
`config["work_types"][work_type]["providers"]` that is currently eligible (§6, §7). A
|
||
candidate with no eligible provider is skipped (not dispatched this cycle, tried again
|
||
next cycle); the walk continues to the next candidate.
|
||
|
||
### 10. Dispatch loop (worker pool, event-driven)
|
||
|
||
Up to `parallel` tasks run concurrently (`concurrent.futures.ThreadPoolExecutor`).
|
||
Each completion is handled **serially** in the main thread (state updates and git
|
||
commits never race). On every completion — and before the very first dispatch — the
|
||
gates (§6) are recomputed from a fresh probe (§5); a task is submitted to fill a free
|
||
worker slot only while an eligible `(item, work_type, provider)` triple remains. The
|
||
loop exits (dispatch mode, non-dryrun) when no eligible candidate remains. `--once`
|
||
dispatches at most one task total, then exits without waiting for further slots.
|
||
|
||
### 11. Task execution
|
||
|
||
For the selected `(item_or_dossier, work_type, provider)`:
|
||
|
||
1. Resolve the prompt template: `data/idle-draft/prompts/{research,draft,review-suggest,topic-ideas}.md`
|
||
(`review` work type uses `review-suggest.md`).
|
||
2. Render the template (`string.Template`, `$placeholder` substitution) with the
|
||
resolved paths for that item (overview, agent, research, draft as applicable),
|
||
dossier name, slug, style directory, source register path, and a temp output path.
|
||
3. Resolve the profile directory from `config["providers"][provider]["profile"]`
|
||
(`~` expanded). Build the child environment: `CLAUDE_CONFIG_DIR=<profile>`, plus —
|
||
if `<profile>/provider.env` exists — `ANTHROPIC_BASE_URL` (if set),
|
||
`ANTHROPIC_API_KEY` (read from `ANTHROPIC_API_KEY_FILE`, if set), and any other
|
||
`KEY=value` line verbatim. A profile with no `provider.env` (plain Anthropic) gets
|
||
only `CLAUDE_CONFIG_DIR`. This logic is a local re-implementation of
|
||
`scripts/claude-profile` lines ~320–440 — **`claude-profile` itself is never
|
||
invoked** (it has interactive pickers and terminal theming unsuitable for headless
|
||
cron use).
|
||
4. Build the argv: `claude -p --max-turns <N>` (`N` = 25 unless overridden), plus
|
||
`--model <id>` if `provider.env` sets `MODEL_ID`, plus `--add-dir <dir>` for each
|
||
entry in `config["evidence_dirs"]` **only** for `research` and `topic_ideas` work
|
||
types (the only ones that cite external evidence), plus `--allowedTools
|
||
"<comma-joined list>"` if `config["work_types"][work_type]["allowed_tools"]` is
|
||
present and non-empty (see Config schema below). Headless `claude -p` runs cannot
|
||
answer permission prompts, so any tool beyond the CLI's defaults must be granted
|
||
this way or the child run stalls/fails; when the key is absent or an empty list,
|
||
the flag is omitted entirely (unchanged pre-existing behaviour).
|
||
5. Run the subprocess: `cwd=<repo>`, `env=<built env>`, prompt piped via **stdin**
|
||
(not as an argv element — avoids `ARG_MAX` on large rendered prompts, same lesson
|
||
`claude-profile` already applies to its system-prompt injection), timeout = per-task
|
||
timeout (1800s unless overridden).
|
||
6. Classify the result:
|
||
- **Timeout** → transient failure. Do not increment `attempts`.
|
||
- **Non-zero exit** whose stderr matches a retryable signature (`429`, `5xx`,
|
||
`rate limit`, `overloaded`, `timeout`, `temporarily unavailable`, connection-reset
|
||
markers) → transient failure. Do not increment `attempts`.
|
||
- **Non-zero exit**, no retryable signature, or **exit 0 but validation fails below**
|
||
→ content failure. Increment `state.items[item].attempts[work_type]`. At
|
||
`max_attempts` (2) content failures for that work type, set
|
||
`state.items[item].blocked` with a reason (the item is then excluded from all
|
||
further dispatch until a human runs `mark unblock`).
|
||
- **Exit 0 and validation passes** → success.
|
||
7. Validation (on exit 0, before promotion): the temp output file must be non-empty and
|
||
its first non-blank line must be a top-level Markdown heading (`# ...`). For
|
||
`research` work type specifically: every absolute path matching `/home/\S+` cited in
|
||
the file must exist on disk (`Path.exists()`); any dead path fails validation. Citations
|
||
may carry a trailing line-range suffix in `path:120-145` or `path:120` form (per the
|
||
research prompt template's required citation format); this suffix is stripped — via
|
||
`re.sub(r":\d+(-\d+)?$", ...)`, only when the remainder still looks like a path — before
|
||
the existence check, so a valid ranged citation is not flagged as a dead path.
|
||
8. On validation pass: `os.replace()` the temp file to the canonical path
|
||
(`<repo>/<dossier>/<NN-slug>.<work_type>.md`, or append to
|
||
`<dossier>/TOPIC-PROPOSALS.md` for `topic_ideas`) — atomic, never a partial file
|
||
visible under the canonical name.
|
||
9. On content-failure: `os.replace()` the temp file to
|
||
`<repo>/<dossier>/<NN-slug>.<work_type>.md.rejected` (kept for human inspection,
|
||
never promoted, never committed to the canonical name; `topic_ideas` content
|
||
failures are simply discarded — nothing is appended, nothing is blocked, since
|
||
`topic_ideas` has no per-item state entry to carry an attempt counter).
|
||
10. On transient failure: temp file is discarded; no state change; no commit; the
|
||
candidate may be retried on a later cycle.
|
||
11. On success: write `idle-draft.state.json` atomically (temp file + `os.replace()` in
|
||
the same directory), then:
|
||
```
|
||
git -C <repo> add -- <produced-file> idle-draft.state.json
|
||
git -C <repo> commit -m "<message>" -- <produced-file> idle-draft.state.json
|
||
```
|
||
Never `git add -A`, never `git commit -a`. Commit message:
|
||
`idle-draft: <work_type> <item> via <provider> (7d <before>%→<after>%)`, where
|
||
before/after are that provider's `seven_day.utilization_pct` immediately before
|
||
dispatch and immediately after re-probe on completion. No push.
|
||
|
||
### 12. Logging
|
||
|
||
One line per event (gate decision, dispatch, completion, failure, commit) appended to
|
||
`<repo>/idle-draft.log` with an ISO-8601 timestamp, mirrored to stderr.
|
||
|
||
## Dryrun behaviour
|
||
|
||
`--dryrun` runs the full probe (or reads `--probe-json`), computes all gates, builds
|
||
the ready queue, and selects up to `parallel` (or `1`, under `--once`) `(item,
|
||
work_type, provider)` triples exactly as the real dispatch loop's first wave would —
|
||
**without** simulating gate depletion across a re-probe (a real run only knows that
|
||
after actually dispatching; the preview shows the initial wave only, noted as such).
|
||
For each selected triple it prints, and executes nothing:
|
||
|
||
- item or dossier identifier, work type, chosen provider
|
||
- resolved prompt template path
|
||
- the exact `claude` argv list that would run
|
||
- the child env deltas (`CLAUDE_CONFIG_DIR`, `ANTHROPIC_BASE_URL` if set, whether
|
||
`ANTHROPIC_API_KEY` would be exported)
|
||
- the temp output path and the canonical destination path
|
||
|
||
No subprocess is run, no file is written, no state is mutated, no lock is required to
|
||
be free for the preview to work (the lock is still attempted and its outcome reported,
|
||
but a held lock does not block the dryrun preview from computing and printing — real
|
||
dispatch mode is what respects the lock as a hard gate). Exit 0.
|
||
|
||
## State schema
|
||
|
||
`idle-draft.state.json`, repo root — see §4. Humans never hand-edit this file; the
|
||
`mark` subcommand is the only mutation path. Derived state (anything the filesystem
|
||
already says) is never written here.
|
||
|
||
## Config schema
|
||
|
||
`idle-draft.config.json` — see `data/idle-draft/config.example.json` for the exact
|
||
structure. All top-level keys listed in §3 are required.
|
||
|
||
Each entry in `work_types` may carry an optional `allowed_tools` key: a list of Claude
|
||
Code tool names (e.g. `["WebSearch", "WebFetch"]`) granted to that work type's headless
|
||
child via `--allowedTools` (§11 step 4). Present-but-empty and absent are both treated
|
||
as "no extra tools" — the flag is omitted. If present, it must be a JSON array of
|
||
strings; any other shape (a string, a number, a list containing a non-string) is a
|
||
config validation failure — exit **2** at config load, same as a missing required key.
|
||
`config.example.json` sets `allowed_tools: ["WebSearch", "WebFetch"]` on `research` and
|
||
`topic_ideas` (the research prompt template requires at least one `WebSearch` call
|
||
before writing its Subjective stories section; the topic-ideas template requires one as
|
||
a lighter-touch check); `draft` and `review` carry no `allowed_tools` key.
|
||
|
||
## Failure classes
|
||
|
||
| Class | Examples | `attempts` effect | Item outcome |
|
||
|---|---|---|---|
|
||
| Transient | timeout, HTTP 429/5xx, network reset | not incremented | retried next eligible cycle |
|
||
| Content | empty output, missing heading, dead citation path, non-retryable non-zero exit | incremented | `.rejected` kept; `blocked` at `max_attempts` (2) |
|
||
|
||
## Exit codes
|
||
|
||
| Code | Meaning |
|
||
|---|---|
|
||
| 0 | Nothing to do, or all dispatched tasks completed without a fatal problem (individual task failures are logged, not fatal) |
|
||
| 1 | Transient/probe problem prevented the run from proceeding (no usable probe data, lock held is exit 0 not 1 — see §1) |
|
||
| 2 | Config or state validation failure |
|
||
|
||
## Edge cases
|
||
|
||
| Scenario | Handling |
|
||
|---|---|
|
||
| Lock already held | Log one line, exit 0 (not an error — another cron tick is running) |
|
||
| `idle-draft.state.json` missing | Treated as `{"items": {}}`, not an error |
|
||
| `idle-draft.state.json` present but invalid | Exit 2, loud message, run never starts |
|
||
| Item's `.agent.md` missing | Item excluded from all dispatch (commissioning briefs exist today for every current item; this guards future additions) |
|
||
| `seven_day.elapsed_pct` null | Provider ineligible this cycle, never assumed idle |
|
||
| Provider `available: false` | Provider ineligible this cycle |
|
||
| Credential mismatch | Provider ineligible this cycle, loud log line |
|
||
| `research` output cites a path that doesn't exist | Content failure, quarantined as `.rejected`, attempts incremented |
|
||
| `max_attempts` reached | Item `blocked`, excluded until `mark unblock` |
|
||
| `topic_ideas` content failure | Discarded silently (logged, not blocked — no per-item state key exists for a dossier-level work type) |
|
||
| Two work types both ready in the same dossier | Higher stage rank wins (review > draft > research); `topic_ideas` never competes (last-resort only) |
|
||
| `--probe-json` combined with `--dryrun` | Composes normally — no subprocess call either way |
|
||
| `--once` with no eligible work | Exits 0 immediately, no task dispatched |
|
||
| Config missing a required key | Exit 2 |
|
||
| `work_types[*].allowed_tools` present but not a list of strings | Exit 2 |
|
||
| `agent-subscriptions` subprocess fails entirely | Exit 1, loud log line, no dispatch attempted |
|
||
|
||
## Examples
|
||
|
||
```sh
|
||
idle-draft --dryrun # preview one dispatch wave
|
||
idle-draft --once # pilot: one real task, then exit
|
||
idle-draft --probe-json /tmp/fake-probe.json --dryrun # preview against fixture data
|
||
idle-draft --config ~/dev/claude/writing/oreillyconsulting/idle-draft.config.json
|
||
idle-draft mark ai/03-shadow-agents-are-the-new-shadow-it edited
|
||
idle-draft mark ai/03-shadow-agents-are-the-new-shadow-it sampled
|
||
idle-draft status
|
||
```
|