idle-draft: idle-subscription dispatcher for the writing pipeline

Elapsed-paced gates, bounded-parallel event-driven dispatch, filesystem-
derived stages with thin human-gate state, citation validation with
quarantine, credential parity check, prompt templates as data.
Per IDLE-DRAFT-PLAN.md in writing/oreillyconsulting.

Claude-Session: https://claude.ai/code/session_01YQDoWNM7XPPii28khFWoMc
This commit is contained in:
Paul O'Reilly
2026-08-02 21:18:02 +12:00
parent f78d292f05
commit 6c4bf449aa
8 changed files with 2398 additions and 0 deletions

View File

@@ -0,0 +1,24 @@
{
"parallel": 2,
"providers": {
"anthropic": { "profile": "~/.claude-oreillyit-anthropic", "threshold_pct": 80,
"five_hour_ceiling": 50, "min_idle": 5 },
"minimax": { "profile": "~/.claude-oreillyit-minimax", "threshold_pct": 80,
"five_hour_ceiling": 50, "min_idle": 5 }
},
"work_types": {
"review": { "providers": ["anthropic"] },
"draft": { "providers": ["anthropic"] },
"research": { "providers": ["anthropic", "minimax"] },
"topic_ideas": { "providers": ["anthropic", "minimax"] }
},
"dossiers": ["ai", "ai-technical", "devops-2020", "devops-2020-technical"],
"review_score_threshold": 11,
"max_unreviewed_research_per_dossier": 3,
"max_open_topic_proposals": 6,
"evidence_dirs": [
"~/dev/claude/octopus/goes",
"~/dev/claude/projects/agent-runtimes",
"~/dev/claude/projects/agent-runtimes-work-items"
]
}

View File

@@ -0,0 +1,55 @@
You are producing the **published-track draft article** for one topic in an O'Reilly
Consulting writing dossier. This is unattended, headless work — a human edits this
draft before it moves further, but there is no human in the loop while you write it.
Follow this brief exactly; do not invent structure.
## Item
- Dossier: `$dossier`
- Topic: `$slug`
- Topic brief: $overview_path
- Commissioning brief — **this is the contract for the article; follow its required
structure exactly**: $agent_path
- Research file — your evidence source, already validated: $research_path
- Dossier commissioning guide: $dossier_agents_path
- Project-root voice, style, and citation rules — read before writing a single line:
$agents_root_path
- Voice reference (read this one in full; it is the canonical synthesis): $style_dir/goes-target-voice.md
## What to produce
Follow the commissioning brief's required structure exactly — it is the contract, not
a suggestion. Do not add sections it doesn't name; do not omit sections it does.
## Voice rules (non-negotiable)
- **Sentence-case headings** everywhere: "The decision," not "The Decision."
- **Consulting dossiers** (`ai`, `devops-2020`): open with a judgment the reader would
pay for, not a category they already know. **Do not name vendors in prose.** If a
source credits a specific vendor or product, say "the platform team" or "their
engineering organisation" instead — vendor names stay in the source register and the
commissioning brief, never in the published copy.
- **Technical dossiers** (`ai-technical`, `devops-2020-technical`): open with the
specific toil the reader lives with, not the category a vendor markets. Naming a
specific tool, script, or system by its actual identifier is fine here — this is
implementation detail, not vendor marketing.
- **Cite the file, not the concept**, for any code or spec reference (path + line
range). **Make a claim, cite the case, name the source**, for any research finding.
Use the dossier's correct status-label family (see `$agents_root_path`) on every
cited claim.
- **Distinguish research from implementation.** Do not conflate "the research finds X"
with "the implementation does Y." The research is the question; the implementation
is one possible answer; state the connection explicitly.
- No hedge words where the evidence supports a direct claim. No marketing language. No
AI tells (see the burn list in `$style_dir/tone-and-voice.md` if you are unsure what
that means).
## Output
Write the complete Markdown article to exactly this path, and nothing else:
$output_path
The dispatcher validates this file (non-empty, starts with a top-level heading) and
promotes it to its canonical location itself. A human edits this draft next — write it
as if a careful editor, not a lenient one, will read it in the next hour.

View File

@@ -0,0 +1,89 @@
You are producing the **research file** for one topic in an O'Reilly Consulting
writing dossier. This is unattended, headless work — there is no human in the loop
until a reviewer samples your output later. Follow this brief exactly.
## Item
- Dossier: `$dossier`
- Topic: `$slug`
- Topic brief (read first, in full): $overview_path
- Commissioning brief (read second, in full): $agent_path
- Dossier commissioning guide (dossier-specific rules): $dossier_agents_path
- Project-root voice and citation rules (read before writing a single line): $agents_root_path
- Source register (existing sources for this dossier — read before searching for new
ones; do not duplicate a source it already lists): $source_register_path
Evidence you may cite must live under one of these directories (you have read access
via `--add-dir`; do not invent paths outside them):
$evidence_dirs
## What to produce
Write a Markdown research file with two kinds of content, matching what the topic
brief's evidence section and the commissioning brief already point at, extended with
anything new and relevant you find under the evidence directories above:
1. **Objective evidence** — concrete examples, code references (file path + line range,
not just "the script does X"), measurables, named findings. Every claim is sourced.
2. **Subjective stories** — case-study-like narratives that make the evidence concrete
for a reader (a scene, a named organisation's experience, a practitioner's account)
— still sourced, but framed as narrative rather than a bare statistic.
## Citation rules (non-negotiable — read `$agents_root_path` for the full statement)
- **Cite the file, not the concept.** Any code or spec reference must name the file
path and the line range: `path/to/file.py:120-145`, not "the script does X."
- **Cite the requirement ID where one exists** (e.g. `H-*`, `AU-*`, `WT-*`, `KX-*`,
`BRK-*`, `WF-*` prefixes in the agent-runtimes spec) — quote the ID, not a prose
paraphrase.
- **Quote metric markers verbatim** where GOES research uses them (`{p.3}`, `{s.4}`,
`{t.00:14:32}`) — do not renumber or invent markers.
- Use the dossier's status-label family and apply it correctly:
- `ai` / `ai-technical` dossiers use **implementation maturity**: "Implemented and
running in production" / "Designed and specified, implementation in progress" /
"Spec-only / future." A claim about an `implemented` feature may report a deployed
behaviour or measured metric; a `spec-only` claim may not.
- `devops-2020` / `devops-2020-technical` dossiers use **evidence quality**: "Named
organisation, self-reported" / "Research-survey finding" / "GOES synthesis / bet" /
"Vendor data, treat with caution." Label every cited source with the correct one.
- Every source you cite gets an explicit status label. Do not present an unlabelled
claim as settled fact.
## Required structure
```markdown
# Research: <topic title, sentence case>
## Objective evidence
<bulleted or short-paragraph entries, each with file/line or source citation and
status label>
## Subjective stories
<narrative entries, each still sourced and status-labelled>
## Register delta
<a bulleted list of every source used in this file that `$source_register_path` does
NOT already list — path, one-line description, proposed status label. If every source
you used is already in the register, write "None — all sources already registered."
This section is the only place new sources are proposed; you do not edit the register
file itself.>
```
All headings are **sentence case** ("Objective evidence," not "Objective Evidence").
## Output
Write the complete Markdown file to exactly this path, and nothing else — do not touch
any other file in the repository, do not edit the source register, do not create the
canonical `$slug.research.md` file yourself:
$output_path
The dispatcher validates this file (non-empty, starts with a top-level heading, every
absolute `/home/...` path you cite resolves on disk) and promotes it to its canonical
location itself. If any cited absolute path does not exist, your output will be
rejected — verify paths before citing them, don't guess.

View File

@@ -0,0 +1,62 @@
You are producing the **review file** for one human-edited draft in an O'Reilly
Consulting writing dossier. The draft has already been scored per-paragraph on the
5-dimension `/15` rubric (Evidence, Judgment, Voice, Rhythm, Register — see
`score-paragraphs`); your job is the **second pass**: concrete, actionable improvement
suggestions for every paragraph that scored below the threshold.
## Item
- Dossier: `$dossier`
- Topic: `$slug`
- Human-edited draft (already scored, score blocks inline): $draft_path
- Grading template — apply the same rubric and voice bar when suggesting fixes:
$review_prompt_path
- Voice reference: $style_dir/goes-target-voice.md
- Score threshold on the `/15` scale: **$review_score_threshold** — any paragraph
scoring below this needs a suggestion; paragraphs at or above it need none.
## What to produce
For every paragraph in the draft whose inline score block shows a total below
`$review_score_threshold`:
- Quote the paragraph's opening few words so a human can locate it.
- Name which of the five dimensions is weakest and why, in one sentence.
- Give a **concrete rewrite direction** — not "improve the evidence," but what
specific evidence, source, or file/line citation would raise the score, or what
specific sentence-level change (cut a hedge word, vary sentence length, cut a
vendor name, cite the file instead of the concept) would fix it.
Do not rewrite the paragraph yourself — suggest, don't replace. The human author makes
the edit.
## Required structure
```markdown
# Review: <topic title, sentence case>
## Summary
<one paragraph: how many paragraphs scored below threshold, out of how many total,
and the single most common weak dimension>
## Suggestions
### Paragraph opening: "<first few words...>"
- Score: <total>/15 (weakest dimension: <name>)
- Suggestion: <concrete, specific>
<repeat per below-threshold paragraph, in document order>
```
All headings are **sentence case**.
## Output
Write the complete Markdown file to exactly this path, and nothing else:
$output_path
The dispatcher validates this file (non-empty, starts with a top-level heading) and
promotes it to its canonical location itself.

View File

@@ -0,0 +1,67 @@
You are proposing **new candidate topics** for one dossier in the O'Reilly Consulting
writing pipeline. This runs only when no other work is currently eligible — the
dossier's existing topics are all blocked on a human step or already have enough work
in flight. Do not invent structure; follow this brief.
## Dossier
- Dossier: `$dossier`
- Dossier commissioning guide: $dossier_agents_path
- Project-root voice and citation rules: $agents_root_path
- Voice reference: $style_dir/goes-target-voice.md
Evidence you may cite must live under one of these directories (you have read access
via `--add-dir`):
$evidence_dirs
## Do not duplicate an existing topic
These titles already exist as numbered topic briefs or open proposals in this dossier.
Do **not** propose anything that duplicates or trivially rephrases one of these:
$existing_titles
## What to produce
Propose **up to $max_new** new candidate topics for this dossier, each grounded in
something you actually found under the evidence directories above — not a generic
industry-trend guess. For each candidate:
- A working title (sentence case, matching the dossier's existing topic-title style).
- A one-paragraph pitch: the decision or toil the topic would address, matching the
audience of this dossier (CTO/VP for consulting dossiers, implementing engineers for
technical dossiers — see `$dossier_agents_path` for which this is).
- 24 evidence pointers: file paths under the evidence directories, with a one-line
note on what each contains. Use the dossier's correct status-label family (see
`$agents_root_path`) on each.
You are proposing, not commissioning: **never create a numbered `NN-slug` file.** A
human promotes a proposal to a numbered brief later.
## Required structure (append this, do not replace the file)
```markdown
## <Working title, sentence case>
**Pitch:** <one paragraph>
**Evidence:**
- <path> — <one-line note, status label>
- <path> — <one-line note, status label>
```
One `##` block per proposed topic. All headings sentence case. No vendor names in
prose for consulting dossiers (`ai`, `devops-2020`) — vendor names may stay in the
evidence pointers' one-line notes only.
## Output
Write **only the new `##` blocks** (not the whole file — this content will be appended
to the dossier's existing `TOPIC-PROPOSALS.md`) to exactly this path:
$output_path
The dispatcher validates this file (non-empty, starts with a top-level `##` heading)
and appends it to the canonical proposals file itself.

1273
scripts/idle-draft Executable file

File diff suppressed because it is too large Load Diff

355
specs/idle-draft.spec.md Normal file
View File

@@ -0,0 +1,355 @@
# idle-draft
## Purpose
Cron-invoked dispatcher that consumes idle Anthropic/MiniMax subscription capacity to
advance the O'Reilly Consulting writing pipeline (`~/dev/claude/writing/oreillyconsulting`)
unattended: probes usage via `agent-subscriptions`, computes idle capacity per provider,
dispatches ready work items to headless `claude -p` with bounded parallelism, re-probes
on every completion, validates output, and commits results.
Design contract: `~/dev/claude/writing/oreillyconsulting/IDLE-DRAFT-PLAN.md`. This spec
translates that plan into the repo's implementation contract; where the two disagree,
the plan wins and this file should be corrected.
## Usage
```
idle-draft [OPTIONS]
idle-draft mark <dossier/NN-slug> edited|sampled|approved|unblock [OPTIONS]
idle-draft status [OPTIONS]
```
### Options (dispatch mode and `status`)
| Flag | Default | Description |
|---|---|---|
| `--config FILE` | `./idle-draft.config.json` | Path to the config file |
| `--repo DIR` | config file's directory | Writing repo root (contains the dossiers) |
| `--once` | off | Dispatch at most one task, then exit (pilot mode) |
| `--parallel N` | from config, or `2` | Max concurrent tasks (overrides config) |
| `--dryrun`, `-n` | off | Probe + gate + select, print what would be dispatched, execute and mutate nothing |
| `--probe-json FILE` | none | Read probe output from FILE instead of running `agent-subscriptions` (test hook / pilot aid) |
| `--help`, `-h` | — | Show usage and exit 0 |
### `mark` subcommand
```
idle-draft mark <dossier/NN-slug> edited|sampled|approved|unblock [--config FILE] [--repo DIR]
```
The only human mutation path into `idle-draft.state.json`. Validates the item resolves
to an existing `<repo>/<dossier>/NN-slug.overview.md`, loads and schema-validates the
state file, applies exactly one field mutation, writes atomically.
| Action | Effect |
|---|---|
| `edited` | `human_edit_done = true` |
| `sampled` | `research_sampled = true` |
| `approved` | `approved = true` |
| `unblock` | `blocked = null` |
### `status` subcommand
Human-readable report: per-item derived stage, human-gate flags, next eligible action
(or the reason it is not eligible), and per-provider gate values (`idle_points`,
`five_hour_pct`, eligibility) from a live probe (or `--probe-json`). Read-only; takes no
lock, mutates nothing.
## Behaviour
### 1. Locking
Before doing anything else (dispatch mode only — `mark` and `status` do not take the
lock), `flock` (`fcntl.flock`, `LOCK_EX | LOCK_NB`) on `<repo>/.idle-draft.lock`. If
already locked by another invocation, log one line and exit **0** — a concurrent cron
tick is not a failure.
### 2. Log rotation
At startup, if `<repo>/idle-draft.log` exceeds 5 MB, rename it to `idle-draft.log.1`
(overwriting any existing `.1`) before appending further.
### 3. Config load
Parse `--config` as JSON. Required top-level keys: `parallel`, `providers`, `work_types`,
`dossiers`, `review_score_threshold`, `max_unreviewed_research_per_dossier`,
`max_open_topic_proposals`, `evidence_dirs` — matching `data/idle-draft/config.example.json`
verbatim. Missing required keys or malformed structure → log loudly, exit **2**.
`--parallel` on the CLI overrides the config value.
### 4. State load and validation
Read `<repo>/idle-draft.state.json`. Missing file is not an error — treat as
`{"items": {}}`. If present:
- Must parse as JSON.
- Top level must be an object with **only** the key `items` (unknown top-level keys
rejected).
- `items` must be an object. Each key must be of the form `<dossier>/<NN-slug>` and
must resolve to an existing `<repo>/<dossier>/<NN-slug>.overview.md` — an item entry
for a file that doesn't exist is rejected.
- Each item value must be an object containing **only** these keys (all optional,
defaults shown): `human_edit_done` (bool, default `false`), `research_sampled` (bool,
default `false`), `approved` (bool, default `false`), `blocked` (string or `null`,
default `null`), `attempts` (object mapping `research`/`draft`/`review` → non-negative
int, default `{}`). Any other key, or a wrong-typed value, is rejected.
Any validation failure aborts the run loudly (message to stderr and the log) with
**exit 2** — never "best effort," never silently ignored. This applies to `mark` and
`status` too (both load and validate the state file before proceeding).
### 5. Probe
Run `agent-subscriptions --output json` as a subprocess and parse stdout as JSON,
**unless** `--probe-json FILE` is given, in which case that file's contents are used
verbatim instead (no subprocess call — the test hook / pilot aid). A subprocess
failure (non-zero exit, unparseable stdout, timeout) is logged loudly; the run exits
**1** if no usable probe data was obtained at all (dispatch loop never starts).
### 6. Gates (evaluated per provider, every cycle — including on re-probe after each
completion)
For each provider in `config["providers"]`, using that provider's `seven_day` and
`five_hour` window records from the probe report:
```
idle_points = threshold_pct × elapsed_pct usage_pct (on the seven_day window)
```
A provider is **eligible** for a new dispatch iff **all** of:
- the probe marks it `available: true`
- `seven_day.elapsed_pct` is non-null (null → "cannot pace", fail closed)
- `seven_day.utilization_pct` is non-null
- `five_hour.utilization_pct` is non-null
- `idle_points > min_idle`
- `five_hour.utilization_pct < five_hour_ceiling`
- the credential-parity check (§7) passes for that provider
All three gate values (`idle_points`, `five_hour.utilization_pct`, eligibility) are
logged for every provider on every cycle, whether or not the provider ends up used.
### 7. Credential parity
Before a provider can be selected, idle-draft resolves the credential its profile would
export at launch and compares it (by content, via SHA-256) against the credential
`agent-subscriptions` used to probe that same provider:
- **Profile side:** if `<profile>/provider.env` defines `ANTHROPIC_API_KEY_FILE`, read
that file's content. If the profile has no `provider.env` (the default Anthropic
profile), use the same token file `agent-subscriptions` reads for Anthropic
(`~/dev/claude/secrets/anthropic/api_key`).
- **Probe side:** for `anthropic`, the same `~/dev/claude/secrets/anthropic/api_key`
file content. For `minimax`, the SOPS-decrypted `ANTHROPIC_AUTH_TOKEN` value from
`agent-subscriptions`' MiniMax path (same SOPS file, same key file, same dotenv key
— these constants are replicated locally and must be kept in sync with
`scripts/agent-subscriptions` if that script's paths change).
A mismatch (or either side unreadable) makes the provider ineligible for this cycle and
logs a loud line: `credential mismatch: <provider> profile≠probe (<reason>)`. The gate
must meter the account that actually spends, not the account the profile file merely
names.
### 8. Work-item discovery and stage derivation
For each dossier in `config["dossiers"]`, glob `<repo>/<dossier>/*.overview.md` matching
`^(\d+)-(.+)\.overview\.md$`. For each `NN-slug`, derive the next eligible work type
(or "not eligible, because...") purely from which sibling files exist plus the item's
state entry:
| Condition | Result |
|---|---|
| `state.blocked` is set | not eligible: blocked |
| `state.approved` is true | not eligible: approved (terminal) |
| `NN-slug.agent.md` missing | not eligible: no commissioning brief |
| `NN-slug.research.md` missing | next: `research` |
| `NN-slug.draft.md` missing | next: `draft` |
| `NN-slug.review.md` exists | not eligible: awaiting human revise/approve |
| `state.human_edit_done` is not true | not eligible: waiting on human edit |
| (all of the above pass) | next: `review` |
In-flight items (a task currently dispatched for that item in this run) are excluded
from consideration for further dispatch until the in-flight task completes.
**Cold-start throttle:** per dossier, count `NN-slug.research.md` files that exist and
whose state entry does **not** have `research_sampled: true`. Once that count reaches
`max_unreviewed_research_per_dossier`, no further `research` candidates are offered for
that dossier this cycle (draft/review candidates in that dossier are unaffected).
**`topic_ideas`:** dossier-level (not tied to an `NN-slug`), considered only when the
combined candidate list above (across all dossiers) is empty. For each dossier, eligible
iff the count of proposals already recorded in `<dossier>/TOPIC-PROPOSALS.md` (one `##`
heading per proposal; file absent counts as 0) is below `max_open_topic_proposals`.
### 9. Prioritisation
Ready `(item, work_type)` candidates (plus, only when the list would otherwise be
empty, `(dossier, topic_ideas)` candidates) are sorted by:
1. Stage rank, descending: `review` (3) > `draft` (2) > `research` (1) > `topic_ideas` (0).
2. Numeric filename prefix, ascending (`topic_ideas` sorts as `0`, always last within
its own rank tier — moot since it only appears when nothing else is ready).
3. Dossier config order (`config["dossiers"]` index), ascending, as the tiebreak.
For each candidate in this order, the provider is the **first** entry in
`config["work_types"][work_type]["providers"]` that is currently eligible (§6, §7). A
candidate with no eligible provider is skipped (not dispatched this cycle, tried again
next cycle); the walk continues to the next candidate.
### 10. Dispatch loop (worker pool, event-driven)
Up to `parallel` tasks run concurrently (`concurrent.futures.ThreadPoolExecutor`).
Each completion is handled **serially** in the main thread (state updates and git
commits never race). On every completion — and before the very first dispatch — the
gates (§6) are recomputed from a fresh probe (§5); a task is submitted to fill a free
worker slot only while an eligible `(item, work_type, provider)` triple remains. The
loop exits (dispatch mode, non-dryrun) when no eligible candidate remains. `--once`
dispatches at most one task total, then exits without waiting for further slots.
### 11. Task execution
For the selected `(item_or_dossier, work_type, provider)`:
1. Resolve the prompt template: `data/idle-draft/prompts/{research,draft,review-suggest,topic-ideas}.md`
(`review` work type uses `review-suggest.md`).
2. Render the template (`string.Template`, `$placeholder` substitution) with the
resolved paths for that item (overview, agent, research, draft as applicable),
dossier name, slug, style directory, source register path, and a temp output path.
3. Resolve the profile directory from `config["providers"][provider]["profile"]`
(`~` expanded). Build the child environment: `CLAUDE_CONFIG_DIR=<profile>`, plus —
if `<profile>/provider.env` exists — `ANTHROPIC_BASE_URL` (if set),
`ANTHROPIC_API_KEY` (read from `ANTHROPIC_API_KEY_FILE`, if set), and any other
`KEY=value` line verbatim. A profile with no `provider.env` (plain Anthropic) gets
only `CLAUDE_CONFIG_DIR`. This logic is a local re-implementation of
`scripts/claude-profile` lines ~320440 — **`claude-profile` itself is never
invoked** (it has interactive pickers and terminal theming unsuitable for headless
cron use).
4. Build the argv: `claude -p --max-turns <N>` (`N` = 25 unless overridden), plus
`--model <id>` if `provider.env` sets `MODEL_ID`, plus `--add-dir <dir>` for each
entry in `config["evidence_dirs"]` **only** for `research` and `topic_ideas` work
types (the only ones that cite external evidence).
5. Run the subprocess: `cwd=<repo>`, `env=<built env>`, prompt piped via **stdin**
(not as an argv element — avoids `ARG_MAX` on large rendered prompts, same lesson
`claude-profile` already applies to its system-prompt injection), timeout = per-task
timeout (1800s unless overridden).
6. Classify the result:
- **Timeout** → transient failure. Do not increment `attempts`.
- **Non-zero exit** whose stderr matches a retryable signature (`429`, `5xx`,
`rate limit`, `overloaded`, `timeout`, `temporarily unavailable`, connection-reset
markers) → transient failure. Do not increment `attempts`.
- **Non-zero exit**, no retryable signature, or **exit 0 but validation fails below**
→ content failure. Increment `state.items[item].attempts[work_type]`. At
`max_attempts` (2) content failures for that work type, set
`state.items[item].blocked` with a reason (the item is then excluded from all
further dispatch until a human runs `mark unblock`).
- **Exit 0 and validation passes** → success.
7. Validation (on exit 0, before promotion): the temp output file must be non-empty and
its first non-blank line must be a top-level Markdown heading (`# ...`). For
`research` work type specifically: every absolute path matching `/home/\S+` cited in
the file must exist on disk (`Path.exists()`); any dead path fails validation.
8. On validation pass: `os.replace()` the temp file to the canonical path
(`<repo>/<dossier>/<NN-slug>.<work_type>.md`, or append to
`<dossier>/TOPIC-PROPOSALS.md` for `topic_ideas`) — atomic, never a partial file
visible under the canonical name.
9. On content-failure: `os.replace()` the temp file to
`<repo>/<dossier>/<NN-slug>.<work_type>.md.rejected` (kept for human inspection,
never promoted, never committed to the canonical name; `topic_ideas` content
failures are simply discarded — nothing is appended, nothing is blocked, since
`topic_ideas` has no per-item state entry to carry an attempt counter).
10. On transient failure: temp file is discarded; no state change; no commit; the
candidate may be retried on a later cycle.
11. On success: write `idle-draft.state.json` atomically (temp file + `os.replace()` in
the same directory), then:
```
git -C <repo> add -- <produced-file> idle-draft.state.json
git -C <repo> commit -m "<message>" -- <produced-file> idle-draft.state.json
```
Never `git add -A`, never `git commit -a`. Commit message:
`idle-draft: <work_type> <item> via <provider> (7d <before>%→<after>%)`, where
before/after are that provider's `seven_day.utilization_pct` immediately before
dispatch and immediately after re-probe on completion. No push.
### 12. Logging
One line per event (gate decision, dispatch, completion, failure, commit) appended to
`<repo>/idle-draft.log` with an ISO-8601 timestamp, mirrored to stderr.
## Dryrun behaviour
`--dryrun` runs the full probe (or reads `--probe-json`), computes all gates, builds
the ready queue, and selects up to `parallel` (or `1`, under `--once`) `(item,
work_type, provider)` triples exactly as the real dispatch loop's first wave would —
**without** simulating gate depletion across a re-probe (a real run only knows that
after actually dispatching; the preview shows the initial wave only, noted as such).
For each selected triple it prints, and executes nothing:
- item or dossier identifier, work type, chosen provider
- resolved prompt template path
- the exact `claude` argv list that would run
- the child env deltas (`CLAUDE_CONFIG_DIR`, `ANTHROPIC_BASE_URL` if set, whether
`ANTHROPIC_API_KEY` would be exported)
- the temp output path and the canonical destination path
No subprocess is run, no file is written, no state is mutated, no lock is required to
be free for the preview to work (the lock is still attempted and its outcome reported,
but a held lock does not block the dryrun preview from computing and printing — real
dispatch mode is what respects the lock as a hard gate). Exit 0.
## State schema
`idle-draft.state.json`, repo root — see §4. Humans never hand-edit this file; the
`mark` subcommand is the only mutation path. Derived state (anything the filesystem
already says) is never written here.
## Config schema
`idle-draft.config.json` — see `data/idle-draft/config.example.json` for the exact
structure. All top-level keys listed in §3 are required.
## Failure classes
| Class | Examples | `attempts` effect | Item outcome |
|---|---|---|---|
| Transient | timeout, HTTP 429/5xx, network reset | not incremented | retried next eligible cycle |
| Content | empty output, missing heading, dead citation path, non-retryable non-zero exit | incremented | `.rejected` kept; `blocked` at `max_attempts` (2) |
## Exit codes
| Code | Meaning |
|---|---|
| 0 | Nothing to do, or all dispatched tasks completed without a fatal problem (individual task failures are logged, not fatal) |
| 1 | Transient/probe problem prevented the run from proceeding (no usable probe data, lock held is exit 0 not 1 — see §1) |
| 2 | Config or state validation failure |
## Edge cases
| Scenario | Handling |
|---|---|
| Lock already held | Log one line, exit 0 (not an error — another cron tick is running) |
| `idle-draft.state.json` missing | Treated as `{"items": {}}`, not an error |
| `idle-draft.state.json` present but invalid | Exit 2, loud message, run never starts |
| Item's `.agent.md` missing | Item excluded from all dispatch (commissioning briefs exist today for every current item; this guards future additions) |
| `seven_day.elapsed_pct` null | Provider ineligible this cycle, never assumed idle |
| Provider `available: false` | Provider ineligible this cycle |
| Credential mismatch | Provider ineligible this cycle, loud log line |
| `research` output cites a path that doesn't exist | Content failure, quarantined as `.rejected`, attempts incremented |
| `max_attempts` reached | Item `blocked`, excluded until `mark unblock` |
| `topic_ideas` content failure | Discarded silently (logged, not blocked — no per-item state key exists for a dossier-level work type) |
| Two work types both ready in the same dossier | Higher stage rank wins (review > draft > research); `topic_ideas` never competes (last-resort only) |
| `--probe-json` combined with `--dryrun` | Composes normally — no subprocess call either way |
| `--once` with no eligible work | Exits 0 immediately, no task dispatched |
| Config missing a required key | Exit 2 |
| `agent-subscriptions` subprocess fails entirely | Exit 1, loud log line, no dispatch attempted |
## Examples
```sh
idle-draft --dryrun # preview one dispatch wave
idle-draft --once # pilot: one real task, then exit
idle-draft --probe-json /tmp/fake-probe.json --dryrun # preview against fixture data
idle-draft --config ~/dev/claude/writing/oreillyconsulting/idle-draft.config.json
idle-draft mark ai/03-shadow-agents-are-the-new-shadow-it edited
idle-draft mark ai/03-shadow-agents-are-the-new-shadow-it sampled
idle-draft status
```

473
tests/test-idle-draft.sh Executable file
View File

@@ -0,0 +1,473 @@
#!/usr/bin/env bash
# Test script for idle-draft: exercises --dryrun / --probe-json against fixture
# dirs, plus direct unit tests of the pure logic functions (gates, stage
# derivation, prioritisation, credential parity, citation validation, state
# validation) by importing the script as a Python module.
#
# No API call, no `claude` invocation, and no real profile dir is ever touched.
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
SCRIPT="$REPO_ROOT/scripts/idle-draft"
RED='\033[31m'; GREEN='\033[32m'; RESET='\033[0m'
PASS=0
FAIL=0
pass() { printf " ${GREEN}PASS${RESET}: %s\n" "$1"; PASS=$((PASS + 1)); }
fail() { printf " ${RED}FAIL${RESET}: %s\n" "$1"; [[ -n "${2:-}" ]] && printf " %s\n" "$2"; FAIL=$((FAIL + 1)); }
assert_contains() {
local output="$1" expected="$2" label="$3"
if echo "$output" | grep -qF -- "$expected"; then pass "$label"; else fail "$label" "Expected to find: $expected"; fi
}
assert_not_contains() {
local output="$1" unexpected="$2" label="$3"
if echo "$output" | grep -qF -- "$unexpected"; then fail "$label" "Did not expect: $unexpected"; else pass "$label"; fi
}
assert_exit_code() {
local actual="$1" expected="$2" label="$3"
if [[ "$actual" -eq "$expected" ]]; then pass "$label"; else fail "$label" "Expected exit $expected, got $actual"; fi
}
TMPDIR=""
cleanup() { [[ -n "$TMPDIR" && -d "$TMPDIR" ]] && rm -rf "$TMPDIR"; }
trap cleanup EXIT
TMPDIR=$(mktemp -d)
FIXTURE_REPO="$TMPDIR/repo"
FIXTURE_CONFIG="$TMPDIR/idle-draft.config.json"
FIXTURE_PROBE="$TMPDIR/probe.json"
PROFILE_ANTHROPIC="$TMPDIR/profiles/anthropic"
PROFILE_MINIMAX="$TMPDIR/profiles/minimax"
# --- Build a fixture writing-repo tree ---------------------------------
mkdir -p "$FIXTURE_REPO/ai" "$FIXTURE_REPO/style" "$PROFILE_ANTHROPIC" "$PROFILE_MINIMAX"
(
cd "$FIXTURE_REPO" && git init -q && git config user.email t@example.com && git config user.name test
)
printf '# AGENTS root\n' > "$FIXTURE_REPO/AGENTS.md"
printf '# ai AGENTS\n' > "$FIXTURE_REPO/ai/AGENTS.md"
printf '# Source register\n' > "$FIXTURE_REPO/ai/SOURCE-REGISTER.md"
printf '# goes target voice\n' > "$FIXTURE_REPO/style/goes-target-voice.md"
printf '# review prompt\n' > "$FIXTURE_REPO/style/review-prompt.md"
# Item 01: only overview+agent -> next = research
printf '# Topic one\n\nPitch.\n' > "$FIXTURE_REPO/ai/01-topic-one.overview.md"
printf '# Commissioning brief: topic one\n' > "$FIXTURE_REPO/ai/01-topic-one.agent.md"
# Item 02: overview+agent+research -> next = draft
printf '# Topic two\n\nPitch.\n' > "$FIXTURE_REPO/ai/02-topic-two.overview.md"
printf '# Commissioning brief: topic two\n' > "$FIXTURE_REPO/ai/02-topic-two.agent.md"
printf '# Research: topic two\n\nEvidence.\n' > "$FIXTURE_REPO/ai/02-topic-two.research.md"
# Item 03: research+draft, human_edit_done -> next = review
printf '# Topic three\n\nPitch.\n' > "$FIXTURE_REPO/ai/03-topic-three.overview.md"
printf '# Commissioning brief: topic three\n' > "$FIXTURE_REPO/ai/03-topic-three.agent.md"
printf '# Research: topic three\n\nEvidence.\n' > "$FIXTURE_REPO/ai/03-topic-three.research.md"
printf '# Topic three draft\n\nBody.\n' > "$FIXTURE_REPO/ai/03-topic-three.draft.md"
cat > "$FIXTURE_REPO/idle-draft.state.json" <<'EOF'
{
"items": {
"ai/03-topic-three": { "human_edit_done": true }
}
}
EOF
(cd "$FIXTURE_REPO" && git add -A && git commit -q -m init)
cat > "$FIXTURE_CONFIG" <<EOF
{
"parallel": 2,
"providers": {
"anthropic": { "profile": "$PROFILE_ANTHROPIC", "threshold_pct": 80, "five_hour_ceiling": 50, "min_idle": 5 },
"minimax": { "profile": "$PROFILE_MINIMAX", "threshold_pct": 80, "five_hour_ceiling": 50, "min_idle": 5 }
},
"work_types": {
"review": { "providers": ["anthropic"] },
"draft": { "providers": ["anthropic"] },
"research": { "providers": ["anthropic", "minimax"] },
"topic_ideas": { "providers": ["anthropic", "minimax"] }
},
"dossiers": ["ai"],
"review_score_threshold": 11,
"max_unreviewed_research_per_dossier": 3,
"max_open_topic_proposals": 6,
"evidence_dirs": ["$TMPDIR/evidence"]
}
EOF
cat > "$FIXTURE_PROBE" <<'EOF'
{
"probed_at": "2026-08-02T00:00:00+00:00",
"providers": [
{"provider": "Anthropic", "available": true, "windows": {
"five_hour": {"utilization_pct": 10.0, "reset_at": null, "reset_in_seconds": 1000, "window_seconds": 18000, "elapsed_pct": 0.5},
"seven_day": {"utilization_pct": 5.0, "reset_at": null, "reset_in_seconds": 100000, "window_seconds": 604800, "elapsed_pct": 0.5}
}},
{"provider": "MiniMax", "available": true, "windows": {
"five_hour": {"utilization_pct": 10.0, "reset_at": null, "reset_in_seconds": 1000, "window_seconds": 18000, "elapsed_pct": 0.5},
"seven_day": {"utilization_pct": 5.0, "reset_at": null, "reset_in_seconds": 100000, "window_seconds": 604800, "elapsed_pct": 0.5}
}}
]
}
EOF
# ============================================================
echo "=== idle-draft: CLI / dryrun tests ==="
echo ""
echo "-- help --"
output=$("$SCRIPT" --help 2>&1); code=$?
assert_exit_code "$code" 0 "--help exits 0"
assert_contains "$output" "Usage:" "--help shows usage"
assert_contains "$output" "--dryrun" "--help mentions --dryrun"
assert_contains "$output" "mark" "--help mentions mark subcommand"
assert_contains "$output" "status" "--help mentions status subcommand"
echo "-- dryrun: highest-priority ready item wins (review > draft > research) --"
output=$("$SCRIPT" --config "$FIXTURE_CONFIG" --repo "$FIXTURE_REPO" --probe-json "$FIXTURE_PROBE" --dryrun --parallel 1 2>&1)
code=$?
assert_exit_code "$code" 0 "dryrun exits 0"
assert_contains "$output" "item: ai/03-topic-three" "dryrun picks item 03 (review-ready, highest stage rank)"
assert_contains "$output" "work_type: review" "dryrun names work_type review"
assert_contains "$output" "provider: anthropic" "dryrun names provider anthropic"
assert_not_contains "$output" "would run" "dryrun does not execute claude"
echo "-- dryrun: exact resolved claude argv for a research item --"
# Force only the research-ready item to be eligible by pointing --repo at a
# single-item fixture (item 01 only).
SOLO="$TMPDIR/solo-repo"
mkdir -p "$SOLO/ai" "$SOLO/style"
cp "$FIXTURE_REPO/ai/01-topic-one.overview.md" "$SOLO/ai/"
cp "$FIXTURE_REPO/ai/01-topic-one.agent.md" "$SOLO/ai/"
cp "$FIXTURE_REPO/AGENTS.md" "$SOLO/AGENTS.md"
cp -r "$FIXTURE_REPO/style" "$SOLO/style"
cp "$FIXTURE_REPO/ai/SOURCE-REGISTER.md" "$SOLO/ai/SOURCE-REGISTER.md"
(cd "$SOLO" && git init -q && git config user.email t@example.com && git config user.name test && git add -A && git commit -q -m init)
output=$("$SCRIPT" --config "$FIXTURE_CONFIG" --repo "$SOLO" --probe-json "$FIXTURE_PROBE" --dryrun --once 2>&1)
assert_contains "$output" "work_type: research" "dryrun (solo fixture) picks research"
assert_contains "$output" "argv: ['claude', '-p', '--max-turns', '25', '--add-dir'" "dryrun prints resolved claude argv with --add-dir for research"
assert_contains "$output" "$TMPDIR/evidence" "dryrun argv includes configured evidence dir"
echo "-- dryrun: no mutation --"
before_hash=$(cd "$FIXTURE_REPO" && git rev-parse HEAD)
"$SCRIPT" --config "$FIXTURE_CONFIG" --repo "$FIXTURE_REPO" --probe-json "$FIXTURE_PROBE" --dryrun >/dev/null 2>&1
after_hash=$(cd "$FIXTURE_REPO" && git rev-parse HEAD)
[[ "$before_hash" == "$after_hash" ]] && pass "dryrun makes no git commits" || fail "dryrun makes no git commits"
[[ ! -f "$FIXTURE_REPO/ai/03-topic-three.review.md" ]] && pass "dryrun writes no output file" || fail "dryrun writes no output file"
echo "-- state validation failure exits 2 --"
BADSTATE="$TMPDIR/badstate-repo"
mkdir -p "$BADSTATE/ai"
cp "$FIXTURE_REPO/ai/01-topic-one.overview.md" "$BADSTATE/ai/"
cp "$FIXTURE_REPO/ai/01-topic-one.agent.md" "$BADSTATE/ai/"
echo '{"items": {"ai/01-topic-one": {"unknown_field": true}}}' > "$BADSTATE/idle-draft.state.json"
output=$("$SCRIPT" --config "$FIXTURE_CONFIG" --repo "$BADSTATE" --probe-json "$FIXTURE_PROBE" --dryrun 2>&1)
code=$?
assert_exit_code "$code" 2 "unknown item field in state exits 2"
assert_contains "$output" "unknown key" "state error names the unknown key"
echo '{"items": {"ai/does-not-exist": {}}}' > "$BADSTATE/idle-draft.state.json"
output=$("$SCRIPT" --config "$FIXTURE_CONFIG" --repo "$BADSTATE" --probe-json "$FIXTURE_PROBE" --dryrun 2>&1)
code=$?
assert_exit_code "$code" 2 "state item not resolving to overview.md exits 2"
echo "-- config validation failure exits 2 --"
BADCONFIG="$TMPDIR/bad.config.json"
echo '{"parallel": 2}' > "$BADCONFIG"
output=$("$SCRIPT" --config "$BADCONFIG" --repo "$FIXTURE_REPO" --probe-json "$FIXTURE_PROBE" --dryrun 2>&1)
code=$?
assert_exit_code "$code" 2 "config missing required keys exits 2"
echo "-- mark subcommand round-trip --"
MARKREPO="$TMPDIR/mark-repo"
mkdir -p "$MARKREPO/ai"
cp "$FIXTURE_REPO/ai/01-topic-one.overview.md" "$MARKREPO/ai/"
cp "$FIXTURE_REPO/ai/01-topic-one.agent.md" "$MARKREPO/ai/"
output=$("$SCRIPT" mark ai/01-topic-one edited --config "$FIXTURE_CONFIG" --repo "$MARKREPO" 2>&1)
code=$?
assert_exit_code "$code" 0 "mark edited exits 0"
assert_contains "$(cat "$MARKREPO/idle-draft.state.json")" '"human_edit_done": true' "mark edited sets human_edit_done"
"$SCRIPT" mark ai/01-topic-one sampled --config "$FIXTURE_CONFIG" --repo "$MARKREPO" >/dev/null 2>&1
assert_contains "$(cat "$MARKREPO/idle-draft.state.json")" '"research_sampled": true' "mark sampled sets research_sampled"
"$SCRIPT" mark ai/01-topic-one approved --config "$FIXTURE_CONFIG" --repo "$MARKREPO" >/dev/null 2>&1
assert_contains "$(cat "$MARKREPO/idle-draft.state.json")" '"approved": true' "mark approved sets approved"
output=$("$SCRIPT" mark ai/99-nope edited --config "$FIXTURE_CONFIG" --repo "$MARKREPO" 2>&1); code=$?
assert_exit_code "$code" 2 "mark on nonexistent item exits 2"
echo "-- status subcommand --"
output=$("$SCRIPT" status --config "$FIXTURE_CONFIG" --repo "$FIXTURE_REPO" --probe-json "$FIXTURE_PROBE" 2>&1)
code=$?
assert_exit_code "$code" 0 "status exits 0"
assert_contains "$output" "ai/01-topic-one" "status lists item 01"
assert_contains "$output" "next=research" "status shows item 01 next=research"
assert_contains "$output" "ai/03-topic-three" "status lists item 03"
assert_contains "$output" "next=review" "status shows item 03 next=review"
assert_contains "$output" "anthropic:" "status shows provider gate line"
echo ""
echo "=== idle-draft: pure-logic unit tests (imported module) ==="
echo ""
PYOUT=$(python3 - "$SCRIPT" "$TMPDIR" <<'PYEOF'
import sys, json, importlib.util, importlib.machinery
from pathlib import Path
script_path, tmpdir = sys.argv[1], Path(sys.argv[2])
loader = importlib.machinery.SourceFileLoader("idle_draft_under_test", script_path)
spec = importlib.util.spec_from_loader(loader.name, loader)
m = importlib.util.module_from_spec(spec)
loader.exec_module(m)
results = []
def check(name, cond, detail=""):
results.append(("PY-PASS" if cond else "PY-FAIL", name, detail))
# --- Gate math ---
pcfg = {"threshold_pct": 80, "min_idle": 5, "five_hour_ceiling": 50}
def probe(elapsed, usage_sd, usage_fh, available=True):
return {
"available": available,
"windows": {
"seven_day": {"elapsed_pct": elapsed, "utilization_pct": usage_sd},
"five_hour": {"utilization_pct": usage_fh},
},
}
g = m.compute_gate(pcfg, probe(0.5, 5.0, 10.0))
check("gate: eligible when idle_points > min_idle and 5h < ceiling",
g["eligible"] is True and abs(g["idle_points"] - 35.0) < 1e-9, g)
g = m.compute_gate(pcfg, probe(0.5, 75.0, 10.0))
check("gate: ineligible when idle_points <= min_idle",
g["eligible"] is False and g["idle_points"] is not None, g)
g = m.compute_gate(pcfg, probe(0.5, 5.0, 60.0))
check("gate: ineligible when five_hour >= ceiling", g["eligible"] is False, g)
g = m.compute_gate(pcfg, probe(None, 5.0, 10.0))
check("gate: null elapsed_pct -> ineligible, idle_points None, fail-closed",
g["eligible"] is False and g["idle_points"] is None and "cannot pace" in g["reason"], g)
g = m.compute_gate(pcfg, probe(0.5, None, 10.0))
check("gate: null seven_day utilization -> ineligible", g["eligible"] is False, g)
g = m.compute_gate(pcfg, probe(0.5, 5.0, None))
check("gate: null five_hour utilization -> ineligible", g["eligible"] is False, g)
g = m.compute_gate(pcfg, {"available": False, "error": "boom"})
check("gate: unavailable provider -> ineligible", g["eligible"] is False and "unavailable" in g["reason"], g)
g = m.compute_gate(pcfg, None)
check("gate: no probe data -> ineligible", g["eligible"] is False, g)
g = m.compute_gate(pcfg, probe(1.0, 5.0, 10.0))
check("gate: elapsed_pct=1.0 (fully elapsed) computes idle_points=threshold-usage",
abs(g["idle_points"] - 75.0) < 1e-9, g)
# --- Stage derivation ---
def files(agent=True, research=False, draft=False, review=False):
return {"overview": True, "agent": agent, "research": research, "draft": draft, "review": review}
wt, reason = m.next_work_type_for_item(files(agent=False), {})
check("stage: missing agent.md -> None", wt is None and "commissioning brief" in reason, reason)
wt, reason = m.next_work_type_for_item(files(agent=True), {})
check("stage: agent only -> research", wt == "research", (wt, reason))
wt, reason = m.next_work_type_for_item(files(agent=True, research=True), {})
check("stage: research done -> draft", wt == "draft", (wt, reason))
wt, reason = m.next_work_type_for_item(files(agent=True, research=True, draft=True), {"human_edit_done": False})
check("stage: draft done, not human-edited -> None (waiting on human)",
wt is None and "human" in reason, (wt, reason))
wt, reason = m.next_work_type_for_item(files(agent=True, research=True, draft=True), {"human_edit_done": True})
check("stage: draft done, human-edited -> review", wt == "review", (wt, reason))
wt, reason = m.next_work_type_for_item(files(agent=True, research=True, draft=True, review=True), {"human_edit_done": True})
check("stage: review.md exists -> None (awaiting human revise/approve)", wt is None, (wt, reason))
wt, reason = m.next_work_type_for_item(files(agent=True), {"blocked": "too many failures"})
check("stage: blocked item -> None regardless of files", wt is None and "blocked" in reason, (wt, reason))
wt, reason = m.next_work_type_for_item(files(agent=True), {"approved": True})
check("stage: approved item -> None (terminal)", wt is None and "approved" in reason, (wt, reason))
# --- Prioritisation / cold-start / topic_ideas via build_ready_queue on the fixture repo ---
repo = tmpdir / "repo"
config = json.loads((tmpdir / "idle-draft.config.json").read_text())
state = m.load_state(repo / "idle-draft.state.json", repo)
queue = m.build_ready_queue(repo, config, state, set())
work_types_in_order = [c["work_type"] for c in queue]
check("priority: review-ready item ranks before draft/research candidates",
work_types_in_order[0] == "review", work_types_in_order)
check("priority: full queue is review, draft, research (stage rank descending)",
work_types_in_order == ["review", "draft", "research"], work_types_in_order)
# Cold-start throttle: two dossiers' worth of unreviewed research under a low cap
throttle_repo = tmpdir / "throttle-repo"
(throttle_repo / "ai").mkdir(parents=True, exist_ok=True)
for n, slug in [("01", "a"), ("02", "b"), ("03", "c")]:
base = throttle_repo / "ai" / f"{n}-{slug}"
(base.with_suffix("")).parent.mkdir(exist_ok=True, parents=True)
(throttle_repo / "ai" / f"{n}-{slug}.overview.md").write_text(f"# {slug}\n")
(throttle_repo / "ai" / f"{n}-{slug}.agent.md").write_text("# brief\n")
(throttle_repo / "ai" / "01-a.research.md").write_text("# Research: a\n")
(throttle_repo / "ai" / "02-b.research.md").write_text("# Research: b\n")
throttle_config = dict(config)
throttle_config["max_unreviewed_research_per_dossier"] = 2
throttle_state = m.default_state()
tqueue = m.build_ready_queue(throttle_repo, throttle_config, throttle_state, set())
research_candidates = [c["slug"] for c in tqueue if c["work_type"] == "research"]
check("cold-start throttle: item 03's research is suppressed once 2 unreviewed research files exist",
"03-c" not in research_candidates, research_candidates)
draft_candidates = [c["slug"] for c in tqueue if c["work_type"] == "draft"]
check("cold-start throttle: draft candidates for 01/02 unaffected by the research throttle",
set(draft_candidates) == {"01-a", "02-b"}, draft_candidates)
# topic_ideas: only offered when nothing else is eligible
idle_repo = tmpdir / "idle-repo"
(idle_repo / "ai").mkdir(parents=True, exist_ok=True)
(idle_repo / "ai" / "01-done.overview.md").write_text("# done\n")
(idle_repo / "ai" / "01-done.agent.md").write_text("# brief\n")
idle_state = {"items": {"ai/01-done": {"approved": True}}}
iqueue = m.build_ready_queue(idle_repo, config, idle_state, set())
check("topic_ideas: offered when no other work is eligible",
len(iqueue) == 1 and iqueue[0]["work_type"] == "topic_ideas", iqueue)
(idle_repo / "ai" / "TOPIC-PROPOSALS.md").write_text("\n".join(f"## Idea {i}" for i in range(6)))
capped_config = dict(config)
capped_config["max_open_topic_proposals"] = 6
iqueue2 = m.build_ready_queue(idle_repo, capped_config, idle_state, set())
check("topic_ideas: suppressed once max_open_topic_proposals reached",
len(iqueue2) == 0, iqueue2)
# --- Credential parity ---
ok, why = m._credential_parity_from_values("secret-abc", "secret-abc")
check("credential parity: matching values -> True", ok is True, why)
ok, why = m._credential_parity_from_values("secret-abc", "secret-xyz")
check("credential parity: mismatched values -> False", ok is False, why)
ok, why = m._credential_parity_from_values("", "secret-abc")
check("credential parity: empty profile value -> False", ok is False, why)
cred_repo = tmpdir / "cred"
cred_repo.mkdir(exist_ok=True)
profile_dir = cred_repo / "profile"
profile_dir.mkdir(exist_ok=True)
key_file = cred_repo / "key.txt"
key_file.write_text("shared-secret-value\n")
(profile_dir / "provider.env").write_text(f"ANTHROPIC_BASE_URL=https://example.invalid\nANTHROPIC_API_KEY_FILE={key_file}\n")
resolved = m.resolve_profile_credential_value("minimax", profile_dir)
check("credential parity: resolves ANTHROPIC_API_KEY_FILE from provider.env", resolved == "shared-secret-value", resolved)
ok, why = m.credential_parity("minimax", profile_dir, probe_credential_fn=lambda: "shared-secret-value")
check("credential parity: full check matches via injected probe reader", ok is True, why)
ok, why = m.credential_parity("minimax", profile_dir, probe_credential_fn=lambda: "different-secret")
check("credential parity: full check flags mismatch via injected probe reader", ok is False, why)
# --- provider.env parsing (replicates claude-profile) ---
parsed = m.parse_provider_env(profile_dir / "provider.env")
check("provider.env: base_url parsed", parsed["base_url"] == "https://example.invalid", parsed)
check("provider.env: api_key_file parsed", parsed["api_key_file"] == str(key_file), parsed)
extra_env_dir = cred_repo / "profile-extra"
extra_env_dir.mkdir(exist_ok=True)
(extra_env_dir / "provider.env").write_text("MODEL_ID=claude-haiku-4-5-20251001\nSOME_EXTRA_VAR=hello\n")
parsed2 = m.parse_provider_env(extra_env_dir / "provider.env")
check("provider.env: MODEL_ID recognised", parsed2["model_id"] == "claude-haiku-4-5-20251001", parsed2)
check("provider.env: unrecognised key falls into extra", parsed2["extra"].get("SOME_EXTRA_VAR") == "hello", parsed2)
env = m.build_child_env(profile_dir, base_env={})
check("build_child_env: CLAUDE_CONFIG_DIR set", env["CLAUDE_CONFIG_DIR"] == str(profile_dir), env)
check("build_child_env: ANTHROPIC_BASE_URL exported from provider.env", env.get("ANTHROPIC_BASE_URL") == "https://example.invalid", env)
check("build_child_env: ANTHROPIC_API_KEY read from key file", env.get("ANTHROPIC_API_KEY") == "shared-secret-value", env)
no_provider_env_dir = cred_repo / "profile-plain"
no_provider_env_dir.mkdir(exist_ok=True)
env2 = m.build_child_env(no_provider_env_dir, base_env={})
check("build_child_env: profile without provider.env gets only CLAUDE_CONFIG_DIR",
"ANTHROPIC_BASE_URL" not in env2 and "ANTHROPIC_API_KEY" not in env2 and env2["CLAUDE_CONFIG_DIR"] == str(no_provider_env_dir),
env2)
# --- Citation validation ---
real_path = cred_repo / "exists.txt"
real_path.write_text("x")
good_text = f"# Research: topic\n\nSee {real_path} for detail.\n"
ok, why = m.validate_output("research", good_text)
check("citation validation: accepts research output with an existing cited path", ok is True, why)
bad_text = "# Research: topic\n\nSee /home/nonexistent-user/definitely-not-here.md for detail.\n"
ok, why = m.validate_output("research", bad_text)
check("citation validation: rejects research output citing a dead path", ok is False, why)
ok, why = m.validate_output("research", "")
check("citation validation: rejects empty output", ok is False, why)
ok, why = m.validate_output("research", "no heading here\njust text\n")
check("citation validation: rejects output missing a top-level heading", ok is False, why)
ok, why = m.validate_output("topic_ideas", "## A proposal\n\nBody.\n")
check("citation validation: topic_ideas accepts ## as its top-level heading", ok is True, why)
ok, why = m.validate_output("topic_ideas", "# Wrong heading level\n")
check("citation validation: topic_ideas rejects a single # heading", ok is False, why)
# --- State validation ---
try:
m.validate_state({"items": {}, "bogus": 1}, repo)
check("state validation: unknown top-level key raises", False)
except m.StateValidationError:
check("state validation: unknown top-level key raises", True)
try:
m.validate_state({"items": {"ai/does-not-exist": {}}}, repo)
check("state validation: item not resolving to overview.md raises", False)
except m.StateValidationError:
check("state validation: item not resolving to overview.md raises", True)
try:
m.validate_state({"items": {"ai/01-topic-one": {"attempts": {"bogus_type": 1}}}}, repo)
check("state validation: unknown attempts work_type raises", False)
except m.StateValidationError:
check("state validation: unknown attempts work_type raises", True)
try:
m.validate_state({"items": {"ai/01-topic-one": {"human_edit_done": "yes"}}}, repo)
check("state validation: wrong-typed bool field raises", False)
except m.StateValidationError:
check("state validation: wrong-typed bool field raises", True)
# Valid state should not raise
m.validate_state({"items": {"ai/01-topic-one": {"human_edit_done": True, "attempts": {"draft": 1}}}}, repo)
check("state validation: well-formed state passes", True)
for status, name, detail in results:
print(f"{status}: {name}", detail if detail else "")
n_fail = sum(1 for s, _, _ in results if s == "PY-FAIL")
sys.exit(1 if n_fail else 0)
PYEOF
)
PY_EXIT=$?
echo "$PYOUT"
py_pass=$(echo "$PYOUT" | grep -c '^PY-PASS' || true)
py_fail=$(echo "$PYOUT" | grep -c '^PY-FAIL' || true)
PASS=$((PASS + py_pass))
FAIL=$((FAIL + py_fail))
echo ""
echo "Results: $PASS passed, $FAIL failed"
if [[ $FAIL -gt 0 ]]; then
exit 1
fi
printf "${GREEN}All tests passed.${RESET}\n"