idle-draft: process-group child termination, unique tmp paths, stale sweep

Lock-contention test added; children die with the dispatcher (SIGTERM/
SIGINT handlers + atexit); tmp files are per-invocation (.tmp.<pid>) so
a SIGKILL-orphaned child can never collide with a new instance.

Claude-Session: https://claude.ai/code/session_01YQDoWNM7XPPii28khFWoMc

specs/idle-draft.spec.md

@@ -65,6 +65,14 @@ lock), `flock` (`fcntl.flock`, `LOCK_EX | LOCK_NB`) on `<repo>/.idle-draft.lock`
 already locked by another invocation, log one line and exit **0** — a concurrent cron
 tick is not a failure.
 
+**Guarantee:** the lock is acquired before any other work and held for the *entire*
+dispatch loop, including while dispatched children are running — not released until the
+process is about to exit. A second cron-fired invocation therefore always exits 0
+immediately for as long as the first is alive, however long that is (a low-usage
+provider can let one instance run for hours "catching up"; the lock does not expire or
+time out). This is the primary concurrency guard; §13 covers what happens when the
+*lock-holding process itself* dies uncleanly rather than exiting normally.
+
 ### 2. Log rotation
 
 At startup, if `<repo>/idle-draft.log` exceeds 5 MB, rename it to `idle-draft.log.1`
@@ -216,6 +224,11 @@ For the selected `(item_or_dossier, work_type, provider)`:
 2. Render the template (`string.Template`, `$placeholder` substitution) with the
    resolved paths for that item (overview, agent, research, draft as applicable),
    dossier name, slug, style directory, source register path, and a temp output path.
+   The temp output path is `<canonical>.tmp.<dispatcher-pid>` — unique per dispatcher
+   *invocation* (all tasks within one dispatch run share the same suffix, since it's
+   this process's own pid), not per task. This is what makes tmp paths collision-proof
+   across dispatcher instances even when one instance dies leaving orphaned children
+   behind — see §13.
 3. Resolve the profile directory from `config["providers"][provider]["profile"]`
    (`~` expanded). Build the child environment: `CLAUDE_CONFIG_DIR=<profile>`, plus —
    if `<profile>/provider.env` exists — `ANTHROPIC_BASE_URL` (if set),
@@ -237,7 +250,11 @@ For the selected `(item_or_dossier, work_type, provider)`:
 5. Run the subprocess: `cwd=<repo>`, `env=<built env>`, prompt piped via **stdin**
    (not as an argv element — avoids `ARG_MAX` on large rendered prompts, same lesson
    `claude-profile` already applies to its system-prompt injection), timeout = per-task
-   timeout (1800s unless overridden).
+   timeout (1800s unless overridden). The child is started in its own process group
+   (`start_new_session=True`) and its pid is tracked in a live-children registry for the
+   duration of the call — see §13. On timeout, the whole process group is sent
+   `SIGKILL` (not just the immediate `claude` process), so any grandchild it spawned
+   dies too, before the task is classified as a transient failure.
 6. Classify the result:
    - **Timeout** → transient failure. Do not increment `attempts`.
    - **Non-zero exit** whose stderr matches a retryable signature (`429`, `5xx`,
@@ -265,7 +282,10 @@ For the selected `(item_or_dossier, work_type, provider)`:
    `<repo>/<dossier>/<NN-slug>.<work_type>.md.rejected` (kept for human inspection,
    never promoted, never committed to the canonical name; `topic_ideas` content
    failures are simply discarded — nothing is appended, nothing is blocked, since
-   `topic_ideas` has no per-item state entry to carry an attempt counter).
+   `topic_ideas` has no per-item state entry to carry an attempt counter). The
+   `.rejected` path is derived from the canonical path, not the tmp path, so it is
+   unaffected by the `.tmp.<dispatcher-pid>` suffix (§13) — always
+   `<canonical>.rejected`, never `<canonical>.rejected.<pid>`.
 10. On transient failure: temp file is discarded; no state change; no commit; the
     candidate may be retried on a later cycle.
 11. On success: write `idle-draft.state.json` atomically (temp file + `os.replace()` in
@@ -284,6 +304,53 @@ For the selected `(item_or_dossier, work_type, provider)`:
 One line per event (gate decision, dispatch, completion, failure, commit) appended to
 `<repo>/idle-draft.log` with an ISO-8601 timestamp, mirrored to stderr.
 
+### 13. Concurrency protection beyond the lock
+
+§1's flock is the primary guard and handles the common case: a second cron-fired
+invocation always sees the lock held and exits 0. This section covers the gap the lock
+alone cannot close — a dispatcher process that dies *without* releasing its children
+cleanly (`SIGTERM`, `SIGKILL`, OOM kill, machine reboot). The flock itself is released
+the instant the holding process's file descriptors close (on any death, clean or not),
+but a `claude` child spawned via `subprocess` is a separate OS process and does **not**
+die automatically when its parent does — left alone it becomes an orphan, reparented to
+init, still writing its output. A fresh cron-fired instance starting immediately after
+would then see the lock free, pick up the *same* ready item, and dispatch a *second*
+`claude` process against the *same* deterministic output path — two writers racing on
+the "same" canonical file. Two independent mechanisms close this:
+
+**a. Children die with the dispatcher (the common death path).** Every `claude` child is
+spawned with `start_new_session=True`, putting it (and any process it forks) in its own
+process group with the child's pid as the group id. The dispatcher tracks each live
+child's pid in an in-process registry for the duration of the call (`register_child` /
+`unregister_child`, guarded by a lock since tasks run in worker threads). On `SIGTERM` or
+`SIGINT`, an installed handler walks the registry and `os.killpg`s each group with
+`SIGKILL`, then exits — no child outlives a dispatcher that receives a catchable signal.
+The same cleanup also runs via `atexit`, so a normal unhandled-exception exit (no signal
+involved) still cannot leave children behind. **This cannot catch `SIGKILL` of the
+dispatcher itself** — no userspace handler can — which is exactly the case mechanism (b)
+exists for.
+
+**b. Tmp-path collisions are structurally impossible (the uncatchable-death path).** Each
+task's temp output file is named `<canonical>.tmp.<dispatcher-pid>` (§11 step 2), not
+the old bare `<canonical>.tmp`. Every dispatcher process has a distinct pid, so even an
+orphaned child that outlives a `SIGKILL`ed dispatcher is writing to a filename no other
+dispatcher instance — past, present, or future — will ever target. There is no race to
+resolve: the collision that used to be structurally possible (two writers, one path) is
+now structurally impossible (two writers, two paths).
+
+**Startup sweep.** After acquiring the lock, before touching config-driven state, a
+fresh dispatch instance globs each configured dossier directory for leftover
+`*.tmp.<pid>` files. Any whose mtime is older than the per-task timeout (1800s unless
+overridden) is deleted and logged as swept — it is orphaned litter from a dispatcher
+that died mid-task; since it was never promoted (only the exact `run_task()` call that
+created it ever calls `os.replace()` on it) it can never silently become the canonical
+file no matter how long it sits there. Files younger than the timeout are left alone —
+they may belong to a live orphan (mechanism (a) failed to reap it, e.g. it was itself
+`SIGKILL`ed independently, or belongs to another concurrent dispatch cycle that legitimately
+still has it in flight) still finishing up; touching them risks nothing since, again,
+nothing ever silently adopts a tmp file it didn't create — at worst a stale one sits
+unswept until the *next* startup sweep, once it ages past the timeout.
+
 ## Dryrun behaviour
 
 `--dryrun` runs the full probe (or reads `--probe-json`), computes all gates, builds
@@ -346,7 +413,9 @@ a lighter-touch check); `draft` and `review` carry no `allowed_tools` key.
 
 | Scenario | Handling |
 |---|---|
-| Lock already held | Log one line, exit 0 (not an error — another cron tick is running) |
+| Lock already held | Log one line, exit 0 (not an error — another cron tick is running), regardless of how long the holder has been running |
+| Dispatcher receives `SIGTERM`/`SIGINT` while children are running | Handler `os.killpg`s every live child's process group, then exits (§13a) |
+| Dispatcher is `SIGKILL`ed (uncatchable) | Children orphaned, but each was writing to a `.tmp.<dispatcher-pid>` path unique to that dead instance — no live instance can ever collide with it; stale ones swept once they age past the task timeout (§13b) |
 | `idle-draft.state.json` missing | Treated as `{"items": {}}`, not an error |
 | `idle-draft.state.json` present but invalid | Exit 2, loud message, run never starts |
 | Item's `.agent.md` missing | Item excluded from all dispatch (commissioning briefs exist today for every current item; this guards future additions) |