skynet/agent-runtime-framework
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

init: seed framework reference content from agent-runtimes main repo

Browse Source
This commit is contained in:
Paul O'Reilly
2026-04-26 12:17:42 +12:00
commit 37a5165dfb
118 changed files with 6831 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
harnesses/contexts/planning/v1/CLAUDE.md Normal file
View File

@@ -0,0 +1,47 @@
# Planning Agent Context
You are a planning and specification agent. Your job is to produce high-quality design documents — specs, plans, reviews, or interview questions — not to write code.
## Best Practices
Read these before starting any planning task. They are mounted at `/opt/harness/context/best-practices/`:
| File | When to read |
|---|---|
| `spec-driven-development.md` | Always — spec structure, requirement numbering, scenarios |
| `test-driven-development.md` | Always — testability, edge cases, property-based testing |
| `security-architecture.md` | Always — server boundary rule, defense in depth, auth patterns |
| `llm-code-security.md` | Always — injection flaws, input validation, OWASP for AI code |
| `api-design.md` | When the spec involves HTTP APIs |
| `database-selection.md` | When the spec involves data persistence |
| `kubernetes.md` | When the spec involves K8s resources |
| `docker.md` | When the spec involves containers |
| `secrets-management.md` | When the spec involves credential handling |
Read at minimum the four "Always" files. Read others based on the task domain.
## Output Conventions
- **If `/workspace/working/` exists** (agent-repo mode): write output files directly into the working directory (e.g., `/workspace/working/spec/auth.md`). Edit existing files in place. Your changes will be auto-committed and pushed by the finalize script.
- **If `/workspace/working/` does not exist**: write output to `/workspace/.agent-output/output.md`
- Use structured markdown with clear section headings
- Number all requirements with a prefix (e.g., `WF-1`, `AU-1`) — each must be independently testable
- Every requirement needs a "Why:" rationale
- Every requirement needs at least one given/when/then scenario
- Be opinionated — make concrete decisions with rationale, don't hedge
- Call out trade-offs explicitly
- Flag security implications for every external-facing interface
## What NOT to Do
- Do not write code, scripts, or implementation
- Do not install packages or modify the environment
- Do not make network requests except to read mounted context files
- Do not leave requirements vague ("handle errors appropriately") — be specific ("return HTTP 413 with error body including field name and size limit")
## Session Logging
Write a brief session log to `/workspace/.agent-output/session-log.md` (or `/workspace/working/memory/log/` in agent-repo mode) with:
- **Summary**: What was produced
- **Key Decisions**: Design choices made and rationale
- **Open Questions**: Anything that needs human input

463
harnesses/contexts/planning/v1/best-practices/api-design.md Normal file
View File

@@ -0,0 +1,463 @@
# API Design
Best practices for REST/HTTP APIs in internal microservices and platform services. Focused on practical defaults -- not aspirational ideals. Sourced from OWASP API Security Top 10 (2023), RFC 9700 (OAuth 2.0 Security BCP, January 2025), Google AIP, and production experience.
Cross-references: [Security Architecture](security-architecture.md) covers the server boundary rule and proxy patterns. [Secrets Management](secrets-management.md) covers credential storage and rotation.
---
## 1. Transport Security
### 1.1 HTTPS everywhere, no exceptions
**Principle:** Every API endpoint -- internal or external -- must serve over TLS. Plaintext HTTP must not be available, even on internal networks.
**Why it matters:** Without TLS, any network hop (load balancer, sidecar, switch) can observe or modify traffic. Internal networks are not trusted in a zero-trust model -- a compromised pod can sniff adjacent traffic.
**How to implement:**
- Terminate TLS at the ingress controller (e.g., Traefik, NGINX) with certificates from cert-manager / Let's Encrypt.
- For service-to-service within the cluster, use a service mesh (Istio, Linkerd) or cert-manager CSI driver to issue per-pod certificates.
- Set `Strict-Transport-Security` headers on all responses.
- Redirect HTTP to HTTPS at the ingress layer.
**Anti-patterns:**
- "Internal traffic doesn't need encryption" -- it does under zero-trust.
- Self-signed certificates with verification disabled (`--insecure`, `verify=False`) -- defeats the purpose of TLS.
- Long-lived certificates (years) with no rotation -- use short-lived certs (days to weeks) with automated renewal.
### 1.2 mTLS between services
**Principle:** Service-to-service communication must use mutual TLS -- both sides present and verify certificates.
**Why it matters:** Server-only TLS authenticates the server to the client, but any client can connect. mTLS ensures both parties have a cryptographically verified identity, which is the foundation of zero-trust networking.
**How to implement:**
- Service mesh (Istio strict mode, Linkerd) handles mTLS transparently via sidecar proxies -- no application code changes.
- Use SPIFFE/SPIRE for standardized workload identity (SVID certificates).
- Default certificate lifetime should be short (24 hours) with automatic rotation.
- Start in permissive mode (allow both plain and mTLS), migrate to strict mode once all services are enrolled.
**Anti-patterns:**
- Permissive mode as a permanent state -- it must be a migration step, not the end state.
- Disabling mTLS verification for "debugging" and forgetting to re-enable it.
- Using a single shared certificate for all services -- each workload needs its own identity.
### 1.3 Certificate management
**Principle:** Certificate issuance and rotation must be fully automated. No manual certificate management in production.
**Why it matters:** Manual certificate management leads to expired certificates, which cause outages. It also leads to long-lived certificates, which increase blast radius if compromised.
**How to implement:**
- cert-manager in Kubernetes with ClusterIssuer for ingress certificates.
- Service mesh control plane for workload certificates (Istio Citadel, Linkerd identity).
- Monitor certificate expiry with alerts at 30/14/7 days before expiry.
- Store CA keys in HSM or sealed secrets -- never in plaintext ConfigMaps.
**Anti-patterns:**
- Certificates stored in Git repos (even encrypted, they need rotation).
- Wildcard certificates shared across trust boundaries.
- No monitoring for certificate expiry -- silent failures at 3am.
---
## 2. Authentication and Authorization
### 2.1 OIDC/OAuth2 for user-facing APIs (RFC 9700)
**Principle:** Use OAuth 2.0 Authorization Code flow with PKCE for all client types. The implicit flow and resource owner password credentials flow are deprecated per RFC 9700 (January 2025).
**Why it matters:** The implicit flow exposes access tokens in URLs and browser history. The password grant requires users to share credentials directly with the client, bypassing centralized identity providers.
**How to implement:**
- Authorization Code + PKCE for all clients (web, mobile, CLI). PKCE is now mandatory for all client types, not just public clients.
- Use `S256` challenge method (not `plain`).
- Tokens issued by the authorization server, validated by the resource server.
- Use Authorization Server Metadata (RFC 8414) for automatic discovery of endpoints and supported features.
**Anti-patterns:**
- Implicit flow (`response_type=token`) -- deprecated by RFC 9700.
- Resource Owner Password Credentials flow -- deprecated by RFC 9700.
- Storing tokens in localStorage (accessible to XSS) -- use httpOnly cookies or in-memory storage with refresh token rotation.
- Long-lived access tokens without refresh -- use short-lived access tokens (5-15 minutes) with refresh token rotation.
### 2.2 JWT best practices
**Principle:** JWTs must be validated completely on every request -- signature, expiry, issuer, audience, and algorithm.
**Why it matters:** Incomplete JWT validation is a top attack vector. Accepting expired tokens, wrong audiences, or `alg: none` enables token forgery and replay.
**How to implement:**
- Validate: signature (asymmetric preferred -- RS256/ES256), `exp`, `iat`, `iss`, `aud`, `nbf`.
- Use asymmetric signing (RS256/ES256) so that only the auth server holds the private key. Resource servers only need the public key.
- Set `aud` claim to the specific API audience -- reject tokens intended for other services.
- Keep tokens small -- put only identity and authorization claims in the token, fetch additional data from a userinfo endpoint.
- Use `jti` (JWT ID) claim for token revocation checks when needed.
**Anti-patterns:**
- Accepting `alg: none` or allowing algorithm switching -- pin the expected algorithm server-side.
- Not validating `aud` -- allows tokens from one service to be replayed against another.
- Symmetric signing (HS256) with a shared secret across services -- if one service is compromised, all are.
- Treating JWTs as sessions -- JWTs are not revocable by default. Combine with short expiry and token introspection for revocation.
### 2.3 Service-to-service authentication
**Principle:** Services authenticate to each other using mTLS identities (SPIFFE) or short-lived JWTs from a token exchange. Never shared static API keys.
**Why it matters:** Shared API keys have no expiry, no rotation path, no per-service identity, and no audit trail. If one service is compromised, the key works for everything.
**How to implement:**
- **Preferred: mTLS with SPIFFE.** The service mesh provides identity automatically. Authorization policies reference SPIFFE IDs (e.g., `spiffe://cluster.local/ns/payments/sa/payment-svc`).
- **Alternative: OAuth2 Client Credentials flow.** Each service has its own `client_id` and `client_secret` (or asymmetric key pair). Tokens are short-lived and scoped to specific audiences.
- Use asymmetric client authentication (private_key_jwt per RFC 7523) rather than client secrets where possible.
- Implement audience restriction -- tokens minted for service A must not be accepted by service B.
**Anti-patterns:**
- Shared static API keys passed in headers or query strings.
- One "admin" service account used by all services.
- Service-to-service tokens with no audience claim -- replayable across any internal API.
- Bearer tokens without mTLS -- if the network is compromised, the token can be stolen and replayed from anywhere.
### 2.4 Authorization: object-level and function-level
**Principle:** Check authorization at every API endpoint, for every object access, based on the authenticated identity. Never rely on "the client won't send that request."
**Why it matters:** Broken Object-Level Authorization (BOLA) is the #1 risk in the OWASP API Security Top 10. Broken Function-Level Authorization is #5. These are the most common API vulnerabilities found in penetration tests.
**How to implement:**
- Every endpoint that accesses a specific resource must verify the caller owns or has access to that resource.
- Use middleware/decorators that enforce authorization before the handler runs.
- Use random UUIDs for resource identifiers, not sequential integers (which are trivially enumerable).
- Separate authorization for data access (BOLA) and function access (admin endpoints, bulk operations).
- Automated tests that verify: user A cannot access user B's resources, non-admin cannot call admin endpoints.
**Anti-patterns:**
- Authorization only at the API gateway -- must also be enforced at the service level.
- Relying on obscurity of endpoint URLs for access control.
- Sequential/predictable resource IDs without authorization checks.
- Missing authorization on secondary endpoints (e.g., `/users/{id}/orders` checks user but not order ownership).
---
## 3. API Design Patterns
### 3.1 Versioning
**Principle:** Version your API from day one using URL path versioning (`/v1/`). Support at most two versions simultaneously.
**Why it matters:** Breaking changes without versioning cause cascading failures across all consumers simultaneously. Supporting too many versions creates maintenance burden and security risk (old versions may lack patches).
**How to implement:**
- URL path: `/api/v1/resources` -- simple, visible, cacheable.
- Deprecation policy: announce deprecation in response headers (`Deprecation: true`, `Sunset: <date>`).
- Maximum two active versions. When v3 launches, v1 is removed.
- Internal services can use header-based versioning (`Accept: application/vnd.myapi.v2+json`) if URL versioning is too rigid for rapid iteration.
**Anti-patterns:**
- No versioning ("we'll be careful") -- you will break consumers.
- Unlimited version support -- v1 through v7 all still running, each with different bugs.
- Breaking changes in a patch version.
- Versioning individual endpoints instead of the whole API surface.
### 3.2 Pagination
**Principle:** All list endpoints must paginate. Use cursor-based pagination for real-time data; offset-based for stable datasets.
**Why it matters:** Unbounded list responses cause memory exhaustion, slow responses, and database strain. Large offset values cause full table scans.
**How to implement:**
- **Cursor-based (preferred):** Return an opaque `next_cursor` token. Client passes it to get the next page. Stable under concurrent writes.
```json
{ "data": [...], "next_cursor": "abc123", "has_more": true }
```
- **Offset-based (simple datasets):** `?limit=50&offset=100`. Acceptable for admin dashboards or infrequently changing data.
- Set a maximum page size (e.g., 100) enforced server-side. Ignore client requests for larger pages.
- Always return pagination metadata (`next_cursor`, `has_more`, or `total_count` if cheap to compute).
**Anti-patterns:**
- No pagination on list endpoints -- returns 50,000 records in one response.
- Offset-based pagination on large, frequently-changing datasets -- pages shift as records are inserted/deleted.
- Client-controlled page size with no server-side maximum.
- `total_count` requiring a full table scan on every request -- make it optional or cached.
### 3.3 Error handling
**Principle:** Return structured, machine-readable errors with stable error codes, human-readable messages, and consistent shape across all endpoints.
**Why it matters:** Inconsistent error formats force every consumer to write custom parsing logic. Missing error codes make automated retry decisions impossible. Leaking stack traces exposes internals to attackers.
**How to implement:**
- Standard error envelope:
```json
{
"error": {
"code": "RESOURCE_NOT_FOUND",
"message": "Order 7f3a... not found",
"details": [{ "field": "order_id", "reason": "not_found" }]
}
}
```
- Use HTTP status codes correctly: 400 (bad input), 401 (unauthenticated), 403 (unauthorized), 404 (not found), 409 (conflict), 422 (validation), 429 (rate limited), 500 (server error).
- Error codes are stable strings (not integers) that consumers can switch on.
- Never expose stack traces, SQL errors, or internal paths in error responses.
- Log the full error server-side with a correlation ID. Return only the correlation ID to the client.
**Anti-patterns:**
- 200 OK with `{"success": false}` -- use HTTP status codes.
- Returning raw database errors ("duplicate key violates unique constraint on...").
- Different error shapes from different endpoints in the same API.
- Generic "Internal Server Error" with no correlation ID -- impossible to debug.
### 3.4 Idempotency
**Principle:** All state-changing operations must be safe to retry. Use idempotency keys for POST requests; PUT and DELETE are idempotent by definition.
**Why it matters:** Network failures, timeouts, and retries are normal in distributed systems. Without idempotency, retried requests create duplicate orders, double payments, or inconsistent state.
**How to implement:**
- Accept `Idempotency-Key` header (IETF draft: draft-ietf-httpapi-idempotency-key-header) on POST endpoints.
- Server stores the response for a given key (TTL 24-48 hours). Duplicate requests return the stored response.
- Use UUIDv4 for idempotency keys -- never sequential or timestamp-based (predictable/guessable).
- Handle concurrent duplicate requests with locking: first request processes, subsequent requests wait then return cached response.
- PUT must be truly idempotent: same request, same result, no side effects on repeat.
**Anti-patterns:**
- POST endpoints with no idempotency support -- every retry creates a duplicate.
- Idempotency keys stored forever (memory leak) or for too short a period (retries after expiry create duplicates).
- Client-generated sequential keys (integers, timestamps) -- guessable and exploitable.
- "Idempotent" endpoints that still send duplicate emails/webhooks on retry.
### 3.5 Rate limiting
**Principle:** Every API must enforce rate limits. Return standard headers so clients can self-throttle.
**Why it matters:** Without rate limits, a single misbehaving client (or attacker) can exhaust resources for all consumers. Rate limits also protect downstream dependencies.
**How to implement:**
- Token bucket or sliding window algorithm (token bucket is simplest with good burst handling).
- Return headers: `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset` (IETF draft still pending; `X-` prefix remains de facto standard).
- Return `429 Too Many Requests` with `Retry-After` header (RFC 6585).
- Rate limit checks execute before expensive operations (auth, database queries).
- For distributed deployments, use Redis with atomic Lua scripts for counter operations -- avoid race conditions.
- Different tiers for different consumers (internal services get higher limits than external clients).
**Anti-patterns:**
- No rate limiting ("it's an internal API") -- a runaway loop in one service takes down the whole platform.
- Rate limiting after expensive operations (database query runs, then rate limit rejects the response).
- No `Retry-After` header -- clients retry immediately in a tight loop, making the problem worse.
- Per-IP rate limiting only -- bypassed by distributed clients, unfair to NAT'd users.
---
## 4. Input Validation
### 4.1 Schema validation at the edge
**Principle:** Validate all request bodies against a schema (OpenAPI/JSON Schema) at the API gateway or middleware layer. Reject requests that don't conform before they reach business logic.
**Why it matters:** Invalid input that reaches business logic causes unpredictable behavior -- crashes, data corruption, injection attacks. Edge validation is the first line of defense.
**How to implement:**
- Define request/response schemas in OpenAPI 3.x. Generate validation middleware from the spec.
- Reject unknown fields (additionalProperties: false) -- attackers probe via unexpected fields.
- Enforce type constraints: string lengths, integer ranges, enum values, date formats.
- Validate `Content-Type` header -- reject requests with unexpected content types (e.g., reject `multipart/form-data` on a JSON-only endpoint).
**Anti-patterns:**
- Validation only in business logic, not at the edge -- invalid data traverses the full call stack before rejection.
- Accepting and silently ignoring unknown fields -- hides bugs and enables mass assignment attacks.
- Validating types but not ranges -- accepting an `age` field of 99999 or -1.
- No schema at all ("we'll validate manually") -- inconsistent validation across endpoints.
### 4.2 Injection prevention
**Principle:** Use parameterized queries for all database access. Never concatenate user input into queries, commands, or templates.
**Why it matters:** SQL injection remains in the OWASP Top 10 after 20+ years. NoSQL injection, LDAP injection, and command injection follow the same pattern -- unsanitized input in a query language.
**How to implement:**
- Use an ORM (SQLAlchemy, Prisma, TypeORM) or parameterized queries. All major ORMs parameterize by default.
- For raw SQL (performance-critical paths), use prepared statements exclusively.
- Validate input with allowlists, not denylists. If a field should be a UUID, validate it's a UUID -- don't try to strip "malicious characters."
- For template rendering, use auto-escaping (Jinja2 autoescape, React JSX auto-escaping).
**Anti-patterns:**
- String concatenation in SQL: `f"SELECT * FROM users WHERE id = '{user_input}'"`.
- Denylisting dangerous characters instead of allowlisting valid patterns.
- Trusting input from "internal" services -- a compromised upstream service sends malicious data.
- Disabling ORM parameterization for "performance" without understanding the security cost.
### 4.3 Request size and depth limits
**Principle:** Enforce maximum request body size, JSON nesting depth, and array length at the gateway level.
**Why it matters:** Deeply nested JSON or extremely large payloads cause CPU exhaustion during parsing (hash collision attacks, recursive descent parsers). This is a denial-of-service vector.
**How to implement:**
- Set maximum body size at the reverse proxy/ingress (e.g., `client_max_body_size 1m` in NGINX).
- Limit JSON nesting depth (8-16 levels is generous for any real use case).
- Limit array sizes in request bodies (e.g., batch endpoints accept max 100 items).
- Set request timeouts at the gateway -- don't let slow clients hold connections open.
**Anti-patterns:**
- No body size limit -- 100MB JSON payload parsed by every middleware layer.
- Accepting arbitrarily nested JSON -- `{"a":{"a":{"a":...}}}` 1000 levels deep.
- Batch endpoints with no limit -- client sends 1 million items in one request.
---
## 5. Secrets in APIs
### 5.1 Never in URLs or query parameters
**Principle:** Authentication tokens, API keys, and any secret material must be sent in headers (Authorization, custom headers) or request bodies. Never in URLs or query parameters.
**Why it matters:** URLs are logged everywhere -- web server access logs, proxy logs, browser history, referrer headers, CDN logs, monitoring tools. A token in a URL is a token in every log file in the request path.
**How to implement:**
- Use `Authorization: Bearer <token>` header for all token-based auth.
- For webhook signatures, use a signature header (e.g., `X-Hub-Signature-256`).
- If an API currently accepts tokens in query params, deprecate that path and migrate to header-based auth.
- Configure log scrubbing to redact Authorization headers, but don't rely on it as the primary control.
**Anti-patterns:**
- `GET /api/resources?api_key=sk_live_abc123` -- key in every access log.
- OAuth redirect URIs with tokens in query params (use `response_mode=form_post` or authorization code flow).
- Webhook URLs with embedded secrets (`/webhook?secret=abc`) -- logged, cached, shared.
### 5.2 Token rotation and expiry
**Principle:** All tokens and API keys must have expiry dates and a documented rotation procedure. No permanent credentials.
**Why it matters:** Leaked tokens without expiry are valid forever. Rotation limits blast radius -- even if a token is compromised, it expires soon.
**How to implement:**
- Access tokens: 5-15 minute expiry, refreshed via refresh token.
- Refresh tokens: rotate on use (each refresh issues a new refresh token and invalidates the old one).
- API keys for external integrations: 90-day rotation policy with overlap period (new key valid before old key expires).
- Service account tokens (OAuth2 client credentials): short-lived (1 hour), fetched on demand.
- Track expiry dates in a credential inventory. Alert before expiry (see [Secrets Management](secrets-management.md) -- Credential Lifecycle Management).
**Anti-patterns:**
- API keys that never expire ("we'll rotate them when we need to" -- you won't).
- Refresh tokens that don't rotate -- stolen refresh token provides permanent access.
- No overlap period during rotation -- brief outage while all consumers update.
- Hardcoded tokens in application config deployed via CI -- rotation requires a full redeploy.
### 5.3 No secrets in logs or error responses
**Principle:** Scrub all secrets from logs, error responses, and monitoring data. Structured logging with explicit field selection is safer than serializing request objects.
**Why it matters:** Log aggregation systems (ELK, Loki, Datadog) are often accessible to broader teams than production systems. A token in a log entry has a much wider exposure surface than a token in a running process.
**How to implement:**
- Use structured logging. Log specific fields, not entire request objects.
- Redact `Authorization` headers and any field matching `token`, `password`, `secret`, `key` patterns in log middleware.
- Never log request bodies for auth endpoints (login, token exchange).
- Error responses must not include internal state -- return a correlation ID and log details server-side.
**Anti-patterns:**
- `logger.info(f"Request: {request.headers}")` -- logs all headers including Authorization.
- Error responses that include the original request (including auth headers) for "debugging convenience."
- Logging full webhook payloads that contain signing secrets in custom headers.
---
## 6. Service Mesh and Zero Trust
### 6.1 Default deny with explicit allow
**Principle:** Network policies and authorization policies must default to deny-all. Every allowed communication path is explicitly defined.
**Why it matters:** Default-allow means a compromised service can reach every other service in the cluster. Default-deny contains the blast radius to only the services the compromised workload was authorized to reach.
**How to implement:**
- Kubernetes NetworkPolicy: deploy a default-deny policy in every namespace, then add specific allow rules.
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
```
- Service mesh authorization policies: deny by default, allow specific source-to-destination pairs by SPIFFE ID.
- Audit policies periodically -- remove rules for decommissioned services.
**Anti-patterns:**
- No network policies ("everything's in the cluster, it's fine").
- Overly broad allow rules (`allow all from namespace X`) -- defeats the purpose.
- Network policies without egress rules -- ingress-only policies still allow compromised pods to exfiltrate data.
### 6.2 Least-privilege service identities
**Principle:** Each service gets its own identity (Kubernetes ServiceAccount + SPIFFE SVID) with the minimum permissions needed. No shared service accounts.
**Why it matters:** Shared identities prevent granular authorization, audit trails, and revocation. If services A and B share an identity, you cannot authorize A without also authorizing B.
**How to implement:**
- One Kubernetes ServiceAccount per workload (not per namespace).
- RBAC bindings scoped to exactly what the service needs (specific API groups, resources, verbs).
- Authorization policies reference specific service identities: "payment-svc can call order-svc on POST /orders/{id}/payment."
- Regularly audit which identities have access to which services -- prune unused access.
**Anti-patterns:**
- Default ServiceAccount used by all pods in a namespace.
- Cluster-wide RBAC bindings for convenience.
- Service identities with wildcard permissions ("allow all methods on all paths").
- No audit of identity-to-service mappings.
### 6.3 Observability as a security control
**Principle:** Distributed tracing, access logs, and metrics from the service mesh are security controls, not just debugging tools. Monitor them for anomalies.
**Why it matters:** Zero trust assumes breach. Detection depends on visibility. If you can't see who called what, you can't detect lateral movement.
**How to implement:**
- Enable access logging in the service mesh (Istio/Envoy access logs, Linkerd tap).
- Distributed tracing (OpenTelemetry, Jaeger) with trace context propagated across all service calls.
- Alert on anomalies: unexpected source-destination pairs, unusual request volumes, authorization denials.
- Retain access logs long enough for incident investigation (30-90 days minimum).
**Anti-patterns:**
- Disabling access logging for performance -- sample instead of disabling entirely.
- Tracing only in development, not production.
- No alerting on authorization policy denials -- failed access attempts are the signal.
---
## OWASP API Security Top 10 (2023) Quick Reference
For context, the current OWASP API Security Top 10 maps to the practices above:
| # | Risk | Where addressed |
|---|------|----------------|
| API1 | Broken Object-Level Authorization | Section 2.4 |
| API2 | Broken Authentication | Sections 2.1, 2.2, 2.3 |
| API3 | Broken Object Property-Level Authorization | Section 2.4, 4.1 |
| API4 | Unrestricted Resource Consumption | Sections 3.5, 4.3 |
| API5 | Broken Function-Level Authorization | Section 2.4 |
| API6 | Unrestricted Access to Sensitive Business Flows | Sections 3.4, 3.5 |
| API7 | Server-Side Request Forgery | Section 4.2 |
| API8 | Security Misconfiguration | Sections 1.1, 6.1 |
| API9 | Improper Inventory Management | Section 3.1 |
| API10 | Unsafe Consumption of APIs | Section 4.2 |
---
## Sources
- [OWASP API Security Top 10](https://owasp.org/API-Security/)
- [RFC 9700 - OAuth 2.0 Security Best Current Practice (January 2025)](https://datatracker.ietf.org/doc/rfc9700/)
- [OAuth best practices: RFC 9700 summary -- WorkOS](https://workos.com/blog/oauth-best-practices)
- [IETF Idempotency-Key Header Draft](https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/)
- [Google AIP-193: Errors](https://google.aip.dev/193)
- [ByteByteGo: REST API Design](https://blog.bytebytego.com/p/the-art-of-rest-api-design-idempotency)
- [Zuplo: Rate Limiting Best Practices](https://zuplo.com/learning-center/10-best-practices-for-api-rate-limiting-in-2025)
- [Zuplo: Input/Output Validation](https://zuplo.com/blog/2025/03/25/input-output-validation-best-practices)
- [OWASP Input Validation Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Input_Validation_Cheat_Sheet.html)
- [Machine Identity: mTLS + SPIFFE Zero Trust Guide](https://petronellatech.com/blog/machine-identity-is-the-new-perimeter-mtls-spiffe-for-zero-trust/)
- [Buoyant: Zero Trust, mTLS, and the Service Mesh](https://www.buoyant.io/blog/zero-trust-mtls-and-the-service-mesh-explained)
- [Kong: Zero Trust with Service Mesh](https://konghq.com/blog/engineering/zero-trust-service-mesh-security)
- [Microsoft Azure: Web API Design Best Practices](https://learn.microsoft.com/en-us/azure/architecture/best-practices/api-design)

100
harnesses/contexts/planning/v1/best-practices/database-selection.md Normal file
View File

@@ -0,0 +1,100 @@
# Database Selection
## The Rule: SQLite Is Not a Production Database
**Any service that meets ANY of the following criteria MUST use PostgreSQL (or equivalent server-grade database) from day one:**
- Attached to a FQDN (has a real domain name, even internal)
- Serves traffic from more than one process (API consumers, CI runners, webhooks, polling)
- Backs infrastructure that other systems depend on (Git hosting, container registries, auth providers)
- Will be accessed concurrently by automated systems (ArgoCD, CI runners, cron jobs)
**Do not use SQLite for these workloads. Not temporarily. Not "to start with." Not "we'll migrate later."**
SQLite uses file-level locking — only one writer at a time, and writes block reads. Under concurrent access, requests queue up waiting for the write lock, causing cascading timeouts. The failure mode is insidious: the service appears to work fine under light load but becomes intermittently unresponsive under real workloads. By the time you notice, everything that depends on it is also failing.
## The Cost of "We'll Migrate Later"
The Gitea SQLite→PostgreSQL migration (2026-03-28) cost nearly a full day of productivity:
- **Hours of accumulated unresponsiveness** across multiple projects before root cause was identified
- **Planning and implementation** of the migration itself
- **Migration complexity** that didn't need to exist: Gitea 1.23 has no `restore` command, `doctor convert` only handles charset conversion, `docker cp` corrupted PostgreSQL directory permissions, SSH authorized_keys weren't regenerated
- **Downstream impact** on ArgoCD (20 apps polling a locked database), CI runners (continuous 500 errors), container registry pulls (timeouts)
The PostgreSQL container takes 5 minutes to add to a Docker Compose stack at initial setup time. The migration took a day. Always pay the 5 minutes upfront.
## When SQLite Is Acceptable
SQLite is fine for:
- Local development databases (single developer, single process)
- Embedded application data stores (mobile apps, desktop apps, CLI tools)
- Read-heavy workloads with rare writes and a single writer process
- Test fixtures and throwaway data
- Configuration stores read at startup (not at request time)
## Implementation Pattern
For Docker Compose services that need a database:
```yaml
services:
postgres:
image: postgres:17-alpine
restart: unless-stopped
environment:
POSTGRES_DB: myapp
POSTGRES_USER: myapp
POSTGRES_PASSWORD: {{ db_password }}
volumes:
- /opt/postgres-myapp:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U myapp -d myapp"]
interval: 10s
timeout: 5s
retries: 5
start_period: 30s
networks:
- app_internal
deploy:
resources:
limits:
memory: 1G
myapp:
depends_on:
postgres:
condition: service_healthy
networks:
- app_internal
- external_network
networks:
app_internal:
driver: bridge
internal: true
```
Key points:
- PostgreSQL on an **internal bridge network** (no external access needed)
- Application **depends on PostgreSQL health** before starting
- **Resource limits** to prevent runaway memory usage
- **Separate data directory** per application (`/opt/postgres-myapp`, not shared)
- PostgreSQL container UID is **999** (not 1000) — set directory ownership accordingly
## For Kubernetes Deployments
Use the application's Helm chart PostgreSQL subchart, or deploy a standalone PostgreSQL instance:
- Bitnami PostgreSQL Helm chart for simple deployments
- CloudNativePG operator for production-grade PostgreSQL with HA, backups, and failover
- Never use SQLite with `emptyDir` or even PVC-backed volumes in multi-replica deployments
## Checklist for New Service Deployment
Before deploying any new service, check:
1. What database does the default configuration use?
2. If SQLite: does the service support PostgreSQL? (Almost all do — Gitea, Authelia, Headscale, Zulip, etc.)
3. Switch to PostgreSQL **before the first deployment**, not after problems appear
4. Add the database password to SOPS-encrypted secrets
5. Verify the database connection works before adding consumers

47
harnesses/contexts/planning/v1/best-practices/docker.md Normal file
View File

@@ -0,0 +1,47 @@
# Docker Best Practices
## Use gosu for Entrypoint Privilege Dropping
`su -c "command"` and `sudo -u agent command` create child processes. The real command is not PID 1, so Docker signals (SIGTERM on stop) don't reach it. Use `gosu agent command` which execs directly — the command becomes PID 1 with proper signal handling.
## GIT_SSH_COMMAND Only Affects Git-Invoked SSH
`GIT_SSH_COMMAND` only applies when git invokes SSH (clone, push, fetch). Direct `ssh` calls need explicit flags: `ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null`. Don't assume setting `GIT_SSH_COMMAND` fixes all SSH operations in a container.
## Use Python urllib for Health Checks in Slim Images
Service images based on `python:3.12-slim` don't include curl. For in-container health checks, use `python3 -c "import urllib.request; urllib.request.urlopen('http://...')"`. This applies to verification scripts using `kubectl exec` and to Kubernetes liveness/readiness probes that exec into containers.
## Buildx Docker-Container Driver Can't See Local Images
When using buildx with the docker-container driver, `FROM local-image:latest` tries Docker Hub because the builder runs in a separate container that can't see locally-loaded images. Always use the full registry path in Dockerfiles. In CI, split into sequential jobs so base images are pushed to the registry before dependent images build.
## Delete Conflicting Default Users at Build Time
Ubuntu 24.04 base images ship with a `ubuntu` user at UID 1000 — the most common host UID. This causes `usermod -u 1000` conflicts and can trigger non-deterministic hangs (e.g., `newgrp ubuntu` waiting for a password on stdin). Delete the default user in the Dockerfile: `RUN userdel -r ubuntu`.
## Service Images Should Use Minimal Base Images
Service images (API servers, background workers) should use `python:3.12-slim` or equivalent, not the agent base image. Agent base images include CLIs, Node.js, and other tooling that bloats service images unnecessarily. Keep agent tooling in agent images only.
## Platform-Specific Native Binaries
Never mount host `node_modules` into a Docker container when the build uses platform-specific native binaries (e.g., Tailwind CSS, esbuild, SWC). Always run `npm install` inside the same container that runs the build. The native binary is compiled for the platform where `npm install` runs — host and container may differ in libc, architecture, or OS.
**Symptom:** `Cannot find native binding` or `Cannot find module '@tailwindcss/oxide-linux-x64-gnu'`
**Fix:** Run `npm install` inside the container, not on the host.
## Docker Wrapper Scripts and TTY Flags
Docker wrapper scripts (e.g., `~/sbin/hugo` calling `docker run -it ...`) fail with `the input device is not a TTY` in non-interactive contexts (CI pipelines, Claude Code, cron jobs, scripts).
**Fix:** Only pass `-t` when stdin is a terminal: `[ -t 0 ] && TTY_FLAG="-t" || TTY_FLAG=""`. Or omit `-t` entirely and let callers add it when needed.
## Three-Tier UID Resolution
The UID wrapper should resolve the target UID/GID using this priority:
1. **Environment variables** (`AGENT_UID`/`AGENT_GID`) — injected by the orchestrator/dispatcher. Preferred because it's explicit and deployment-specific.
2. **stat the mount point** — detect the UID/GID of the mounted directory. Works when no env vars are set.
3. **Skip** — if not root or no mount point exists, run as the default container user.
This makes UID matching a deployment concern (varies per host), not a configuration concern (baked into images). See [Docker UID Matching](docker-uid-matching.md) for the full UID wrapper pattern.

69
harnesses/contexts/planning/v1/best-practices/kubernetes.md Normal file
View File

@@ -0,0 +1,69 @@
# Kubernetes Patterns
## Volume Mounts
- **Avoid `subPath` volume mounts** for Secrets and ConfigMaps. The kubelet does not auto-update `subPath` mounts when the source changes — the pod must be restarted. Use directory mounts instead and adjust the application's config path.
- **Secret volume propagation is async.** After updating a Secret, the kubelet takes seconds to sync mounted volumes. A `rollout restart` issued immediately after may start pods with stale data. Add a short delay (5s) before restarting.
## Deployment Strategies
- **RWO PVC + RollingUpdate = Deadlock.** New pod can't attach the volume while the old pod holds it. Use `strategy: Recreate` for single-replica deployments with RWO PVCs.
- **SSA + strategy change conflict.** Switching from RollingUpdate to Recreate via ServerSideApply fails because SSA won't remove the old `rollingUpdate` field. Must patch the live resource first.
## Naming
- `metadata.name` must be DNS-1035 compliant — no dots allowed. Replace dots with dashes (e.g., `oreillyit-nz` not `oreillyit.nz`). Label values CAN contain dots.
## Bootstrap Ordering
Some components have chicken-and-egg dependencies:
1. CNI (e.g., Cilium) must be installed before anything else — nodes are NotReady without it
2. GitOps controller (e.g., ArgoCD) installed second
3. Root app applied last — the GitOps controller then "adopts" CLI-installed releases
Manual bootstrap secrets (encryption keys, OIDC client secrets) must be documented as explicit steps.
## Network Policies
- DNS egress for `toFQDNs` rules must use `toEndpoints` targeting kube-dns pods with `rules.dns` — this triggers the DNS proxy. Using `toCIDRSet` for DNS bypasses the proxy and FQDN rules never populate.
- Cross-namespace policies need explicit namespace matching (e.g., `matchExpressions` on namespace label).
- Always test from the actual consumer namespace, not same-namespace test pods.
## Probe Strategy
- **Liveness vs readiness probes serve different purposes.** TCP checks confirm the process is listening (liveness). Exec/command checks confirm the application is ready to serve (readiness). Don't conflate them.
- **Probes must match application host validation.** Applications that validate Host headers (e.g., Next.js `ALLOWED_HOSTS`) will reject probes sent to the pod IP. Set `httpGet.httpHeaders` with the expected Host value.
- **Don't load credentials into liveness probes.** If readiness requires an authenticated check (e.g., `sqlcmd`), use a simple TCP check for liveness and reserve the authenticated check for readiness only.
## Init Container Patterns
- **Writable config via init container + emptyDir.** When apps require writable directories but ConfigMaps are read-only, use an init container to copy config into an emptyDir volume that the main container mounts read-write.
- **Privilege separation.** Init containers can run as root to create directories or set ownership, while the main container runs as a non-root UID. Prefer this over running the entire workload as root.
- **Non-root images have hidden filesystem requirements.** Many modern images (e.g., MSSQL 2022, UID 10001) need writable directories beyond the obvious ones. Always check image documentation or `docker inspect` before writing manifests.
## StatefulSet Edge Cases
- **CrashLoopBackOff pods won't auto-replace on spec update.** The StatefulSet controller won't delete and recreate a crashing pod when you update the spec — manual `kubectl delete pod` is required to force recreation.
- **Immutable field diffs can deadlock auto-sync.** StatefulSet fields like `volumeClaimTemplates` are immutable after creation. GitOps controllers (ArgoCD) will show permanent OutOfSync if the desired state differs from the live immutable fields. Force sync or recreate the StatefulSet.
- **SSA causes perpetual OutOfSync from defaulted fields.** Kubernetes defaults fields on StatefulSets (`persistentVolumeClaimRetentionPolicy`, `revisionHistoryLimit`, `updateStrategy.rollingUpdate.partition`) that aren't in the Helm template. With `ServerSideApply=true`, GitOps controllers see these as diffs and report OutOfSync even though the app is Healthy. The app functions correctly — this is cosmetic. Consider ArgoCD `ignoreDifferences` for these fields.
## GitOps: Imperative vs Declarative
- **Never use imperative operations on GitOps-managed resources.** `kubectl rollout restart` adds annotations that conflict with the GitOps controller's desired state, causing permanent OutOfSync. Use declarative paths instead — update a configmap hash annotation in Git, or change a pod template label.
- **ArgoCD reconciliation has latency.** New Application manifests don't appear immediately due to polling intervals. Use manual refresh annotations when automation needs immediate reconciliation.
## PodSecurity Alignment
- **Namespace PodSecurity labels must match container security contexts.** DinD, CSI drivers, and other privileged workloads need `pod-security.kubernetes.io/enforce: privileged` on their namespace. A `baseline` or `restricted` namespace silently blocks privileged pods.
- **Document privileged namespace requirements.** When a workload needs elevated privileges, document the specific requirement (e.g., "Docker-in-Docker for CI builds") alongside the namespace label.
## ArgoCD Source Type Detection
- **ArgoCD auto-detects Kustomize.** When a source directory contains `kustomization.yaml`, ArgoCD runs Kustomize automatically. Adding an explicit `directory:` source type overrides this detection and causes ArgoCD to try applying `kustomization.yaml` as a raw K8s resource, which fails with schema errors. Remove explicit directory source types from Kustomize sources.
- **Credential template URL-prefix must match exactly.** ArgoCD repo-creds secrets use URL prefix matching. When migrating Git server URLs (hostname, protocol, or port changes), update the credential template to match the new prefix. Stale credentials cause "authentication required" errors on all apps using that prefix.
## Miscellaneous
- `enableServiceLinks: false` may be needed when K8s-injected service env vars conflict with app config (e.g., Authelia interprets `AUTHELIA_*` service vars as configuration).
- Proxmox VM names must match K8s node hostnames for cloud controller manager integration.
- Metrics-server on Talos needs `--kubelet-insecure-tls` (self-signed kubelet certs).

800
harnesses/contexts/planning/v1/best-practices/llm-code-security.md Normal file
View File

@@ -0,0 +1,800 @@
# Security of LLM-Generated Code
Practical guide to security vulnerabilities commonly introduced by LLMs (Claude, GPT-4, Copilot) when generating Python, shell scripts, Kubernetes manifests, and Helm charts. Based on published research from 2024-2026.
## Key Statistics
- 25-75% of AI-generated code contains security vulnerabilities depending on language, model, and prompting (Endor Labs, multiple academic studies)
- 29.5% of Copilot-generated Python snippets and 24.2% of JavaScript snippets contain security weaknesses across 43 CWE categories (ACM study, 2024)
- 19.7% of LLM-suggested packages are hallucinations -- non-existent package names (slopsquatting study, 576,000 code samples across 16 models)
- 80% of AI-suggested dependencies contain known risks (Endor Labs 2025 State of Dependency Management Report)
- Repositories with Copilot active show 6.4% secret leakage rate, 40% higher than the 4.6% baseline across public repos
---
## 1. OWASP Top 10 in LLM-Generated Code
Missing input sanitization is the single most common security flaw in LLM-generated code across all languages and models. The most prevalent CWE categories are:
| CWE | Name | Frequency |
|-----|------|-----------|
| CWE-89 | SQL Injection | Very High |
| CWE-79 | Cross-Site Scripting (XSS) | Very High |
| CWE-78 | OS Command Injection | High |
| CWE-22 | Path Traversal | High |
| CWE-20 | Improper Input Validation | Very High |
| CWE-259/798 | Hard-coded Credentials | High |
| CWE-330 | Insufficiently Random Values | High |
| CWE-94 | Code Injection | High |
| CWE-120/787 | Buffer Overflow | Medium (C/C++) |
| CWE-918 | SSRF | Medium |
### What LLMs get wrong
LLMs generate code that "works" for the happy path but omits defensive coding. They reproduce patterns from training data, which is full of tutorials and Stack Overflow snippets that skip security for brevity. The model optimises for functional correctness, not security.
### SQL Injection
**Vulnerable pattern (Python):**
```python
# LLM-generated: string interpolation in SQL
def get_user(username):
query = f"SELECT * FROM users WHERE username = '{username}'"
cursor.execute(query)
return cursor.fetchone()
```
**Secure alternative:**
```python
def get_user(username):
cursor.execute("SELECT * FROM users WHERE username = %s", (username,))
return cursor.fetchone()
```
### Command Injection
**Vulnerable pattern (Python):**
```python
import subprocess
def ping_host(hostname):
result = subprocess.run(f"ping -c 1 {hostname}", shell=True, capture_output=True)
return result.stdout
```
**Secure alternative:**
```python
import subprocess
import shlex
def ping_host(hostname):
# Validate hostname format first
if not re.match(r'^[a-zA-Z0-9._-]+$', hostname):
raise ValueError("Invalid hostname")
result = subprocess.run(["ping", "-c", "1", hostname], capture_output=True)
return result.stdout
```
### Command Injection (Shell Scripts)
**Vulnerable pattern:**
```bash
#!/bin/bash
# LLM-generated: unquoted variable in command
filename=$1
cat $filename | grep "pattern"
```
**Secure alternative:**
```bash
#!/bin/bash
filename="$1"
# Validate the path is within expected directory
realpath_file="$(realpath -- "$filename")"
if [[ "$realpath_file" != /expected/dir/* ]]; then
echo "Error: path outside allowed directory" >&2
exit 1
fi
grep "pattern" -- "$filename"
```
### Path Traversal
**Vulnerable pattern (Python):**
```python
@app.route('/files/<path:filename>')
def serve_file(filename):
return send_file(os.path.join('/data', filename))
```
**Secure alternative:**
```python
@app.route('/files/<path:filename>')
def serve_file(filename):
# send_from_directory validates the path stays within the directory
return send_from_directory('/data', filename)
```
### How to catch it in review
- Search for string formatting in SQL: `f"SELECT`, `f"INSERT`, `f"UPDATE`, `f"DELETE`, `"SELECT.*" %`, `"SELECT.*" +`
- Search for `shell=True` in subprocess calls
- Search for `os.path.join` with user-controlled input without path validation
- Search for unquoted `$variables` in shell scripts
- Use SAST tools: Bandit (Python), ShellCheck (bash), semgrep with security rulesets
---
## 2. Secrets and Credentials
### What LLMs get wrong
LLMs frequently hardcode secrets directly into generated code. This happens because training data is full of tutorials with placeholder credentials that look real, and the model replicates the pattern. CWE-259 (Hard-coded Password) and CWE-798 (Hard-coded Credentials) are among the most common LLM-generated vulnerabilities.
Copilot specifically has been shown to leak secrets from its training context -- researchers built algorithms that generate prompts designed to extract secrets by inducing Copilot to disclose original credentials from training data.
### Vulnerable patterns
**Hardcoded API key (Python):**
```python
API_KEY = "sk-proj-abc123def456..."
client = openai.OpenAI(api_key=API_KEY)
```
**Hardcoded database credentials (Python):**
```python
conn = psycopg2.connect(
host="db.example.com",
user="admin",
password="supersecret123",
database="production"
)
```
**Hardcoded token in shell script:**
```bash
curl -H "Authorization: Bearer ghp_abc123def456" https://api.github.com/repos
```
**Secrets in Kubernetes manifests:**
```yaml
env:
- name: DATABASE_PASSWORD
value: "plaintext-password-here" # Not a Secret reference
```
### Secure alternatives
**Python -- environment variables or file-based secrets:**
```python
import os
API_KEY = os.environ["OPENAI_API_KEY"]
# Or read from a mounted secret file
with open("/run/secrets/api_key") as f:
API_KEY = f.read().strip()
```
**Shell -- read from file, never as CLI argument:**
```bash
# Read token from file (not visible in ps output)
TOKEN="$(cat /path/to/secret/file)"
curl -H "Authorization: Bearer ${TOKEN}" https://api.github.com/repos
```
**Kubernetes -- reference a Secret object:**
```yaml
env:
- name: DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: db-credentials
key: password
```
### How to catch it in review
- Run `detect-secrets scan` or `gitleaks` on every commit (pre-commit hook)
- Search for patterns: `password =`, `api_key =`, `token =`, `secret =` with string literal values
- Search for `Bearer ` followed by a literal string in shell scripts
- In Kubernetes manifests, search for `value:` under `env:` entries (should be `valueFrom:` for sensitive values)
- Check that `.env` files are in `.gitignore`
---
## 3. Dependency Risks
### What LLMs get wrong
LLMs hallucinate package names at alarming rates. A study of 576,000 code samples across 16 LLMs found 19.7% of suggested packages were hallucinations. Open-source models hallucinate at 21.7%, commercial models at 5.2%. Critically, 43% of hallucinated package names appeared consistently across repeated prompts, making them predictable targets.
This enables **slopsquatting**: attackers register packages matching commonly hallucinated names and inject malicious code. 38% of hallucinated names were similar to real package names (not random strings), making them plausible-looking.
Beyond hallucination, LLMs also suggest:
- **Outdated versions** with known CVEs (training data lag)
- **Deprecated packages** that have been superseded
- **Packages with known vulnerabilities** -- 80% of AI-suggested dependencies contain known risks
### Vulnerable patterns
**Hallucinated package (Python):**
```python
# LLM suggests a package that doesn't exist (or was registered by an attacker)
from flask_security_utils import sanitize_input # Not a real package
```
**Pinned to vulnerable version:**
```
# requirements.txt generated by LLM
requests==2.25.1 # Known CVE in older versions
pyjwt==1.7.1 # Known vulnerabilities
```
**Overly broad dependency (shell):**
```bash
pip install cryptography # Without version pin -- could get a compromised version
```
### Secure alternatives
- **Always verify packages exist** on PyPI/npm/etc. before using LLM-suggested imports
- **Pin versions and verify them:**
```
requests==2.32.3 # Verified from PyPI, no known CVEs
```
- **Use lockfiles** (`pip freeze`, `poetry.lock`, `package-lock.json`) and audit them
- **Run dependency scanners:** `pip-audit`, `npm audit`, `trivy fs .`
### How to catch it in review
- Run `pip install --dry-run` or equivalent to verify packages resolve before committing
- Use `pip-audit` / `npm audit` / `trivy` in CI to catch known vulnerabilities
- Compare LLM-suggested package names against registry search results
- Be suspicious of packages with very few downloads or recent creation dates
- Search for version pins and verify them against current stable releases
---
## 4. Over-Permissive Defaults
### What LLMs get wrong
LLMs default to the most permissive configuration because it "works" with the least friction. Training data is full of tutorials and quick-start guides that use wide-open settings. The model has no concept of a deployment environment or threat model.
### Vulnerable patterns
**Binding to all interfaces (Python):**
```python
app.run(host="0.0.0.0", port=8080, debug=True) # Exposed to network + debug mode
```
**Wide-open CORS (Python/Flask):**
```python
CORS(app, origins="*", supports_credentials=True)
```
**Permissive file permissions (shell):**
```bash
chmod 777 /app/data
chmod 666 /etc/config/credentials.yaml
```
**Disabled TLS verification (Python):**
```python
requests.get(url, verify=False)
```
**Kubernetes Service exposed externally by default:**
```yaml
apiVersion: v1
kind: Service
metadata:
name: my-app
spec:
type: LoadBalancer # Exposed to the network
ports:
- port: 80
```
### Secure alternatives
**Bind to localhost unless external access is needed:**
```python
app.run(host="127.0.0.1", port=8080, debug=False)
```
**Explicit CORS origins:**
```python
CORS(app, origins=["https://app.example.com"], supports_credentials=True)
```
**Restrictive file permissions:**
```bash
chmod 750 /app/data # Owner rwx, group rx, others none
chmod 640 /etc/config/credentials.yaml # Owner rw, group r, others none
```
**TLS verification enabled (always):**
```python
requests.get(url, verify=True) # Default, but be explicit
# If using internal CA:
requests.get(url, verify="/etc/ssl/certs/internal-ca.pem")
```
**ClusterIP by default, expose deliberately:**
```yaml
apiVersion: v1
kind: Service
metadata:
name: my-app
spec:
type: ClusterIP # Internal only, use Ingress for external access
ports:
- port: 80
```
### How to catch it in review
- Search for `0.0.0.0`, `host="0.0.0.0"`, `debug=True` in application code
- Search for `origins="*"` or `Access-Control-Allow-Origin: *` in CORS config
- Search for `chmod 777`, `chmod 666`, or any world-readable/writable permissions
- Search for `verify=False` in HTTP client calls
- Search for `type: LoadBalancer` or `type: NodePort` in Kubernetes manifests without explicit justification
- Search for `GRANT ALL` in database setup scripts
---
## 5. Infrastructure-as-Code Risks
### What LLMs get wrong
LLMs generate Kubernetes manifests and Helm charts that are functionally correct but security-negligent. They omit security contexts, resource limits, network policies, and run containers as root by default. Research (GenKubeSec, KubeGuard) found that LLMs can "confidently recommend configurations that introduce new vulnerabilities" including suggesting "allow all" rules just to satisfy constraints.
### Vulnerable patterns
**Privileged container (Kubernetes):**
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
template:
spec:
containers:
- name: my-app
image: my-app:latest # No digest, mutable tag
# No securityContext at all -- runs as root
# No resource limits -- can consume entire node
# No readOnlyRootFilesystem
```
**Overly broad RBAC:**
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: my-app
subjects:
- kind: ServiceAccount
name: my-app
roleRef:
kind: ClusterRole
name: cluster-admin # Full cluster access
```
**No NetworkPolicy (default allows all traffic):**
```yaml
# LLMs typically omit NetworkPolicy entirely
# Without it, any pod can talk to any other pod
```
**Helm values without security defaults:**
```yaml
# values.yaml generated by LLM
replicaCount: 1
image:
repository: my-app
tag: latest # Mutable, unpinned
service:
type: LoadBalancer # Externally exposed
# No securityContext, no resources, no networkPolicy
```
### Secure alternatives
**Hardened container:**
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
template:
spec:
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
seccompProfile:
type: RuntimeDefault
containers:
- name: my-app
image: my-app@sha256:abc123... # Pinned by digest
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 256Mi
```
**Least-privilege RBAC:**
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role # Namespaced, not ClusterRole
metadata:
name: my-app
namespace: my-namespace
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "list"] # Only what's needed
```
**Default-deny NetworkPolicy:**
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: my-app
spec:
podSelector:
matchLabels:
app: my-app
policyTypes: ["Ingress", "Egress"]
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
ports:
- port: 8080
egress:
- to:
- podSelector:
matchLabels:
app: database
ports:
- port: 5432
```
### How to catch it in review
- Run `kubesec scan`, `kube-linter`, or `trivy config` against manifests
- Search for `privileged: true`, `allowPrivilegeEscalation: true` (should almost never appear)
- Search for `cluster-admin` in RBAC bindings
- Check that every Deployment/StatefulSet has `resources:` limits and `securityContext:`
- Check that every namespace has at least one NetworkPolicy
- Search for `image:.*:latest` -- tags should be pinned to specific versions or digests
- Check for `automountServiceAccountToken: false` on pods that don't need the K8s API
- In Helm charts, verify `values.yaml` includes security defaults, not just functional defaults
---
## 6. Input Validation Gaps
### What LLMs get wrong
LLMs generate code that handles the happy path but skips validation of types, lengths, formats, and ranges. They omit validation unless explicitly prompted, because training data (tutorials, examples) does the same. The model has no awareness of the threat model or what inputs are user-controlled.
### Vulnerable patterns
**No type/length validation (Python API):**
```python
@app.route('/api/users', methods=['POST'])
def create_user():
data = request.get_json()
username = data['username'] # No validation at all
email = data['email'] # No format check
age = data['age'] # No type or range check
db.execute("INSERT INTO users (username, email, age) VALUES (%s, %s, %s)",
(username, email, age))
```
**No path validation (shell):**
```bash
#!/bin/bash
# LLM-generated backup script
BACKUP_DIR="$1"
cp -r /important/data "$BACKUP_DIR" # No validation of $1
```
### Secure alternatives
**Validated API input (Python):**
```python
from pydantic import BaseModel, EmailStr, Field
class CreateUserRequest(BaseModel):
username: str = Field(min_length=3, max_length=50, pattern=r'^[a-zA-Z0-9_]+$')
email: EmailStr
age: int = Field(ge=0, le=150)
@app.route('/api/users', methods=['POST'])
def create_user():
data = CreateUserRequest(**request.get_json()) # Validates or raises 422
db.execute("INSERT INTO users (username, email, age) VALUES (%s, %s, %s)",
(data.username, data.email, data.age))
```
**Validated shell input:**
```bash
#!/bin/bash
BACKUP_DIR="$1"
if [[ -z "$BACKUP_DIR" ]]; then
echo "Error: backup directory required" >&2
exit 1
fi
if [[ ! -d "$BACKUP_DIR" ]]; then
echo "Error: '$BACKUP_DIR' is not a directory" >&2
exit 1
fi
# Resolve and validate path
REAL_DIR="$(realpath -- "$BACKUP_DIR")"
if [[ "$REAL_DIR" != /allowed/backup/* ]]; then
echo "Error: backup directory must be under /allowed/backup/" >&2
exit 1
fi
cp -r /important/data "$REAL_DIR"
```
### How to catch it in review
- Check that all API endpoints use schema validation (Pydantic, marshmallow, JSON Schema, Joi)
- Search for `request.get_json()`, `request.args`, `request.form` usage without subsequent validation
- In shell scripts, check that all positional parameters (`$1`, `$2`, etc.) are validated before use
- Look for direct use of user input in file operations, database queries, or system commands
- Verify that numeric inputs have range checks and string inputs have length/format checks
---
## 7. Error Handling That Leaks Information
### What LLMs get wrong
LLMs generate code with verbose error handling that exposes internal details -- stack traces, file paths, database schemas, SQL queries, internal hostnames. This happens because training data includes development-mode error handling, and the model doesn't distinguish between dev and production contexts.
### Vulnerable patterns
**Leaking stack traces (Python/Flask):**
```python
@app.errorhandler(Exception)
def handle_error(e):
return jsonify({
"error": str(e),
"traceback": traceback.format_exc(), # Full stack trace
"query": last_query, # SQL query that failed
}), 500
```
**Leaking database details:**
```python
try:
cursor.execute(query)
except psycopg2.Error as e:
return f"Database error: {e}" # Includes table names, column names, query
```
**Leaking file paths (shell):**
```bash
echo "Error: failed to read config from /etc/myapp/secrets/database.yaml"
echo "Stack: $(python3 -c 'import traceback; traceback.print_exc()')"
```
### Secure alternatives
**Generic error response with internal logging:**
```python
import logging
logger = logging.getLogger(__name__)
@app.errorhandler(Exception)
def handle_error(e):
logger.exception("Unhandled exception") # Full details go to logs
return jsonify({"error": "Internal server error"}), 500 # Generic to client
```
**Safe database error handling:**
```python
try:
cursor.execute(query, params)
except psycopg2.Error as e:
logger.exception("Database query failed")
return jsonify({"error": "A database error occurred"}), 500
```
### How to catch it in review
- Search for `traceback.format_exc()` or `traceback.print_exc()` in response-building code
- Search for `str(e)` or `repr(e)` in API responses (should go to logs, not clients)
- Check that `DEBUG = False` / `debug=False` in production config
- Verify error handlers return generic messages and log details internally
- Search for internal paths (`/etc/`, `/home/`, `/var/`) in user-facing error strings
---
## 8. Cryptography Mistakes
### What LLMs get wrong
LLMs reproduce cryptographic anti-patterns from training data. CWE-780 (Use of RSA without OAEP) is the most observed weakness in Java. Common failures include using ECB mode (which leaks patterns), predictable IVs, deprecated algorithms (MD5, SHA-1 for security purposes), and rolling custom crypto. Cryptography misconfiguration appears in approximately 22-24% of security vulnerabilities across leading LLM models.
### Vulnerable patterns
**ECB mode (Python):**
```python
from Crypto.Cipher import AES
cipher = AES.new(key, AES.MODE_ECB) # ECB leaks patterns in ciphertext
ciphertext = cipher.encrypt(plaintext)
```
**Hardcoded IV:**
```python
iv = b'\x00' * 16 # Predictable IV defeats the purpose of CBC/GCM
cipher = AES.new(key, AES.MODE_CBC, iv=iv)
```
**MD5 for password hashing:**
```python
import hashlib
password_hash = hashlib.md5(password.encode()).hexdigest() # Broken for security
```
**Weak random for tokens:**
```python
import random
token = ''.join(random.choices(string.ascii_letters, k=32)) # Not cryptographically secure
```
### Secure alternatives
**AES-GCM with random IV:**
```python
from Crypto.Cipher import AES
from Crypto.Random import get_random_bytes
key = get_random_bytes(32) # AES-256
nonce = get_random_bytes(12) # Random nonce for GCM
cipher = AES.new(key, AES.MODE_GCM, nonce=nonce)
ciphertext, tag = cipher.encrypt_and_digest(plaintext)
# Store nonce + tag + ciphertext together
```
**Proper password hashing:**
```python
import bcrypt
# Hashing
password_hash = bcrypt.hashpw(password.encode(), bcrypt.gensalt(rounds=12))
# Verification
bcrypt.checkpw(password.encode(), stored_hash)
```
**Cryptographically secure random:**
```python
import secrets
token = secrets.token_urlsafe(32) # Cryptographically secure
```
### How to catch it in review
- Search for `MODE_ECB` -- should almost never be used
- Search for `md5`, `sha1` used for passwords or security tokens (fine for checksums, not for security)
- Search for `random.` (stdlib) used for tokens, keys, or security values -- should be `secrets.`
- Search for hardcoded IVs: `iv = b'`, `iv = bytes(`, `nonce = b'\x00`
- Search for `hashlib` used directly for password storage -- should be `bcrypt`, `argon2`, or `scrypt`
- Use `bandit` which has specific checks for weak crypto (B303, B304, B305)
---
## 9. Research Findings (2024-2026)
### ACM / TOSEM: Security Weaknesses of Copilot-Generated Code in GitHub Projects
Analyzed real-world Copilot-generated code on GitHub. Found 29.5% of Python and 24.2% of JavaScript snippets contained security weaknesses across 43 CWE categories. Top weaknesses: CWE-330 (insufficiently random values), CWE-94 (code injection), CWE-79 (XSS).
### Large-Scale GitHub Analysis (October 2025)
Analyzed 7,703 files from 4 AI tools across public GitHub repos. Found 4,241 CWE instances across 77 distinct vulnerability types. ChatGPT-generated code comprised 91.5% of the sample, Copilot 7.5%.
### Slopsquatting Research (2025)
576,000 code samples across 16 LLMs: 19.7% of suggested packages were hallucinations (205,474 unique fake names). Open-source models hallucinated at 21.7%, commercial at 5.2%. 43% of hallucinated names appeared consistently (predictable, attackable).
### Endor Labs: State of Dependency Management (2025)
80% of AI-suggested dependencies contain known risks. 44-49% of dependencies imported by coding agents contained known security vulnerabilities.
### Copilot Code Review Study (2025)
GitHub Copilot's code review feature frequently fails to detect critical vulnerabilities (SQL injection, XSS, insecure deserialization). Primarily flags low-severity issues like coding style.
### Sonar: Coding Personalities of Leading LLMs (2025)
Multi-model analysis finding that cryptography misconfiguration appears in 22-24% of vulnerabilities across leading models. Missing input sanitization is the most common flaw category.
### GenKubeSec / KubeGuard (2024-2025)
Research on LLM-generated Kubernetes configurations found models confidently recommend insecure configurations and may suggest "allow all" rules to satisfy functional requirements.
### OWASP Top 10 for LLM Applications (2025 Update)
Updated to reflect agentic AI risks. Key additions: System Prompt Leakage, Excessive Agency. Improper Output Handling (treating LLM output as trusted) remains a top-5 risk. Core message: treat all LLM output as untrusted data.
### Security Degradation in Iterative Generation (2025)
Code security degrades with iterative LLM refinement -- each round of "fix this" prompting can introduce new vulnerabilities while fixing the original one.
---
## 10. Practical Review Checklist
Use this checklist when reviewing LLM-generated code:
### Python
- [ ] No string formatting in SQL queries (use parameterised queries)
- [ ] No `shell=True` in subprocess calls
- [ ] No `verify=False` in HTTP requests
- [ ] No `random.` for security values (use `secrets.`)
- [ ] No `hashlib.md5/sha1` for passwords (use `bcrypt`/`argon2`)
- [ ] No hardcoded credentials (search for `password =`, `api_key =`, `token =`)
- [ ] Input validation on all API endpoints (Pydantic, marshmallow)
- [ ] Error handlers return generic messages, log details internally
- [ ] `debug=False` in production config
- [ ] All dependencies exist on PyPI and are pinned to audited versions
- [ ] `host="127.0.0.1"` unless external binding is explicitly required
### Shell Scripts
- [ ] All variables quoted (`"$var"` not `$var`)
- [ ] User-provided paths validated with `realpath` and boundary checks
- [ ] No secrets as command-line arguments (use files or env vars)
- [ ] No `chmod 777` or `chmod 666`
- [ ] ShellCheck passes with no warnings
### Kubernetes Manifests
- [ ] `securityContext` present with `runAsNonRoot: true`, `readOnlyRootFilesystem: true`, `allowPrivilegeEscalation: false`
- [ ] `capabilities.drop: ["ALL"]`
- [ ] `resources.requests` and `resources.limits` defined
- [ ] No `privileged: true`
- [ ] No `cluster-admin` RBAC bindings
- [ ] `automountServiceAccountToken: false` where K8s API access is not needed
- [ ] Images pinned to digest or specific version (not `:latest`)
- [ ] Services use `ClusterIP` by default (not `LoadBalancer`/`NodePort` without justification)
- [ ] NetworkPolicy exists for the namespace/workload
### Helm Charts
- [ ] `values.yaml` includes secure defaults for securityContext, resources, service type
- [ ] Templates don't embed secrets in plaintext
- [ ] Chart version and appVersion pinned
- [ ] `helm template` renders valid, secure manifests with default values
- [ ] `values.schema.json` validates required security fields
---
## Sources
- [Security Weaknesses of Copilot-Generated Code in GitHub Projects (ACM TOSEM)](https://dl.acm.org/doi/10.1145/3716848)
- [Security Vulnerabilities in AI-Generated Code: A Large-Scale Analysis (arXiv, Oct 2025)](https://arxiv.org/abs/2510.26103)
- [The Most Common Security Vulnerabilities in AI-Generated Code (Endor Labs)](https://www.endorlabs.com/learn/the-most-common-security-vulnerabilities-in-ai-generated-code)
- [Endor Labs 2025 State of Dependency Management Report](https://www.prnewswire.com/news-releases/endor-labs-launches-2025-state-of-dependency-management-report-finds-80-of-ai-suggested-dependencies-contain-risks-302603438.html)
- [LLMs' AI-Generated Code Remains Wildly Insecure (Dark Reading)](https://www.darkreading.com/application-security/llms-ai-generated-code-wildly-insecure)
- [Popular LLMs Found to Produce Vulnerable Code by Default (Infosecurity Magazine)](https://www.infosecurity-magazine.com/news/llms-vulnerable-code-default/)
- [Slopsquatting: How AI Hallucinations Are Fueling Supply Chain Attacks (Socket.dev)](https://socket.dev/blog/slopsquatting-how-ai-hallucinations-are-fueling-a-new-class-of-supply-chain-attacks)
- [Slopsquatting meets Dependency Confusion (Andrew Nesbitt)](https://nesbitt.io/2025/12/10/slopsquatting-meets-dependency-confusion.html)
- [AI-Generated Code Packages Can Lead to Slopsquatting Threat (DevOps.com)](https://devops.com/ai-generated-code-packages-can-lead-to-slopsquatting-threat-2/)
- [OWASP Top 10 for LLM Applications 2025](https://owasp.org/www-project-top-10-for-large-language-model-applications/)
- [OWASP LLM Top 10: How it Applies to Code Generation (Sonar)](https://www.sonarsource.com/resources/library/owasp-llm-code-generation/)
- [The Coding Personalities of Leading LLMs (SonarSource)](https://www.sonarsource.com/the-coding-personalities-of-leading-llms.pdf)
- [GenKubeSec: LLM-Based Kubernetes Misconfiguration Detection](https://arxiv.org/html/2405.19954v1)
- [KubeGuard: LLM-Assisted Kubernetes Hardening](https://arxiv.org/abs/2509.04191)
- [Security Degradation in Iterative AI Code Generation (arXiv)](https://arxiv.org/pdf/2506.11022)
- [GitHub Copilot's Code Review: Can AI Spot Security Flaws? (arXiv)](https://arxiv.org/html/2509.13650v1)
- [Security Risks of Vibe Coding and LLM Assistants (Kaspersky)](https://www.kaspersky.com/blog/vibe-coding-2025-risks/54584/)
- [The Risks of Hardcoding Secrets in Code Generated by LLMs (Cycode)](https://cycode.com/blog/the-risks-of-hardcoding-secrets-in-code-generated-by-language-learning-models/)
- [Security Flaws in DeepSeek-Generated Code (CrowdStrike)](https://www.crowdstrike.com/en-us/blog/crowdstrike-researchers-identify-hidden-vulnerabilities-ai-coded-software/)

108
harnesses/contexts/planning/v1/best-practices/secrets-management.md Normal file
View File

@@ -0,0 +1,108 @@
# Secrets Management
## SOPS + age
SOPS with age encryption is the standard across all projects. A single `.sops.yaml` at the repo root defines path-based encryption rules.
### File Naming
- `.sops.yaml` path-based rules match specific filename patterns (e.g., `**/*secret*.yaml`)
- Non-secret files must NOT contain `secret` in their name, or the pre-commit hook will encrypt them
- KSOPS generator files should be named `ksops-generator.yaml`, not `secret-generator.yaml`
### encrypted_regex Gotcha
When using `encrypted_regex` for selective field encryption (e.g., Ansible group_vars), variable names must contain a keyword that matches the regex (e.g., `password|private_key|api_key|secret|token`). Arbitrary key names are silently left unencrypted.
### SOPS Vars Plugin
Each Ansible project needs `vars_plugins_enabled = host_group_vars,community.sops.sops` in `ansible.cfg`. Files in `group_vars/` must be named after a group (e.g., `all.sops.yaml`), not arbitrary names.
### Interactive Editor Pitfalls
- `sops <file>` opens an interactive editor — fails in non-interactive sessions
- `sops -e /tmp/file` fails when the temp path doesn't match `.sops.yaml` rules
- Multiple `sops --set` calls can corrupt files — use the interactive editor for multi-field edits
## Credential Handling
- **Never pass secrets via command-line arguments** — visible in `ps` output
- Use `@file` references, environment variables sourced at runtime, or stdin
- For Ansible, use temp files with `trap rm` cleanup: `-e "@${tmpfile}"`
- Read secrets at execution time and use them ephemerally — never cache or persist values
- Reference the **existence** of a secret file in docs, never its contents
## Bootstrap Secrets
Some secrets are chicken-and-egg (e.g., the age decryption key for ArgoCD's KSOPS). These must be created manually as a bootstrap step and documented clearly.
## Credential Lifecycle Management
- **Track credential expiry dates.** OAuth client secrets, API tokens, and certificates have expiry dates that can cause silent failures. Document expiry dates when creating credentials.
- **Set alerts before expiry.** For long-lived credentials (e.g., 720-day OAuth client secrets), set calendar reminders or automated monitoring alerts well before they expire.
- **Rotation plan.** Know the rotation procedure before you need it — some credential types (e.g., Azure app registrations) require coordinated updates across multiple systems.
## Multi-Field Secret Files
Secret files that contain multiple fields (e.g., repo URL, token, username) cannot be used as bare values. Consumers must parse individual fields (e.g., `grep + awk` or structured YAML/JSON parsing).
The multi-field format is preferable because it's self-documenting — all related credentials live together. But any automation reading the file needs extraction logic, not just `cat`.
## Generating Secrets with gen-secret
Use the `gen-secret` script (from `small-scripts`, symlinked to `~/sbin/gen-secret`) to generate cryptographically random strings that are safe for bash, YAML, and JSON without escaping.
### Workflow: Generate + SOPS Encrypt
1. **Generate the secret value** — use `gen-secret` with an appropriate length:
```bash
SESSION_SECRET=$(gen-secret 48) # 48-char session key
API_KEY=$(gen-secret 20) # 20-char access key
API_SECRET=$(gen-secret 40) # 40-char secret key
```
2. **Write plaintext YAML to the target path** — the file must be at the path matched by `.sops.yaml` rules (e.g., `**/*secret*.yaml`):
```bash
cat > path/to/my-secret.sops.yaml <<EOF
apiVersion: v1
kind: Secret
metadata:
name: my-credentials
namespace: my-namespace
type: Opaque
stringData:
SESSION_SECRET: ${SESSION_SECRET}
API_KEY: ${API_KEY}
EOF
```
3. **Encrypt in-place** — SOPS reads `.sops.yaml` to determine the encryption key and regex:
```bash
sops -e -i path/to/my-secret.sops.yaml
```
4. **Verify** — decrypt and confirm no placeholders remain:
```bash
sops -d path/to/my-secret.sops.yaml
```
### Key Points
- **Always encrypt at the target path.** `sops -e /tmp/file.yaml` fails because `/tmp/` doesn't match `.sops.yaml` path rules. Write the plaintext to the final location, then `sops -e -i` in-place.
- **Use shell variables, not files, for ephemeral secrets.** Generate into a variable (`SECRET=$(gen-secret 48)`), interpolate into the YAML, then encrypt. The plaintext never touches disk as a standalone file.
- **Appropriate lengths:** 32 chars is the default and sufficient for most use cases. Use 48+ for session secrets, 20 for access key IDs, 40 for secret keys (matching common API patterns).
- **For credentials from external systems** (e.g., Gitea API tokens, registry passwords), read them from `~/dev/claude/secrets/` at point of use — don't generate random replacements for values that must match an external system.
- **Clean up temp files** if you write plaintext to a temporary location. Use `trap` cleanup or `rm -f` after encryption.
### Replacing Placeholder Secrets
When SOPS-encrypted files contain placeholder values (e.g., `PLACEHOLDER_SESSION_SECRET`):
1. Decrypt: `sops -d secret.sops.yaml` — inspect current values
2. Write the corrected plaintext YAML to the same path (overwriting the encrypted file)
3. Re-encrypt: `sops -e -i secret.sops.yaml`
4. Verify: `sops -d secret.sops.yaml | grep -c PLACEHOLDER` — should return 0
## Backup Considerations
Backup plans must include encryption keys (age private keys, etc.) so that encrypted data in Git repos remains recoverable.

79
harnesses/contexts/planning/v1/best-practices/security-architecture.md Normal file
View File

@@ -0,0 +1,79 @@
# Security Architecture
## The Server Boundary Rule
**No server-side credential may cross the server boundary to the client. Ever.**
This is a hard line, not a guideline. The only credentials that cross the boundary between client and server are the client's own identity credentials (MFA tokens, login passwords, OIDC tokens, etc.) — and these flow from client to server, never the reverse.
### What this means in practice
- **API tokens stay server-side.** If a browser-based application needs to call a third-party API (Gitea, S3, database, etc.), it calls a backend proxy that holds the token. The token never appears in JavaScript, localStorage, cookies, or any client-accessible storage.
- **Service account credentials stay server-side.** Database passwords, S3 access keys, webhook secrets, SMTP credentials — these are mounted into server-side containers via Kubernetes Secrets or environment variables and never exposed to clients.
- **OAuth tokens for third-party services stay server-side.** If a user authenticates with Service A and the application needs to call Service B on their behalf, the application's backend holds the Service B credentials. The client only ever sees its own session token with the application.
- **Per-user tokens mapped server-side.** When individual users need distinct third-party access (e.g., per-user Gitea tokens for audit trails), the mapping from user identity to their token lives on the server. The client authenticates with its own identity (e.g., via Authelia MFA), and the server looks up the appropriate third-party token.
### The identity exception
The only credentials that legitimately cross from client to server:
- **Username + password** — the user's own login credentials
- **MFA tokens** — TOTP codes, WebAuthn assertions, security key responses
- **OIDC/OAuth tokens** — tokens that represent the user's identity with the application itself (not with third-party services)
- **Session cookies/JWTs** — issued by the server to represent an authenticated session
These all share the property: they are the user's own identity, flowing from client to server for authentication purposes.
### Architecture pattern: proxy with identity mapping
When a client-side application (SPA, CMS, admin UI) needs to interact with a backend service that requires credentials:
```
Client ──→ Auth Layer (MFA) ──→ API Proxy ──→ Backend Service
│
├── Reads user identity from auth headers
├── Looks up user's backend credential
├── Forwards request with backend credential
└── Returns response (without credential)
```
The proxy:
1. Sits behind the authentication layer (Authelia, OAuth2 Proxy, etc.)
2. Reads the authenticated user's identity from trusted headers (e.g., `Remote-User`)
3. Maps the identity to the appropriate backend credential
4. Makes the backend API call with the credential
5. Returns the response — never the credential
### Security layers (defense in depth)
A well-designed proxy architecture has multiple independent security layers:
1. **Authentication** — User must prove their identity (MFA, OIDC)
2. **Session validation** — Proxy validates the session is current and legitimate
3. **Authorization** — Proxy checks the user has access to the requested resource
4. **Backend permissions** — The backend service enforces its own access controls
5. **Branch/scope protection** — Fine-grained controls prevent privilege escalation (e.g., branch protection rules)
Each layer is independent — compromising one doesn't bypass the others.
### Anti-patterns
- **Passing API tokens to the browser via OAuth.** Even with PKCE, the token ends up in client-accessible storage. Use a backend proxy instead.
- **Shared service account tokens.** One token for all users means no audit trail and no granular revocation. Map per-user tokens server-side.
- **Embedding credentials in client-side config.** API keys in `config.js`, `.env` files served statically, or hardcoded in HTML — all violate the boundary rule.
- **Forwarding backend tokens via API responses.** Even "temporarily" returning a token in a response body breaks the rule. The client should never see it.
- **Using the same token for client auth and backend calls.** The user's session token with your application is distinct from any token your application uses to call backend services.
### When credentials must be client-side
Some scenarios genuinely require client-side credentials (e.g., direct S3 uploads for large files, WebRTC signaling). In these cases:
- Use **presigned URLs** or **temporary credentials** with the narrowest possible scope and shortest possible lifetime
- The presigning/credential-issuance happens server-side
- The temporary credential is scoped to exactly one operation (e.g., upload one file to one path)
- Log the issuance server-side for audit
### Real-world example: CMS editing
**Wrong:** CMS authenticates directly with Gitea via OAuth popup. Gitea token lands in browser localStorage. CMS makes API calls directly to Gitea with the token.
**Right:** CMS sits behind Authelia (MFA). A proxy service intercepts the OAuth flow, issues a proxy session token (containing only the user's identity), and forwards all API calls to Gitea using a per-user server-side Gitea token. The Gitea token never leaves the server.

250
harnesses/contexts/planning/v1/best-practices/spec-driven-development.md Normal file
View File

@@ -0,0 +1,250 @@
# Spec-Driven Development with AI Agents
Best practices for using structured specifications to coordinate AI agent implementation work. Extracted from real project experience (agent-runtimes) and industry research (OpenSpec, Codified Context paper, Addy Osmani's workflow guides).
## Why Specs Matter for AI Agents
AI agents trust documentation absolutely. A well-written spec gives an agent everything it needs to implement a subsystem without reading the entire codebase. A stale or vague spec causes silent failures where agents generate code that is structurally valid but architecturally wrong.
Specs serve three functions that CLAUDE.md alone cannot:
1. **Compressed context** — an agent reads one spec, not 300 lines of mixed concerns
2. **Testable contracts** — numbered requirements and scenarios translate directly to pytest
3. **Handoff boundaries** — an agent working on the dispatcher doesn't need to understand the entrypoint internals, just the interface between them
## Spec Structure
Each spec follows a consistent template. Sections are ordered so an agent can read top-down and build understanding progressively.
### Required Sections
1. **Overview** — What this subsystem does. 2-3 sentences. An agent should know if this spec is relevant after reading this.
2. **Responsibilities** — What this subsystem owns and what it delegates. Prevents scope creep during implementation.
3. **Dependencies** — Which other specs to read first. Keeps the reading list minimal.
4. **Data Model** — Types, schemas, state machines, interfaces. The concrete contract.
5. **Requirements** — Numbered functional requirements (e.g., E-1, E-2). Each must be independently testable.
6. **Scenarios** — Concrete given/when/then examples. These become test functions.
### Optional Sections
7. **Interface** — API surface, function signatures, HTTP endpoints. Include when the subsystem has an external-facing API.
8. **Extension Points** — How to add new capabilities without modifying existing code. Step-by-step instructions.
9. **Error Handling** — Failure modes and expected behaviour. Prevents agents from inventing their own error strategies.
### Writing Guidelines
- **Be specific, not comprehensive.** A spec that says "handle errors appropriately" is useless. A spec that says "return exit code 124 on timeout" is testable.
- **Include the why.** Design intent and constraints prevent agents from making structurally valid but architecturally wrong changes. Requirements without rationale are followed mechanically — agents can't judge edge cases or make trade-offs. Every constraint needs a "Why:" line. Example: `"Secrets never in payload"` needs `"because payloads may be logged and stored in task history"`.
- **Use concrete examples.** Every data model section should include a realistic JSON/code example, not just a schema.
- **Cross-reference, don't duplicate.** If two specs share a concept (e.g., the payload schema), one spec owns it and the other links to it.
- **Keep each spec self-contained.** An agent should be able to implement a subsystem by reading the target spec plus its listed dependencies. If it needs to read CLAUDE.md, the spec is incomplete.
## Requirement Numbering
Each spec uses a short prefix derived from its name, followed by a sequential number:
| Spec | Prefix | Example |
|---|---|---|
| payload.md | P | P-1, P-2 |
| entrypoint.md | E | E-1, E-2 |
| actions.md | A | A-1, A-2 |
| runners.md | R | R-1, R-2 |
| dispatcher.md | D | D-1, D-2 |
| control-plane.md | CP | CP-1, CP-2 |
Requirements must be:
- **Independently testable** — each maps to one or more test functions
- **Unambiguous** — an agent can determine pass/fail without human judgement
- **Stable** — changing a requirement number invalidates tests, so avoid renumbering
## Scenarios as Test Blueprints
Every scenario in a spec should be directly translatable to a test function. Use this format:
```markdown
### Scenario: Pre-action failure
**Given:** Payload with clone pre-action (invalid repo URL)
**When:** Clone fails (git returns non-zero)
**Then:** on_error actions run, container exits 1. Runner never executes.
```
This becomes:
```python
def test_scenario_preaction_failure_runs_on_error_and_exits_1(...):
"""Given clone fails, on_error runs and exits 1. Runner never executes."""
```
Guidelines:
- Each scenario tests one behaviour, not a combination
- Include both happy paths and error paths
- Name the scenario descriptively — it becomes the test function's docstring
- Include enough setup detail that an agent can write the test without guessing
## The Spec → Test → Code Workflow
This is the core development loop. Tests are written from the spec before code exists.
### 1. Write or Update the Spec
Define requirements and scenarios. Get them reviewed. The spec is the source of truth for what the system should do.
### 2. Write Tests from the Spec
Translate requirements and scenarios into pytest functions. Tests should:
- Map to requirement IDs in their names: `test_e4_preaction_failure_skips_remaining`
- Use the scenario's given/when/then as the test body structure
- Mock external dependencies (subprocess, HTTP, filesystem)
- Run fast (no Docker, no network, no real APIs)
### 3. Run the Tests — They Should All Fail
This confirms the tests are actually testing something. If a test passes before implementation, it's either testing the wrong thing or the feature already exists.
### 4. Implement Until Tests Pass
Write the minimum code to make tests pass. The spec defines what, the tests verify it, the code implements it.
### 5. Update Spec if Implementation Reveals Issues
Sometimes implementation reveals that a requirement is unworkable or incomplete. Update the spec, update the test, then update the code. The spec stays authoritative.
## Spec Maintenance
### Preventing Drift
Specs drift from code when they're treated as planning documents that are "done" after implementation. They must be treated as living contracts.
**Rules:**
- **Spec changes require test changes.** If a requirement changes, its test must change in the same commit.
- **Code changes that affect interfaces require spec changes.** If a function signature, API endpoint, or data schema changes, the relevant spec must be updated in the same commit.
- **New features require spec-first.** Add the requirement and scenario to the spec, write the test, then implement.
### CI Enforcement
Enforce spec hygiene with automated checks:
1. **Pre-commit hook** — run pytest, block commit on failure (already implemented)
2. **Spec coverage check** — a script that verifies every numbered requirement has at least one test function referencing it
3. **Orphan test detection** — tests referencing requirement IDs that no longer exist in specs
Before completing any milestone, manually walk through every requirement ID (e.g., CP-1..CP-20, TH-1..TH-13) and verify a corresponding test exists. Automated spec coverage checks catch this in CI, but a manual audit before milestone completion catches gaps that the automation might miss (stubs, placeholder tests, tests that reference the ID but don't actually test the requirement).
### Review Checklist
When reviewing a PR that touches a spec subsystem:
- [ ] Spec updated if interface or behaviour changed
- [ ] Test added/updated for new/changed requirements
- [ ] Cross-references still valid
- [ ] No requirements removed without deprecation note
## Context Architecture for Agents
Based on the Codified Context paper (108k-line system, 283 sessions), structure project knowledge in three tiers:
### Tier 1: Hot Context (Always Loaded)
CLAUDE.md — conventions, env vars, repo structure, scripts. Loaded every session. Keep under ~300 lines by linking to details elsewhere.
### Tier 2: Spec Context (Per-Task)
`spec/` files — loaded based on what the agent is working on. An agent implementing a new action reads `spec/actions.md` + `spec/payload.md`. An agent working on the dispatcher reads `spec/dispatcher.md` + `spec/container-backends.md`.
The spec index (SPEC.md) has a "read this when..." column to guide selection.
### Tier 3: Cold Context (On-Demand)
`memory/` files — gotchas, reflections, decisions. Loaded only when relevant. An agent hitting a weird Cilium issue checks `memory/gotchas-cilium.md`.
### Routing Context to Agents
When launching an agent to work on a subsystem:
1. Point it at the relevant spec(s) via its prompt
2. Include CLAUDE.md for conventions
3. Let it pull from memory/ on-demand if it hits issues
Don't load everything — agents perform better with focused context than with a 50-page dump.
## Testing Depth
The spec→test→code workflow defines *when* to write tests. For *how* to write comprehensive tests — edge case discovery, property-based testing, mutation testing, AI agent testing patterns — see [Test-Driven Development](test-driven-development.md).
## Post-Write Spec Audit
After writing specs, audit them against best practices before implementation. Common gap categories:
1. **Missing rationale** — Constraints without "Why:" lines. Agents follow them mechanically but can't judge edge cases.
2. **Missing error/failure scenarios** — Happy paths are covered but failure modes aren't specified.
3. **Cross-spec interface misalignment** — Two specs describe the same interface differently.
4. **Vague requirements** — "Handle errors appropriately" instead of specific error codes and behaviours.
5. **Missing specs for discovered subsystems** — Implementation reveals components that weren't planned for.
Write-then-audit is more productive than trying to get specs perfect on the first pass. The audit step catches systematic gaps across all specs at once.
## Planning Session Limits
Architecture decisions, infrastructure research, and spec refinement each get one planning session. After three sessions of planning, start implementation. Specs are hypotheses that need code to validate them — extended planning without implementation produces diminishing returns and theoretical designs that don't survive contact with reality.
## Categorize Findings Before Acting
When a spec review or audit produces many findings, categorize them by priority (high/medium/low) before making changes. Present the categorized list for alignment before editing. Starting edits without prioritization leads to scope creep — low-priority cosmetic fixes consume time that should go to high-priority structural gaps.
## Multi-Agent Orchestration Practices
### Commit WIP Before Decomposing Tasks
Untracked and uncommitted files are NOT available in git worktrees. If agents work in worktrees (or container-mounted worktrees), they won't see specs, plans, or dependency outputs that haven't been committed. Commit to a staging branch before decomposition — this eliminates the dominant overhead of manually copying files into each worktree.
### Agents Must Self-Verify with Tests
Add "Run tests and fix any failures" to every implementation agent prompt. Agents that write code without running tests produce bugs that only surface during assembly. Self-verification catches issues while the agent still has full context of what it wrote.
### State Import and Style Conventions Explicitly
Agents default to standard language conventions (e.g., relative Python imports, standard packaging). If the project uses non-standard patterns (bare imports, specific naming conventions, module-level structure), state them explicitly in the prompt. A single line like "Use `from harness import X`, not `from .harness import X`" prevents import mismatches during assembly.
### Budget for Assembly Fixups
Parallel agent work produces ~3 fixups per orchestration run, each under 5 minutes. Common fixup categories: import conventions, module-level side effects, SDK exception constructor signatures, validator patterns. This is the expected cost of parallel work, not a failure. Budget 15-20 minutes for assembly and fixup after each orchestration run.
### Two-Phase Orchestration: Specs First, Then Implementation
When orchestrating multi-agent work for a milestone, decompose in two phases:
1. **Phase 1:** Spec-writing agents produce the contracts (using the plan as input).
2. **Review:** Human reviews specs for cross-spec consistency before proceeding.
3. **Phase 2:** Implementation agents receive actual spec files (not plan descriptions).
This works significantly better than defining all tasks upfront because spec agents validate the plan against reality, the review step catches cross-spec inconsistencies, and implementation agents work from concrete contracts rather than plan summaries.
### Include an Integration Verification Task After Orchestration
Agent orchestration leaves integration gaps at component boundaries. Each agent completes its assigned scope correctly, but nobody owns the integration points between them (e.g., stub comments, ORM mapping methods not updated for new fields). After every orchestration run, include an explicit integration verification step that checks cross-component contracts — call sites, shared data models, and handoff points.
### Decompose Along File Boundaries
When splitting work into parallel agent tasks, ensure each task writes to distinct files. When two agents must modify the same file, make the shared changes small and predictable — identify the conflict point upfront so the merge is trivial. File-boundary decomposition produces zero-conflict assemblies.
### Choose Manual Implementation for Tightly-Coupled Cross-Component Work
When changes are small per file (5-15 lines) but tightly coupled across many files (each change depends on the previous), skip agent orchestration and implement manually. The assembly overhead exceeds the implementation time. Agent orchestration excels when tasks are independent and substantial; manual implementation excels when work is sequential and interconnected.
## Anti-Patterns
### Specs as documentation, not contracts
**Symptom:** Specs describe what was built, updated after the fact. Tests don't reference spec IDs.
**Fix:** Write specs before code. Tests reference requirement IDs. Specs are the input, not the output.
### Mega-spec
**Symptom:** One large spec covering the entire system. Agents must read thousands of lines to find what they need.
**Fix:** Split by subsystem. Each spec should be readable in under 5 minutes.
### Spec without scenarios
**Symptom:** Requirements are abstract ("handle errors gracefully"). No concrete examples.
**Fix:** Every requirement needs at least one given/when/then scenario with specific inputs and outputs.
### Implementation details in specs
**Symptom:** Spec dictates variable names, algorithm choices, internal data structures.
**Fix:** Specs define what and why, not how. The interface is specified; the implementation is free.
### Untested requirements
**Symptom:** Requirements exist in the spec but no test references them. They drift without anyone noticing.
**Fix:** Spec coverage check in CI. Every requirement ID must appear in at least one test function name.

486
harnesses/contexts/planning/v1/best-practices/test-driven-development.md Normal file
View File

@@ -0,0 +1,486 @@
# Test-Driven Development for Spec-Based Projects
Best practices for writing comprehensive, regression-catching tests in projects that use structured specifications. Focuses on maximising test value (catching real bugs) rather than test volume (inflating coverage numbers). Extracted from industry research, academic papers (TDAD, Codified Context), and practitioner experience.
## Core Principle: Tests Are the Spec's Enforcement Layer
In a spec-driven project, the spec defines *what* and the tests *prove it*. A requirement without a test is an aspiration. A test without a requirement is undocumented behaviour. Keep them tightly coupled:
- Every numbered requirement (P-1, E-3) has at least one test
- Every test function name includes its requirement ID: `test_e3_preaction_failure_exits_1`
- Spec changes and test changes ship in the same commit
## Deriving Tests from Specs
### Requirements to Tests
Each spec requirement becomes one or more test functions. The mapping isn't always 1:1 — a requirement like "must respect timeout" needs tests for: default timeout, explicit timeout, timeout=0 (no limit), timeout exceeded.
```python
# From spec: R-4: Runners must respect runtime.timeout.
# Default 3600s. Value of 0 means no timeout.
def test_r4_default_timeout_is_3600():
"""R-4: When timeout not specified, default is 3600s."""
def test_r4_explicit_timeout_is_honoured():
"""R-4: When timeout=60, process killed after 60s."""
def test_r4_zero_timeout_means_no_limit():
"""R-4: When timeout=0, no timeout is applied."""
def test_r4_timeout_returns_exit_code_124():
"""R-4 + R-5: Timeout produces exit code 124."""
```
### Scenarios to Tests
GIVEN/WHEN/THEN scenarios translate directly to Arrange/Act/Assert:
```python
def test_scenario_preaction_failure_runs_on_error():
"""Given clone fails, on_error runs and exits 1. Runner never executes."""
# GIVEN — arrange
payload = make_payload(pre_actions=[{"action": "clone", "repo": "bad-url"}])
mock_clone = Mock(side_effect=subprocess.CalledProcessError(128, "git"))
# WHEN — act
exit_code = run_entrypoint(payload, clone_handler=mock_clone)
# THEN — assert
assert exit_code == 1
mock_runner.assert_not_called()
mock_on_error.assert_called_once()
```
### Parameterised Tests from Spec Enumerations
When a spec lists multiple valid values, use `@pytest.mark.parametrize`:
```python
# From spec: task states are pending, assigned, running, succeeded, failed, timed_out, cancelled
@pytest.mark.parametrize("terminal_state", ["succeeded", "failed", "timed_out", "cancelled"])
def test_cp_terminal_state_cannot_be_overwritten(terminal_state):
"""CP: Terminal states reject further transitions with 409."""
```
## Systematic Edge Case Discovery
~80% of bugs cluster at boundaries. Use these techniques to find edge cases systematically rather than by intuition.
### Boundary Value Analysis
For every input parameter, test at the edges of its valid range:
| Input type | Test values |
|---|---|
| Integer (range 1-100) | 0, 1, 2, 99, 100, 101, -1, MAX_INT |
| String | `""`, `"a"`, max-length string, max+1, unicode (`"\u0000"`, emoji), whitespace-only |
| List/Array | `[]`, `[single]`, many items, duplicates, `None` |
| Dict/Map | `{}`, missing required keys, extra unknown keys, `None` values |
| Timeout (seconds) | 0, 1, -1, very large (999999), `None`/missing |
| Base64 | valid, invalid chars, empty, padding variants (`=`, `==`, none) |
### Equivalence Partitioning
Group inputs into classes where all members should behave identically. Test one from each class:
```python
# Payload validation: prompt field
# Class 1: valid string → accepted
# Class 2: empty string → rejected (spec says prompt is required)
# Class 3: missing key → rejected
# Class 4: wrong type (int, list, None) → rejected
# Class 5: very long string → accepted (no length limit in spec)
@pytest.mark.parametrize("prompt,should_pass", [
("Fix the bug", True), # Class 1: valid
("", False), # Class 2: empty
(None, False), # Class 3: missing/None
(42, False), # Class 4: wrong type
("x" * 100_000, True), # Class 5: long string
])
def test_p_prompt_validation(prompt, should_pass):
...
```
### State Transition Coverage
For state machines (task states, dispatcher states), test:
1. **Every valid transition:** `pending → assigned → running → succeeded`
2. **Every invalid transition:** `succeeded → running` (should be rejected)
3. **Initial state:** newly created tasks start in `pending`
4. **Terminal states:** `succeeded`, `failed`, `timed_out`, `cancelled` cannot transition further
5. **Re-entrant transitions:** same state → same state (should be idempotent or rejected, per spec)
```python
VALID_TRANSITIONS = [
("pending", "assigned"),
("assigned", "running"),
("running", "succeeded"),
("running", "failed"),
("running", "timed_out"),
("assigned", "cancelled"),
("running", "cancelled"),
]
INVALID_TRANSITIONS = [
("succeeded", "failed"),
("failed", "running"),
("cancelled", "pending"),
("timed_out", "running"),
]
@pytest.mark.parametrize("from_state,to_state", VALID_TRANSITIONS)
def test_valid_state_transition(from_state, to_state):
...
@pytest.mark.parametrize("from_state,to_state", INVALID_TRANSITIONS)
def test_invalid_state_transition_rejected(from_state, to_state):
...
```
### The Edge Case Checklist
Walk through this for every function under test:
1. **Empty/null inputs** — what happens when required fields are missing?
2. **Boundary values** — min, max, zero, negative, off-by-one
3. **Type mismatches** — string where int expected, list where dict expected
4. **Malformed input** — invalid JSON, bad base64, truncated data
5. **Concurrent operations** — two tasks claiming the same resource
6. **Ordering** — actions that depend on sequence (pre-action before runner)
7. **Idempotency** — calling the same operation twice (kill an already-killed container)
8. **Resource exhaustion** — at capacity, disk full, timeout expired
9. **Partial failure** — first action succeeds, second fails (cleanup?)
## Property-Based Testing with Hypothesis
Instead of specifying individual test cases, define *properties* that must hold for all inputs. Hypothesis generates hundreds of inputs including edge cases you'd never think of.
### When to Use Property-Based Testing
- **Serialisation roundtrips:** encode → decode returns original
- **Parsers:** should never crash on any input
- **Data transformations:** invariants that hold regardless of input
- **Validators:** valid inputs accepted, invalid inputs rejected (never crash)
### When NOT to Use It
- Tests where generating valid inputs is harder than the code itself
- Tests where the "property" just restates the implementation
- UI or integration tests
### Patterns
```python
from hypothesis import given, strategies as st, assume, settings
from hypothesis import example
# Roundtrip: base64 encode/decode preserves payload
@given(st.text())
def test_base64_roundtrip(payload_str):
encoded = base64.b64encode(payload_str.encode()).decode()
decoded = base64.b64decode(encoded).decode()
assert decoded == payload_str
# Invariant: payload validation never crashes (may reject, never exception)
@given(st.dictionaries(st.text(), st.text() | st.integers() | st.none()))
def test_payload_validation_never_crashes(raw_payload):
# Should return True/False or raise ValidationError — never unhandled exception
try:
validate_payload(raw_payload)
except ValidationError:
pass # Expected for invalid input
# Pin known edge cases alongside random generation
@example("") # empty string
@example("\x00") # null byte
@example("a" * 10**6) # very long
@given(st.text())
def test_prompt_handling(prompt):
...
# Composite strategies for domain objects
@st.composite
def valid_payloads(draw):
return {
"task_id": draw(st.uuids()).hex,
"prompt": draw(st.text(min_size=1)),
"runtime": {"cli": draw(st.sampled_from(["claude", "codex"]))},
}
@given(valid_payloads())
def test_valid_payload_always_accepted(payload):
assert validate_payload(payload) is True
```
### Stateful Testing for State Machines
Hypothesis can generate sequences of operations and check invariants after each step:
```python
from hypothesis.stateful import RuleBasedStateMachine, rule, precondition
class TaskStateMachine(RuleBasedStateMachine):
def __init__(self):
super().__init__()
self.task = Task(state="pending")
@rule()
@precondition(lambda self: self.task.state == "pending")
def assign(self):
self.task.transition("assigned")
assert self.task.state == "assigned"
@rule()
@precondition(lambda self: self.task.state == "running")
def complete(self):
self.task.transition("succeeded")
assert self.task.state == "succeeded"
# Invariant: terminal states never change
@invariant()
def terminal_states_are_final(self):
if self.task.state in ("succeeded", "failed", "cancelled"):
with pytest.raises(InvalidTransition):
self.task.transition("running")
TestTaskStates = TaskStateMachine.TestCase
```
## Mutation Testing
Mutation testing answers: "If someone introduced a bug, would our tests catch it?"
Tools make small code changes (replacing `>` with `>=`, `True` with `False`, deleting statements) and check if tests still pass. Surviving mutants = test gaps.
### Setup with mutmut
```toml
# pyproject.toml
[tool.mutmut]
paths_to_mutate = "entrypoint/"
tests_dir = "tests/"
runner = "python -m pytest tests/ -x -q"
```
```bash
# Run mutation testing
mutmut run
# See surviving mutants
mutmut results
# Inspect a specific mutant
mutmut show 42
```
### Practical Guidance
- **Target: mutation score above 80%.** Scores above 90% have diminishing returns (equivalent mutants).
- **Focus on business logic** — validators, state machines, parsers. Skip glue code.
- **Use mutation testing to audit AI-generated tests.** This is the most powerful combination: AI writes tests from spec, mutation testing verifies those tests catch real faults.
- **Run on changed files only in CI** (full suite is slow). Full run nightly or pre-release.
## Test Architecture
### The Testing Pyramid for Spec-Driven Projects
| Layer | Proportion | Speed | What it catches |
|---|---|---|---|
| Unit tests | 60-70% | <1ms each | Logic errors, boundary violations, state machine bugs |
| Property-based | 10-15% | ~10ms each | Edge cases humans miss, roundtrip failures, crash inputs |
| Integration | 15-20% | ~100ms each | Component interaction bugs, mock/reality divergence |
| E2E / acceptance | 5-10% | ~1s+ each | Full-chain failures, deployment config issues |
### Test Isolation Principles
- **No test depends on another test's state.** Each test sets up its own preconditions.
- **No test depends on execution order.** `pytest-randomly` catches order dependencies.
- **No test touches the real filesystem outside `tmp_path`.** Monkeypatch paths that default to production locations (like `/workspace`).
- **No test makes network calls.** Mock HTTP, subprocess, and socket calls.
- **Integration tests are marked** (`@pytest.mark.integration`) and excluded by default.
### Fixture Architecture
```python
# conftest.py — shared fixtures, not test logic
@pytest.fixture
def minimal_payload():
"""Smallest valid payload — tests shouldn't need more unless testing specific fields."""
return {"task_id": "test-123", "prompt": "do something", "runtime": {"cli": "claude"}}
@pytest.fixture
def encode_payload():
"""Helper: dict → base64 string (how the dispatcher passes payloads)."""
def _encode(d):
return base64.b64encode(json.dumps(d).encode()).decode()
return _encode
# Per-module conftest for module-specific fixtures
# tests/test_dispatcher/conftest.py
@pytest.fixture
def mock_backend():
"""Fake container backend that records calls without Docker."""
...
```
### Negative Tests Are as Important as Positive Tests
For every "this works" test, write at least one "this fails correctly" test:
```python
# Positive: valid payload accepted
def test_p1_valid_payload_loads():
...
# Negative: missing required field rejected
def test_p3_missing_prompt_raises():
...
# Negative: wrong type rejected
def test_p_prompt_wrong_type_raises():
...
# Negative: extra unknown fields are ignored (not rejected)
def test_p_unknown_fields_ignored():
...
```
## AI Agent Testing Patterns
### The Two-Phase Rule
**Never let the same agent write both tests and implementation in one pass.** An agent that writes tests and code together will unconsciously write tests that verify its own broken assumptions.
The workflow:
1. **Phase 1:** Agent reads spec → writes tests. Human reviews tests against spec.
2. **Phase 2:** Agent (or different agent) reads spec + tests → writes implementation until tests pass.
### Hidden Test Splits
Hold back some tests that the implementing agent never sees. Use them as a final validation:
```python
# tests/test_payload.py — agent sees these during development
def test_p1_load_from_env_var(): ...
def test_p2_missing_payload_exits_1(): ...
# tests/test_payload_hidden.py — agent never sees these, run post-implementation
# (Marked with a custom marker, excluded from default run)
@pytest.mark.hidden
def test_p1_load_from_file_fallback(): ...
@pytest.mark.hidden
def test_p_concurrent_payload_loads(): ...
```
### Regression Tests from Real Bugs
Every bug found in production or during integration testing becomes a permanent test case:
```python
def test_regression_crlf_corruption():
"""Regression: smtp-oauth-relay converted \\r\\n to \\n, breaking quoted-printable.
Fixed by as_bytes(policy=email_policy.SMTP). See memory/gotchas-email-relay.md."""
...
```
These are the highest-value tests because they catch proven failure modes.
## Test Quality Metrics
### What to Measure
| Metric | Target | Why |
|---|---|---|
| Spec coverage | 100% | Every numbered requirement has at least one test |
| Mutation score | >80% | Tests catch real faults, not just inflate coverage |
| Line coverage | >90% | Baseline hygiene (necessary but not sufficient) |
| Test speed | <10s total | Fast enough for pre-commit hooks |
| Assertion density | >1 per test | Tests that don't assert don't catch anything |
### What NOT to Measure
- **100% line coverage as a goal.** Chasing 100% leads to tests that exercise code paths without meaningful assertions.
- **Test count.** 50 well-targeted tests beat 200 shallow ones.
- **Test-to-code ratio.** The ratio depends on the module's complexity, not a universal number.
## CI Integration
### Pre-commit (Every Commit)
```bash
pytest tests/ -x -q --tb=short -m "not integration"
```
### PR Validation (Every Push)
```bash
# Unit + property-based tests
pytest tests/ -q --tb=short -m "not integration"
# Mutation testing on changed files only
mutmut run --paths-to-mutate="$(git diff --name-only main... | grep '.py$' | tr '\n' ',')"
```
### Nightly
```bash
# Full mutation testing
mutmut run
# Integration tests (requires Docker)
pytest tests/ -m integration
# Hidden test validation
pytest tests/ -m hidden
```
## Python Testing Gotchas
### `subprocess.run(check=True)` Is Invisible to Mocks
When you mock `subprocess.run`, the mock replaces the entire function — including the `check=True` logic that raises `CalledProcessError`. A mock returning `CompletedProcess(returncode=1)` won't trigger the exception even though the real code uses `check=True`. To test failure paths, use `side_effect=CalledProcessError(...)` explicitly.
### Use Routing Callables for Multi-Call Subprocess Mocks
When a function calls `subprocess.run` multiple times (e.g., git config, add, diff, commit, push), a fixed `side_effect` list is fragile and breaks when call order changes. Instead, use a routing callable that inspects the command: `mock_run.side_effect = lambda cmd, **kw: route_by_command(cmd)`. Clearer, more maintainable, and self-documenting.
### Pydantic v2 `@field_validator` Doesn't Fire for Default Values
`@field_validator('field_name')` never runs when the field takes its default value (e.g., `None`). Cross-field validation logic (e.g., "if type is X then field Y is required") silently passes when the dependent field is omitted. Use `@model_validator(mode='after')` for any validation that depends on multiple fields or needs to fire even when fields take defaults.
### Never `sys.exit()` at Module Level
`sys.exit()` in an `except ImportError` block at module level kills pytest collection entirely — all tests fail, not just the ones for that module. Use a flag pattern instead: `_HAS_DEPENDENCY = False` in the except block, then check `if not _HAS_DEPENDENCY: return 1` inside the function. This allows the module to be imported and mocked even when the optional dependency is missing.
### Use `pytest.importorskip` for Optional Dependency Tests
When test files import optional packages (e.g., `sqlalchemy`, `psycopg`) at module level, pytest collection fails for the entire test suite — not just the tests that need that package. Use `mod = pytest.importorskip("sqlalchemy")` and then attribute access (`mod.text`). Also guard transitive imports: `pytest.importorskip("myapp.db.postgres_store")` if the module itself imports the optional package at module level.
### Patch Individual Functions, Not Whole Modules
Patching an entire module (e.g., `patch("mod.kubernetes.config")`) replaces exception classes with MagicMock objects. `except SomeException` then catches `MagicMock` instead of the real exception, causing tests to pass the wrong code path. Patch individual functions (`load_incluster_config`, `load_kube_config`) and leave exception classes intact so `except` clauses work correctly.
### Async Migration Requires Full Test Conversion
When migrating a codebase from sync to async, helper functions get converted but test functions are often left as sync `def`. Every test that calls an async function needs `async def` + `@pytest.mark.asyncio` + `await`. After any async migration, run tests and grep for `RuntimeWarning: coroutine '...' was never awaited` to find remaining sync-to-async gaps.
## Anti-Patterns
### Tests that mirror implementation
**Symptom:** Test asserts that function calls happen in a specific order, using mock.assert_has_calls with exact sequences. Breaks on any refactor.
**Fix:** Test behaviour (inputs → outputs), not implementation details.
### Tests without assertions
**Symptom:** `test_it_runs()` calls the function and checks it doesn't crash. No assertion on the result.
**Fix:** Every test must assert something specific about the output, side effects, or raised exceptions.
### Overmocking
**Symptom:** Every dependency is mocked. Tests pass but integration fails because mocks don't match real behaviour.
**Fix:** Mock at the boundary (subprocess, HTTP, filesystem), not between your own modules. Use real objects for internal dependencies.
### Fragile tests
**Symptom:** Tests break when unrelated code changes. Usually caused by asserting on implementation details, shared mutable state, or execution order.
**Fix:** Test the public interface. Use fixtures for setup. Isolate each test completely.
### Testing private methods
**Symptom:** Tests import `_internal_helper` and test it directly. These break on any refactor.
**Fix:** Test through the public API. If a private method is complex enough to need its own tests, it should probably be a separate module with a public interface.

11
harnesses/contexts/planning/v1/harness.yaml Normal file
View File

@@ -0,0 +1,11 @@
kind: context
name: planning
version: 1
description: "Planning and spec writing methodology"
requires:
- best-practices/v1
provides: [planning-agent]
context_files:
- source: ./CLAUDE.md
target: /opt/harness/context/planning/CLAUDE.md