skynet/best-practices
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Populate best practices from claude-foundations

Browse Source 
Migrates 20 topic files from claude-foundations/best-practices/ to this
standalone repo. Adds BESTPRACTICES.md index, CLAUDE.md conventions, and
updated README.md. Container agents clone this repo to /best-practices.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Paul O'Reilly
2026-03-28 17:46:13 +13:00
parent 1ca7ecfe19
commit 3efe153ca1
23 changed files with 2236 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

26
BESTPRACTICES.md Normal file
View File

@@ -0,0 +1,26 @@
# Best Practices Index
Generalised best practices extracted from real project work via the `/distill-best-practices` skill. Each topic file is self-contained — read only the files relevant to the current project.
## Topics
- [Validation & Deployment](validation.md) — Validate locally, deploy once; full-chain testing; pre-flight checks
- [Security Architecture](security-architecture.md) — Server boundary rule: no credential crosses to the client; proxy + identity mapping pattern; defense in depth; anti-patterns
- [Secrets Management](secrets-management.md) — SOPS + age, credential handling, file naming, encryption gotchas
- [Git & Source Control](git-source-control.md) — Commit practices, GitOps workflows, remote conventions
- [Kubernetes Patterns](kubernetes.md) — Volume mounts, deployment strategies, naming, bootstrap ordering
- [Helm Charts](helm.md) — Schema validation, version verification, values structure
- [Ansible](ansible.md) — Inventory, templates, idempotency, credential safety
- [Scripting](scripting.md) — Shell conventions, verification scripts, idempotency, colour output
- [Documentation Standards](documentation.md) — CLAUDE.md, MEMORY.md, FUTURE.md, README.md structure and tiered memory
- [Milestones & Reflections](milestones.md) — Milestone workflow, verification, reflection process
- [Debugging Methodology](debugging.md) — Systematic diagnosis, full-chain testing, common pitfalls
- [Claude Code Skills](skills-development.md) — Skill authoring, context injection, tool restrictions
- [Linting & Formatting](linting.md) — Tool choices per language, PostToolUse hook, pre-commit integration, formatter contract
- [Spec-Driven Development](spec-driven-development.md) — Spec structure, requirement numbering, test-first workflow, context tiers, anti-patterns
- [Test-Driven Development](test-driven-development.md) — Edge case discovery, property-based testing, mutation testing, AI agent testing patterns, test architecture
- [Networking & Infrastructure](networking.md) — nftables safety, systemd socket activation, Docker forwarding, TLS SNI vs Host header, wildcard certs
- [Docker UID Matching](docker-uid-matching.md) — UID wrapper entrypoint for mounted volumes, gosu pattern, when to use vs K8s securityContext
- [Database Selection](database-selection.md) — SQLite is not a production database; always use PostgreSQL for services with FQDNs, multiple consumers, or concurrent access
- [Docker](docker.md) — gosu PID 1, GIT_SSH_COMMAND scope, slim image health checks, buildx local images, default users, TTY flags, UID resolution
- [Octopus Process Templates](octopus-process-templates.md) — OCL syntax, step template references, channel scoping, parameters, versioning, Platform Hub patterns

37
CLAUDE.md Normal file
View File

@@ -0,0 +1,37 @@
# CLAUDE.md — Best Practices
## Overview
Cross-project best practices extracted from real project work. Maintained by the `/distill-best-practices` skill, which reads memory files from all tracked projects and proposes updates here.
## Repository Structure
```
best-practices/
BESTPRACTICES.md # Index — lists all topic files with descriptions
CLAUDE.md # This file
README.md # Human-readable overview
<topic>.md # One self-contained file per topic
```
## Conventions
- **One file per topic.** Each file is self-contained and readable in isolation.
- **BESTPRACTICES.md is the index.** One line per topic with a link and description. Loaded by `context-load` at session start.
- **Distillation maintains this repo.** The `/distill-best-practices` skill reads memory files from tracked projects and proposes additions/updates here.
- **Topic files follow a consistent structure:** title, overview, then numbered or bulleted practices with rationale.
- **Keep practices actionable.** Each entry should be something a developer or agent can follow, not just an observation.
- **Include rationale.** Every practice should explain *why* — the incident, gotcha, or pattern that motivated it.
- **Prune when fixed.** If a practice was specific to a version bug that's been fixed, remove it.
## For Container Agents
When this repo is cloned to `/best-practices` inside agent containers:
- Read `BESTPRACTICES.md` for the index
- Read only the topic files relevant to your current task
- Do not modify files in this repo from within a container agent
## Source Control
- **Gitea org:** `skynet`
- **Remote:** `git@gitea.oreillyit.nz-ai-enablement:skynet/best-practices.git`

50
README.md
View File

@@ -1,3 +1,51 @@
# best-practices # best-practices
Cross-project best practices extracted from real project work Cross-project best practices extracted from real project work via the `/distill-best-practices` skill.
## How It Works
The knowledge distillation pipeline in `claude-foundations` processes session logs and memory files from all tracked projects, extracting generalisable practices into topic files here.
### Pipeline
1. **`/log`** — Captures session decisions and gotchas into per-project `memory/log/`
2. **`/reflect-logs`** — Processes logs into structured topic memory files
3. **`/distill-best-practices`** — Reads memory files across projects, proposes updates to this repo
### For Humans
Browse `BESTPRACTICES.md` for the full index. Each topic file is self-contained.
### For Agents
Container agents get this repo cloned to `/best-practices`. Read `BESTPRACTICES.md` for the index, then read only the topic files relevant to your task.
## Topics
| File | Description |
|------|-------------|
| `ansible.md` | Inventory, templates, idempotency, credential safety |
| `database-selection.md` | SQLite vs PostgreSQL decision criteria |
| `debugging.md` | Systematic diagnosis, full-chain testing, common pitfalls |
| `docker.md` | gosu PID 1, GIT_SSH_COMMAND scope, slim image patterns |
| `docker-uid-matching.md` | UID wrapper entrypoint, gosu pattern |
| `documentation.md` | CLAUDE.md, MEMORY.md, FUTURE.md, README.md structure |
| `git-source-control.md` | Commit practices, GitOps workflows, remote conventions |
| `helm.md` | Schema validation, version verification, values structure |
| `kubernetes.md` | Volume mounts, deployment strategies, naming, bootstrap ordering |
| `linting.md` | Tool choices per language, PostToolUse hook, pre-commit |
| `milestones.md` | Milestone workflow, verification, reflection process |
| `networking.md` | nftables, systemd sockets, Docker forwarding, TLS |
| `octopus-process-templates.md` | OCL syntax, step templates, Platform Hub patterns |
| `scripting.md` | Shell conventions, verification scripts, idempotency |
| `secrets-management.md` | SOPS + age, credential handling, encryption gotchas |
| `security-architecture.md` | Server boundary rule, proxy patterns, defense in depth |
| `skills-development.md` | Skill authoring, context injection, tool restrictions |
| `spec-driven-development.md` | Spec structure, requirement numbering, test-first workflow |
| `test-driven-development.md` | Edge case discovery, property-based testing, AI agent patterns |
| `validation.md` | Validate locally, deploy once; full-chain testing |
## Source Control
- **Gitea:** `skynet/best-practices`
- **Remote:** `git@gitea.oreillyit.nz-ai-enablement:skynet/best-practices.git`

36
ansible.md Normal file
View File

@@ -0,0 +1,36 @@
# Ansible
## Inventory and Execution
- Always pass `-i inventory.yml` explicitly or run from the directory containing `ansible.cfg`
- Playbooks that can't find inventory skip silently with no error — a common source of "it ran but nothing happened" confusion
- Variables that need customisation go in `inventory.yml` files, not scattered across role defaults
## Role Structure
- Roles follow standard structure: `tasks/main.yml`, `templates/*.j2`, `handlers/main.yml`
- Jinja2 templates have `.j2` extension and include a "managed by Ansible" header comment
## Template Safety
- **Never use placeholder values with `-e` for vars that template config files.** Using `-e "var=dummy"` will overwrite live configs with garbage. Either read real values, use `--skip-tags` to skip templating tasks, or restructure roles so sensitive templates are in a separate tag.
## Credential Safety
- Pass secrets via `@file` not `-e` on the command line — `-e "key=value"` exposes secrets in `ps` output
- Use temp files with `trap rm` cleanup: `-e "@${tmpfile}"`
## Module Gotchas
- `docker_compose_v2` doesn't support `state: restarted` — use `recreate: always` instead
- `ansible.builtin.unarchive` with `remote_src` and `extra_opts: --strip-components` is unreliable — use `get_url` + `command: tar` separately
- `get_url` won't re-download when the URL changes but the destination filename stays the same — use a version marker file to detect changes
## Service Restarts
- Some services (dnsmasq, etc.) need container restarts for config changes to take effect
- Ansible handlers handle this, but always verify the change took effect (e.g., `dig @<ip> <record> +short`)
## Docker Compose
- `network_mode: host` ignores `ports:` mappings — remove `ports:` to avoid warnings

100
database-selection.md Normal file
View File

@@ -0,0 +1,100 @@
# Database Selection
## The Rule: SQLite Is Not a Production Database
**Any service that meets ANY of the following criteria MUST use PostgreSQL (or equivalent server-grade database) from day one:**
- Attached to a FQDN (has a real domain name, even internal)
- Serves traffic from more than one process (API consumers, CI runners, webhooks, polling)
- Backs infrastructure that other systems depend on (Git hosting, container registries, auth providers)
- Will be accessed concurrently by automated systems (ArgoCD, CI runners, cron jobs)
**Do not use SQLite for these workloads. Not temporarily. Not "to start with." Not "we'll migrate later."**
SQLite uses file-level locking — only one writer at a time, and writes block reads. Under concurrent access, requests queue up waiting for the write lock, causing cascading timeouts. The failure mode is insidious: the service appears to work fine under light load but becomes intermittently unresponsive under real workloads. By the time you notice, everything that depends on it is also failing.
## The Cost of "We'll Migrate Later"
The Gitea SQLite→PostgreSQL migration (2026-03-28) cost nearly a full day of productivity:
- **Hours of accumulated unresponsiveness** across multiple projects before root cause was identified
- **Planning and implementation** of the migration itself
- **Migration complexity** that didn't need to exist: Gitea 1.23 has no `restore` command, `doctor convert` only handles charset conversion, `docker cp` corrupted PostgreSQL directory permissions, SSH authorized_keys weren't regenerated
- **Downstream impact** on ArgoCD (20 apps polling a locked database), CI runners (continuous 500 errors), container registry pulls (timeouts)
The PostgreSQL container takes 5 minutes to add to a Docker Compose stack at initial setup time. The migration took a day. Always pay the 5 minutes upfront.
## When SQLite Is Acceptable
SQLite is fine for:
- Local development databases (single developer, single process)
- Embedded application data stores (mobile apps, desktop apps, CLI tools)
- Read-heavy workloads with rare writes and a single writer process
- Test fixtures and throwaway data
- Configuration stores read at startup (not at request time)
## Implementation Pattern
For Docker Compose services that need a database:
```yaml
services:
postgres:
image: postgres:17-alpine
restart: unless-stopped
environment:
POSTGRES_DB: myapp
POSTGRES_USER: myapp
POSTGRES_PASSWORD: {{ db_password }}
volumes:
- /opt/postgres-myapp:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U myapp -d myapp"]
interval: 10s
timeout: 5s
retries: 5
start_period: 30s
networks:
- app_internal
deploy:
resources:
limits:
memory: 1G
myapp:
depends_on:
postgres:
condition: service_healthy
networks:
- app_internal
- external_network
networks:
app_internal:
driver: bridge
internal: true
```
Key points:
- PostgreSQL on an **internal bridge network** (no external access needed)
- Application **depends on PostgreSQL health** before starting
- **Resource limits** to prevent runaway memory usage
- **Separate data directory** per application (`/opt/postgres-myapp`, not shared)
- PostgreSQL container UID is **999** (not 1000) — set directory ownership accordingly
## For Kubernetes Deployments
Use the application's Helm chart PostgreSQL subchart, or deploy a standalone PostgreSQL instance:
- Bitnami PostgreSQL Helm chart for simple deployments
- CloudNativePG operator for production-grade PostgreSQL with HA, backups, and failover
- Never use SQLite with `emptyDir` or even PVC-backed volumes in multi-replica deployments
## Checklist for New Service Deployment
Before deploying any new service, check:
1. What database does the default configuration use?
2. If SQLite: does the service support PostgreSQL? (Almost all do — Gitea, Authelia, Headscale, Zulip, etc.)
3. Switch to PostgreSQL **before the first deployment**, not after problems appear
4. Add the database password to SOPS-encrypted secrets
5. Verify the database connection works before adding consumers

91
debugging.md Normal file
View File

@@ -0,0 +1,91 @@
# Debugging Methodology
## Check Before You Act
- Before writing firewall/network rules, check actual routing (`ip route get <dest>`)
- Before running config management with variables, ensure values are real, not placeholders
- Before assuming a container has a shell, `docker inspect` it
- Before creating API tokens, research all required scopes upfront — iterating one scope at a time costs a push-debug cycle each
## Routing and Networking
- Always run `ip route get <dest>` on the forwarding host first
- macvlan, Docker bridge, and other virtual interfaces mean the "obvious" physical interface is often wrong
- Test from both in-cluster and external perspectives
## Full-Chain Testing
After wiring up any new service:
1. Test direct to backend (bypass all proxies)
2. Test through reverse proxy (bypass DNS)
3. Test end-to-end as a user would
Use `curl --resolve` to test specific paths without depending on DNS propagation.
## When Something Doesn't Sync/Apply
- Check resource exclusions in the GitOps controller immediately
- Check if the resource type requires special permissions or labels
- Check if ServerSideApply conflicts are preventing field changes
- Don't try workarounds before understanding the root cause
## OIDC Integration Checklist
Before starting any OIDC integration, research:
1. What format is the `sub` claim (UUID? username?)
2. Which claims are in the ID token vs userinfo endpoint
3. How the consumer matches RBAC identities (groups? email? username?)
## Log-First Diagnosis
- **CrashLoopBackOff: check logs first.** Error messages in pod logs usually point directly to the fix. Don't tweak configuration or security contexts blindly — `kubectl logs <pod>` first.
- **Discriminate transient from persistent errors.** CSI lock contention, etcd timeouts during first install, and brief connectivity blips are self-healing. Don't spend time debugging errors that resolve on retry. If you see retry/backoff patterns in logs, wait before intervening.
- **Trust controller retry logic.** CSI controllers, operators, and reconciliation loops have built-in retry. Transient failures during rapid provisioning are expected, not bugs.
## Reproduce Before Fixing
When a bug is discovered or reported, **do not start by trying to fix it.** The first step is always to write a test that reproduces the failure:
1. **Write a failing test.** Capture the bug as a test case that demonstrates the broken behaviour. This forces you to understand the bug precisely — what input triggers it, what the wrong output is, and what the correct output should be.
2. **Fix the bug in isolation.** Use a subagent or a separate session to write the fix. The fixing agent gets the failing test as its success criterion — it's done when the test passes. This separation prevents the fixer from unconsciously weakening the test to match a broken implementation.
3. **The test stays forever.** The reproduction test becomes a permanent regression test. It proves the fix works and prevents the bug from returning.
This workflow has several advantages:
- **Forces precise understanding.** Writing a test means you know exactly what's broken, not just "it doesn't work."
- **Prevents partial fixes.** The test defines "done" objectively — the fix either passes or it doesn't.
- **Parallelises work.** While one agent fixes the bug, you can continue other work.
- **Catches regressions.** The test remains in the suite, guarding against the same class of failure.
```python
# Step 1: Write the failing test FIRST
def test_regression_issue_427_empty_payload_crashes():
"""Bug #427: Empty payload causes unhandled TypeError in dispatcher.
Should return a 400 validation error, not crash."""
response = client.post("/dispatch", json={})
assert response.status_code == 400 # Currently crashes with 500
# Step 2: Hand to a subagent/session: "Make this test pass without breaking others"
```
## `GIT_SSH_COMMAND` Only Affects Git-Invoked SSH
`GIT_SSH_COMMAND` (e.g., `ssh -o StrictHostKeyChecking=no`) only applies when `git` invokes SSH internally (clone, push, fetch). Direct `ssh` calls — such as `ssh -T git@host` for connectivity testing — ignore it entirely. When working in containers or CI environments where host keys aren't pre-trusted, direct SSH commands need explicit flags: `ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null`.
## Pattern Mining Before Authoring
Before building a new service, component, or script, read existing patterns in the codebase first. This matches conventions on the first attempt and avoids rework on naming, structure, and integration points. Applies to K8s manifests, CI pipelines, skill authoring, and script structure.
## Grep Your Own Docs
Known issues documented in CLAUDE.md or MEMORY.md but not applied to new scripts/configs waste debugging time. Search your own documentation before writing automation that touches areas with known gotchas.
## API Token Scope Errors
When an API endpoint returns a permission/scope error, read the error response body before guessing. Many APIs (Gitea, GitHub, GitLab) explicitly state the required scope in the error message (e.g., `required=[write:admin]`). This is faster and more reliable than consulting documentation or iterating one scope at a time.
## Minimal Container Images Have No Debug Tools
Distroless and single-binary containers (Garage, distroless Go images, etc.) have no shell, curl, wget, or other debug tools. `kubectl exec` commands will fail.
**For HTTP checks:** Use `kubectl port-forward svc/<name> <local-port>:<svc-port>` and run `curl` locally.
**For verification scripts:** Don't assume exec-based checks will work. Design health checks around port-forward + local tools, or use Kubernetes-native probes.

113
docker-uid-matching.md Normal file
View File

@@ -0,0 +1,113 @@
# Docker UID Matching for Mounted Volumes
When a host directory is mounted into a Docker container, files created by the container process are owned by the container user's UID/GID. If this doesn't match the host user, you get one of two problems:
1. **Permission denied** — the container can't write to the mounted directory
2. **Wrong ownership** — files created inside the container are owned by a different user on the host (e.g., `root` or UID `1001`)
Both are common sources of friction in developer-facing container tools and CI/CD pipelines.
## The Pattern: UID Wrapper Entrypoint
The solution is a small entrypoint wrapper script that:
1. Starts as root
2. Detects the UID/GID of the mounted directory via `stat`
3. Adjusts the container user's UID/GID to match using `usermod`/`groupmod`
4. Drops privileges via `gosu` and execs the real command
This is transparent to the user — they don't need to pass `--user` flags or know their UID.
### Implementation
**Dockerfile:**
```dockerfile
FROM ubuntu:24.04
RUN apt-get update && apt-get install -y --no-install-recommends gosu \
&& rm -rf /var/lib/apt/lists/*
# Create non-root user
RUN useradd -m -s /bin/bash agent
RUN mkdir -p /project && chown agent:agent /project
COPY uid-wrapper.sh /usr/local/bin/uid-wrapper.sh
# Start as root — the wrapper drops privileges after UID adjustment
ENTRYPOINT ["uid-wrapper.sh", "your-actual-command"]
```
**uid-wrapper.sh:**
```bash
#!/bin/bash
set -e
TARGET_DIR="/project" # The mount point to match
AGENT_USER="agent" # The non-root user to adjust
# If not root, skip adjustment (e.g., K8s securityContext sets UID)
if [ "$(id -u)" != "0" ]; then
exec "$@"
fi
if [ -d "$TARGET_DIR" ]; then
HOST_UID=$(stat -c '%u' "$TARGET_DIR")
HOST_GID=$(stat -c '%g' "$TARGET_DIR")
if [ "$HOST_UID" != "0" ]; then
# Adjust GID if different
if [ "$HOST_GID" != "$(id -g $AGENT_USER)" ]; then
groupmod -g "$HOST_GID" "$AGENT_USER" 2>/dev/null || true
fi
# Adjust UID if different
if [ "$HOST_UID" != "$(id -u $AGENT_USER)" ]; then
usermod -u "$HOST_UID" "$AGENT_USER" 2>/dev/null || true
fi
# Fix home directory ownership
chown -R "$AGENT_USER:$(id -g $AGENT_USER)" /home/"$AGENT_USER" 2>/dev/null || true
fi
fi
exec gosu "$AGENT_USER" "$@"
```
### Key Details
- **`gosu` over `su`/`sudo`.** `gosu` execs directly (PID 1 becomes the real process), while `su` creates a child process that breaks signal handling. `gosu` is the standard tool for this pattern.
- **Handle existing UIDs/GIDs.** The host UID may already be taken by another user in the container. Ubuntu 24.04 images ship with a `ubuntu` user at UID 1000 — the most common host UID. The wrapper must evict the conflicting user to a high unused UID (e.g., 59999 and search downward) before assigning the target UID to the agent user. `usermod -u <new> <conflicting_user>` followed by `usermod -u <target> agent`. Same applies to GIDs — use `getent group` to check before `groupmod`. **Simpler alternative:** Delete the conflicting user at build time (`RUN userdel ubuntu` in the Dockerfile). This avoids runtime conflict handling entirely and is preferred when you control the image.
- **SSH agent socket forwarding requires UID match.** When mounting `$SSH_AUTH_SOCK` into a container, the socket is mode 0600 owned by the host UID. The container process must run as the same UID to use it — which the UID wrapper handles naturally. Set `SSH_AUTH_SOCK` in the container env to the mounted path. Note: private keys cannot be extracted via the agent protocol — it only supports signing operations.
- **Skip when not root.** In Kubernetes, `securityContext.runAsUser` sets the UID before the container starts. The wrapper detects this (`id -u != 0`) and skips adjustment — the platform is handling it.
- **Mount point owned by root.** If the mounted directory is owned by root (UID 0), don't adjust — running as root defeats the purpose. The wrapper only adjusts for non-root UIDs.
- **Fix home directory.** After `usermod`, the user's home directory still has the old UID. `chown -R` fixes this. Skip other directories — only fix what the user needs.
## When to Use
- **Developer-facing container tools** where users mount local project directories (e.g., running a CLI tool inside a container)
- **CI/CD containers** that write build artifacts back to a mounted workspace
- **Any container that writes to a host-mounted volume** and the output needs correct ownership
## When NOT to Use
- **Kubernetes with securityContext** — the platform handles UID assignment; the wrapper correctly skips adjustment in this case
- **Containers that don't write to mounted volumes** — unnecessary overhead
- **Images that must run as root** — the wrapper's purpose is to run as non-root; if you need root, you don't need the wrapper
## Alternatives
| Approach | Pros | Cons |
|----------|------|------|
| `docker run --user $(id -u):$(id -g)` | Simple, no image changes | No home directory, no `/etc/passwd` entry, breaks tools that need a user identity |
| `fixuid` | Purpose-built tool | Another binary to install and maintain |
| UID wrapper (this pattern) | Transparent, handles edge cases, works in Docker and K8s | Requires `gosu` in image, starts as root |
| Named volumes only | Docker manages ownership | Can't mount host directories |
## Source
Learned from the agent-runtimes project (M1) where Claude Code refuses `--dangerously-skip-permissions` as root for security reasons. The container needed to run as non-root, but files created in mounted project directories needed to be owned by the host user. The UID wrapper solved both problems transparently.

47
docker.md Normal file
View File

@@ -0,0 +1,47 @@
# Docker Best Practices
## Use gosu for Entrypoint Privilege Dropping
`su -c "command"` and `sudo -u agent command` create child processes. The real command is not PID 1, so Docker signals (SIGTERM on stop) don't reach it. Use `gosu agent command` which execs directly — the command becomes PID 1 with proper signal handling.
## GIT_SSH_COMMAND Only Affects Git-Invoked SSH
`GIT_SSH_COMMAND` only applies when git invokes SSH (clone, push, fetch). Direct `ssh` calls need explicit flags: `ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null`. Don't assume setting `GIT_SSH_COMMAND` fixes all SSH operations in a container.
## Use Python urllib for Health Checks in Slim Images
Service images based on `python:3.12-slim` don't include curl. For in-container health checks, use `python3 -c "import urllib.request; urllib.request.urlopen('http://...')"`. This applies to verification scripts using `kubectl exec` and to Kubernetes liveness/readiness probes that exec into containers.
## Buildx Docker-Container Driver Can't See Local Images
When using buildx with the docker-container driver, `FROM local-image:latest` tries Docker Hub because the builder runs in a separate container that can't see locally-loaded images. Always use the full registry path in Dockerfiles. In CI, split into sequential jobs so base images are pushed to the registry before dependent images build.
## Delete Conflicting Default Users at Build Time
Ubuntu 24.04 base images ship with a `ubuntu` user at UID 1000 — the most common host UID. This causes `usermod -u 1000` conflicts and can trigger non-deterministic hangs (e.g., `newgrp ubuntu` waiting for a password on stdin). Delete the default user in the Dockerfile: `RUN userdel -r ubuntu`.
## Service Images Should Use Minimal Base Images
Service images (API servers, background workers) should use `python:3.12-slim` or equivalent, not the agent base image. Agent base images include CLIs, Node.js, and other tooling that bloats service images unnecessarily. Keep agent tooling in agent images only.
## Platform-Specific Native Binaries
Never mount host `node_modules` into a Docker container when the build uses platform-specific native binaries (e.g., Tailwind CSS, esbuild, SWC). Always run `npm install` inside the same container that runs the build. The native binary is compiled for the platform where `npm install` runs — host and container may differ in libc, architecture, or OS.
**Symptom:** `Cannot find native binding` or `Cannot find module '@tailwindcss/oxide-linux-x64-gnu'`
**Fix:** Run `npm install` inside the container, not on the host.
## Docker Wrapper Scripts and TTY Flags
Docker wrapper scripts (e.g., `~/sbin/hugo` calling `docker run -it ...`) fail with `the input device is not a TTY` in non-interactive contexts (CI pipelines, Claude Code, cron jobs, scripts).
**Fix:** Only pass `-t` when stdin is a terminal: `[ -t 0 ] && TTY_FLAG="-t" || TTY_FLAG=""`. Or omit `-t` entirely and let callers add it when needed.
## Three-Tier UID Resolution
The UID wrapper should resolve the target UID/GID using this priority:
1. **Environment variables** (`AGENT_UID`/`AGENT_GID`) — injected by the orchestrator/dispatcher. Preferred because it's explicit and deployment-specific.
2. **stat the mount point** — detect the UID/GID of the mounted directory. Works when no env vars are set.
3. **Skip** — if not root or no mount point exists, run as the default container user.
This makes UID matching a deployment concern (varies per host), not a configuration concern (baked into images). See [Docker UID Matching](docker-uid-matching.md) for the full UID wrapper pattern.

65
documentation.md Normal file
View File

@@ -0,0 +1,65 @@
# Documentation Standards
Every project maintains four core markdown files.
## CLAUDE.md
The primary reference for Claude sessions. Should contain:
- Project overview and architecture
- Repository structure (keep updated as the project evolves)
- Key design decisions with rationale
- Conventions and coding standards
- Environment details (IPs, URLs, credential references — never actual values)
- Common operations / how-to recipes
- Put critical rules at the top — Claude reads sequentially and earlier content has more influence
## MEMORY.md (Tiered Memory System)
Long-running projects accumulate significant context. Use a **tiered memory** structure:
**MEMORY.md** is a **thin index only** — one-line descriptions with links to topic files in `memory/`. No content lives in MEMORY.md itself. Keep it under ~50 lines.
**memory/** contains the actual content, split by topic:
- `memory/project-status.md` — Current milestone, what's next, blockers
- `memory/gotchas-<topic>.md` — Gotchas grouped by technology
- `memory/process-lessons.md` — How-to-work-with-this-repo lessons
- `memory/m<N>-reflection.md` — One file per milestone reflection
- `memory/decisions.md` — Architecture and design decisions
**Principles:**
- Split by topic, not by time
- Index descriptions matter — they're used to decide what to read
- Prune aggressively — stale memory is worse than no memory
- Each file should be self-contained and greppable
- Deduplicate with CLAUDE.md — stable conventions go in CLAUDE.md, learnings and gotchas go in memory
## FUTURE.md
Backlog of improvement ideas, each with:
- **Problem:** What's painful or manual today
- **Idea:** What the improvement looks like
- **Open questions:** Unknowns to research before starting
- **Depends on:** Other items or milestones that should come first
## README.md
Human-readable project documentation:
- Architecture summary
- Quick start / setup instructions
- Milestone table with status
- Scripts section listing every script with purpose and usage
## Extract Reusable Patterns Early
When a reusable pattern emerges during project work (a gotcha that applies to any project using the same tool, a process lesson that generalises), extract it into a best-practices guide immediately rather than waiting for a dedicated distillation pass. The guide pays for itself when used to audit and improve the current project in the same session, and benefits all future projects.
## CONTEXT.md (Active Work Focus)
Tells Claude (and independent agents) what the project is currently working on. Follows the same thin-index pattern as MEMORY.md.
**CONTEXT.md** is a thin index with links to `context/<topic>.md` detail files.
**Principles:**
- Keep it current — remove entries when work is complete
- Orient agents — this is the primary mechanism for pointing independent agents at the right work
- Complement, don't duplicate — CLAUDE.md has conventions, MEMORY.md has learnings, CONTEXT.md has the current focus

61
git-source-control.md Normal file
View File

@@ -0,0 +1,61 @@
# Git & Source Control
## Commit Practices
- Use meaningful commit messages; prefer small, focused commits over large batches
- Never commit secrets in plaintext — use SOPS + age or equivalent encryption
- Enable pre-commit hooks where appropriate (secret detection, linting, formatting)
## Pre-Commit Hooks
- Block `local_secrets/` and similar directories from being committed
- Auto-encrypt files matching `.sops.yaml` rules that aren't yet encrypted
- Enable with `git config core.hooksPath .githooks`
- Consider secret detection, linting, and formatting hooks
## GitOps Workflow
- All infrastructure changes should be tracked in Git
- No manual changes without corresponding GitOps manifests — anything applied manually (e.g., `kubectl apply`, `helm install`) should immediately get a corresponding tracked manifest
- For ArgoCD-managed clusters: edit in Git, push, sync — never edit live resources directly
## Remote Conventions
- SSH workflows preferred over HTTPS for Git remotes
- Use SSH config host aliases for multi-user setups (e.g., `gitea.example.com-<user>`)
- Remote URL format: `git@<host-alias>:<org>/<repo>.git`
- Optionally push-mirror to GitHub for public visibility
## Access and Clone Gotchas
- **Org repos require explicit collaborator grants.** Don't assume organizational membership implies write access — verify permissions before setting up automation or CI/CD.
- **Shallow clones break push operations.** `git clone --depth 1` is fine for read-only CI jobs, but pipelines that push artifacts, tags, or mirror to other remotes need full clones.
## Version Management
- Use the latest stable version of dependencies unless pinned for a reason
- Verify versions from live sources (`helm search repo`, upstream docs, package registries) — don't rely on memory
- Document the reason in a comment if a version is intentionally pinned below latest
- Check compatibility matrices before upgrading (e.g., Talos ↔ Kubernetes, framework ↔ runtime)
## Placeholder Conventions in Template-Heavy Repos
When files contain multiple templating syntaxes (Go templates `{{ .var }}`, CI variables `${{ }}`, Helm `{{ }}`, etc.), use a distinct placeholder convention for your own substitutions that can't be confused with any templating language:
- **Double-underscore:** `__CUSTOMER_NAME__`, `__DOMAIN__`
- **All-caps curly brace (no spaces):** `{{CUSTOMER_NAME}}` (distinct from Go's `{{ .Title }}` with spaces and dots)
Choose one convention per repo and document it.
## Git Worktrees for Parallel Agent Work
When running parallel agents or tasks that modify the same repo:
- Give each task its own branch and worktree (`git worktree add .worktrees/<task-id> -b <branch>`)
- Tasks with no dependencies branch from HEAD; tasks with one dependency branch from that dependency's branch
- Tasks with multiple dependencies get an octopus merge base branch
- Worktrees share the `.git` object store — fast creation, minimal disk usage
- Keep containers detached (`docker run -d`, not `--rm`) so logs survive for inspection after exit
## API-Created Repos Need SSH User as Collaborator
If a repo is created via API token (user A) but pushes use an SSH alias authenticating as user B, user B has no access by default. Add the SSH-authenticating user as admin collaborator via API before the first push. This applies to Gitea, GitHub, and any platform where API auth and SSH auth use different identities.

25
helm.md Normal file
View File

@@ -0,0 +1,25 @@
# Helm Charts
## Schema Validation
- **Always validate values against the chart schema before committing.** Run `helm show values <repo>/<chart> --version <ver>` to check the actual structure.
- Helm chart schemas change between versions — field names and nesting can differ from documentation or online examples.
- A quick `helm template` test locally catches schema errors before deployment.
- `additionalProperties: false` in chart schemas means any extra keys at the wrong nesting level cause a hard failure.
## Version Verification
- Run `helm search repo` or check upstream docs to confirm latest stable versions
- Don't rely on memory for chart versions — they go stale quickly
- Check compatibility matrices between chart version, app version, and other cluster components
## Multi-Source Applications
- ArgoCD multi-source Applications use `$ref` syntax to combine external Helm charts with Git-stored values files
- Keep values files in Git alongside the ArgoCD Application manifest
## Timeout Handling
- Under cluster pressure (many events, etcd busy), default Helm timeouts may not be enough
- Increase timeout for initial installs (e.g., 10m instead of 5m)
- `helm upgrade --install` is idempotent — retries are safe

69
kubernetes.md Normal file
View File

@@ -0,0 +1,69 @@
# Kubernetes Patterns
## Volume Mounts
- **Avoid `subPath` volume mounts** for Secrets and ConfigMaps. The kubelet does not auto-update `subPath` mounts when the source changes — the pod must be restarted. Use directory mounts instead and adjust the application's config path.
- **Secret volume propagation is async.** After updating a Secret, the kubelet takes seconds to sync mounted volumes. A `rollout restart` issued immediately after may start pods with stale data. Add a short delay (5s) before restarting.
## Deployment Strategies
- **RWO PVC + RollingUpdate = Deadlock.** New pod can't attach the volume while the old pod holds it. Use `strategy: Recreate` for single-replica deployments with RWO PVCs.
- **SSA + strategy change conflict.** Switching from RollingUpdate to Recreate via ServerSideApply fails because SSA won't remove the old `rollingUpdate` field. Must patch the live resource first.
## Naming
- `metadata.name` must be DNS-1035 compliant — no dots allowed. Replace dots with dashes (e.g., `oreillyit-nz` not `oreillyit.nz`). Label values CAN contain dots.
## Bootstrap Ordering
Some components have chicken-and-egg dependencies:
1. CNI (e.g., Cilium) must be installed before anything else — nodes are NotReady without it
2. GitOps controller (e.g., ArgoCD) installed second
3. Root app applied last — the GitOps controller then "adopts" CLI-installed releases
Manual bootstrap secrets (encryption keys, OIDC client secrets) must be documented as explicit steps.
## Network Policies
- DNS egress for `toFQDNs` rules must use `toEndpoints` targeting kube-dns pods with `rules.dns` — this triggers the DNS proxy. Using `toCIDRSet` for DNS bypasses the proxy and FQDN rules never populate.
- Cross-namespace policies need explicit namespace matching (e.g., `matchExpressions` on namespace label).
- Always test from the actual consumer namespace, not same-namespace test pods.
## Probe Strategy
- **Liveness vs readiness probes serve different purposes.** TCP checks confirm the process is listening (liveness). Exec/command checks confirm the application is ready to serve (readiness). Don't conflate them.
- **Probes must match application host validation.** Applications that validate Host headers (e.g., Next.js `ALLOWED_HOSTS`) will reject probes sent to the pod IP. Set `httpGet.httpHeaders` with the expected Host value.
- **Don't load credentials into liveness probes.** If readiness requires an authenticated check (e.g., `sqlcmd`), use a simple TCP check for liveness and reserve the authenticated check for readiness only.
## Init Container Patterns
- **Writable config via init container + emptyDir.** When apps require writable directories but ConfigMaps are read-only, use an init container to copy config into an emptyDir volume that the main container mounts read-write.
- **Privilege separation.** Init containers can run as root to create directories or set ownership, while the main container runs as a non-root UID. Prefer this over running the entire workload as root.
- **Non-root images have hidden filesystem requirements.** Many modern images (e.g., MSSQL 2022, UID 10001) need writable directories beyond the obvious ones. Always check image documentation or `docker inspect` before writing manifests.
## StatefulSet Edge Cases
- **CrashLoopBackOff pods won't auto-replace on spec update.** The StatefulSet controller won't delete and recreate a crashing pod when you update the spec — manual `kubectl delete pod` is required to force recreation.
- **Immutable field diffs can deadlock auto-sync.** StatefulSet fields like `volumeClaimTemplates` are immutable after creation. GitOps controllers (ArgoCD) will show permanent OutOfSync if the desired state differs from the live immutable fields. Force sync or recreate the StatefulSet.
- **SSA causes perpetual OutOfSync from defaulted fields.** Kubernetes defaults fields on StatefulSets (`persistentVolumeClaimRetentionPolicy`, `revisionHistoryLimit`, `updateStrategy.rollingUpdate.partition`) that aren't in the Helm template. With `ServerSideApply=true`, GitOps controllers see these as diffs and report OutOfSync even though the app is Healthy. The app functions correctly — this is cosmetic. Consider ArgoCD `ignoreDifferences` for these fields.
## GitOps: Imperative vs Declarative
- **Never use imperative operations on GitOps-managed resources.** `kubectl rollout restart` adds annotations that conflict with the GitOps controller's desired state, causing permanent OutOfSync. Use declarative paths instead — update a configmap hash annotation in Git, or change a pod template label.
- **ArgoCD reconciliation has latency.** New Application manifests don't appear immediately due to polling intervals. Use manual refresh annotations when automation needs immediate reconciliation.
## PodSecurity Alignment
- **Namespace PodSecurity labels must match container security contexts.** DinD, CSI drivers, and other privileged workloads need `pod-security.kubernetes.io/enforce: privileged` on their namespace. A `baseline` or `restricted` namespace silently blocks privileged pods.
- **Document privileged namespace requirements.** When a workload needs elevated privileges, document the specific requirement (e.g., "Docker-in-Docker for CI builds") alongside the namespace label.
## ArgoCD Source Type Detection
- **ArgoCD auto-detects Kustomize.** When a source directory contains `kustomization.yaml`, ArgoCD runs Kustomize automatically. Adding an explicit `directory:` source type overrides this detection and causes ArgoCD to try applying `kustomization.yaml` as a raw K8s resource, which fails with schema errors. Remove explicit directory source types from Kustomize sources.
- **Credential template URL-prefix must match exactly.** ArgoCD repo-creds secrets use URL prefix matching. When migrating Git server URLs (hostname, protocol, or port changes), update the credential template to match the new prefix. Stale credentials cause "authentication required" errors on all apps using that prefix.
## Miscellaneous
- `enableServiceLinks: false` may be needed when K8s-injected service env vars conflict with app config (e.g., Authelia interprets `AUTHELIA_*` service vars as configuration).
- Proxmox VM names must match K8s node hostnames for cloud controller manager integration.
- Metrics-server on Talos needs `--kubelet-insecure-tls` (self-signed kubelet certs).

94
linting.md Normal file
View File

@@ -0,0 +1,94 @@
# Linting & Formatting
Automated code formatting and linting integrated into the Claude Code workflow via PostToolUse hooks and git pre-commit hooks.
## Architecture
```
claude-foundations/formatters/ ← canonical formatter scripts (one per extension)
project/formatters/ ← symlinks to the formatters this project uses
~/.claude/hooks/post-edit-lint.sh ← dispatches to project formatters on Edit/Write
.git/hooks/pre-commit ← symlink to pre-commit-lint.sh
```
Projects opt in by symlinking only the formatters they need. No formatter = no-op.
## Tool Choices by Language
| Language | Formatter | Linter | Config File |
|----------|-----------|--------|-------------|
| Python | `ruff format` | `ruff check` | `pyproject.toml` |
| Shell | `shfmt` | `shellcheck` | `.editorconfig` |
| TypeScript/JavaScript | `biome format` | `biome check` | `biome.json` |
| SQL | `sqlfluff fix` | `sqlfluff lint` | `.sqlfluff` |
| JSON/YAML/Markdown | `prettier` | — | `.prettierrc` |
### Why these tools?
- **ruff**: Rust-based, extremely fast, replaces black + isort + flake8 + pyflakes in one tool
- **biome**: Rust-based, replaces prettier + eslint for JS/TS in one tool
- **shfmt + shellcheck**: The standard combo for shell scripts; shfmt formats, shellcheck catches bugs
- **prettier**: Handles JSON/YAML/MD well; biome doesn't cover these yet
## Formatter Script Contract
Every script in `formatters/` follows the same interface:
- **Input:** `$1` = absolute file path
- **Behaviour:** format the file in place, then lint it
- **Stdout:** suppressed
- **Stderr:** lint warnings/errors that couldn't be auto-fixed
- **Exit code:** `0` if clean, `1` if lint errors remain
- **Missing tools:** exit `0` silently (`command -v` check)
- **No `set -e`:** individual commands may fail; execution must continue
## PostToolUse Integration
The `post-edit-lint.sh` hook fires on Edit/Write/MultiEdit:
1. Extracts `file_path` and `cwd` from stdin JSON
2. Walks up from `cwd` to find `formatters/` directory
3. Creates a checkpoint using `git hash-object` (fast, no commits)
4. Runs the matching formatter
5. On clean pass: removes checkpoint, exits 0 (silent)
6. On lint errors: keeps checkpoint, exits 2 (feeds errors to Claude)
Exit code 2 is special for PostToolUse — it feeds stderr back to Claude as feedback without blocking the edit.
## Checkpoint Mechanism
Uses `git hash-object -w` to store pre-format content as an orphan blob (~1ms, no commits, no stash). The `.pre-lint` file stores only a 40-char SHA. Falls back to `cp` outside git repos.
```bash
# Revert after lint errors:
git cat-file blob "$(cat file.py.pre-lint)" > file.py
rm file.py.pre-lint
```
## Subagent Fix Pattern
When lint errors occur, Claude sees the errors via stderr. The recommended workflow:
1. Claude reports the errors to the user
2. If the user says "fix", Claude spawns a subagent via the Agent tool
3. The subagent reads the file and errors, makes Edit calls to fix them
4. Each Edit triggers the hook again (re-format, re-lint) in the subagent
5. Fix iterations stay in the subagent's context, not the main conversation
## Pre-commit Integration
`pre-commit-lint.sh` reuses the same formatter scripts:
1. Iterates staged files (Added/Modified only)
2. Runs matching formatters
3. Re-stages formatted files
4. Exits non-zero if lint errors remain (blocks commit)
Install: symlink `.git/hooks/pre-commit` → `pre-commit-lint.sh`, or use `setup-formatters.sh` which does this automatically.
## Setup Checklist
1. Run `scripts/setup-formatters.sh <project> <ext> [<ext> ...]` to symlink formatters
2. Install the required tools (`ruff`, `shfmt`, `shellcheck`, `biome`, `prettier`, `sqlfluff`)
3. Create per-project config files as needed (`pyproject.toml`, `.editorconfig`, etc.)
4. Run `scripts/install-hooks.sh` to install the PostToolUse hook (one-time global setup)
5. Use `/linter scan` to verify everything is connected

52
milestones.md Normal file
View File

@@ -0,0 +1,52 @@
# Milestones & Reflections
## Milestone Structure
Break projects into numbered milestones (M1, M2, ...). This provides clear checkpoints, measurable progress, and natural reflection points.
## Milestone Completion Checklist
Every milestone MUST include:
### 1. Verification Script
`scripts/verify-m<N>.sh` — automated checks confirming all milestone outcomes.
- Idempotent, non-destructive, returns non-zero on failure
- Colour output (green/red) for pass/fail
- Environment-resilient (no sudo, test from accessible side)
- Check for default/insecure credentials
### 2. Milestone Reflection
Write `memory/m<N>-reflection.md` by reviewing the **entire conversation** from milestone start. Cover:
- **Process improvements:** What slowed us down? Wrong assumptions? Where did we go in circles? What would make this faster if redone from scratch?
- **Key knowledge for reproduction:** Critical facts, gotchas, non-obvious config details, version-specific quirks, debugging detours
- **Scripts and automation:** Existing tools that proved valuable, new scripts to build, patterns to extract into reusable automation
- **Future improvements:** Ideas that surfaced but don't belong in current scope — add to FUTURE.md
### 3. Updated README.md
Ensure the scripts section, milestone table, and setup steps are current.
### 4. Updated CLAUDE.md
Reflect new repo structure, conventions, and patterns discovered during the milestone.
## Reflection Quality
Good reflections capture:
- Commit stats (total commits, fix percentage) to measure validation discipline
- Longest detour and root cause
- Most avoidable waste and what would have prevented it
- Concrete checklist items for future similar work
## Evaluate Content Placement Before Building
Before creating a new document, system, or catalog, discuss where it belongs conceptually. Different content types have different lifecycles:
- **Accumulated learnings** → memory files (gotchas, process lessons)
- **Authoritative maintained maps** → CLAUDE.md or dedicated reference docs
- **Behavioral contracts** → spec files
- **Active work state** → CONTEXT.md
Picking the wrong home creates maintenance friction later. A five-minute placement discussion saves a future migration.

33
networking.md Normal file
View File

@@ -0,0 +1,33 @@
# Networking & Infrastructure
## nftables Flush Ruleset on Remote Hosts
On remote hosts, `nftables flush ruleset` followed by a failed rule load leaves the host with NO firewall. SSH survives only on existing connections — new connections are blocked or allowed depending on the default policy.
**Always validate rules before applying:** `nft -c -f <rulefile>` does a dry-run parse. For extra safety, deploy a cron-based auto-rollback timer that reverts rules unless explicitly confirmed (similar to `shutdown -c` pattern).
## systemd Socket Activation Overrides Config File Ports
On modern Linux systems (Ubuntu 24.04+), systemd socket activation controls the listening port for services like SSH. Editing the service config file alone (e.g., `sshd_config Port 2222`) has no effect — the socket unit still binds the original port.
**Check socket activation first:** `systemctl cat <service>.socket` shows whether socket activation is in play. If so, override the socket unit's `ListenStream` directive, not the service config.
## Docker Sets iptables FORWARD Policy to DROP
Docker sets the iptables FORWARD chain default policy to DROP. This affects ALL forwarding on the host, not just Docker traffic. Non-Docker forwarding (VPN, VM bridges, custom NAT) silently breaks.
**Fix:** Add explicit ACCEPT rules in the `DOCKER-USER` chain for non-Docker forwarding needs. This chain is processed before Docker's own rules and persists across Docker restarts.
## HTTP Host Header vs TLS SNI Are Different Layers
When proxying to a backend over HTTPS, two independent identifiers must be set correctly:
- **TLS SNI** — sent during the TLS handshake, used for certificate selection. Missing SNI causes `x509: cannot validate certificate for <IP>`.
- **HTTP Host header** — sent after TLS is established, used for virtual host routing. Missing or wrong Host header causes 404 from the backend.
A reverse proxy must set both. They often need to be the same value, but they're configured independently.
## Wildcard Certs in Auto-Renewing Proxies
Auto-renewing proxies (Caddy, Traefik with Let's Encrypt, etc.) that also support file-loaded certificates treat file-loaded certs as globally available. A wildcard cert loaded for one site block will match ALL matching subdomains, silently preventing automatic certificate issuance for other sites.
**Rule:** Use automatic certificate management for all sites. Don't mix file-loaded and automatic certs unless you understand the matching priority.

221
octopus-process-templates.md Normal file
View File

@@ -0,0 +1,221 @@
# Octopus Deploy Process Templates
Best practices for creating and managing Octopus Deploy process templates using OCL (Octopus Configuration Language) in Platform Hub.
## Key Concepts
- **Step templates** (action templates) are reusable individual steps, created via API or UI, stored in a space's library
- **Process templates** are reusable multi-step deployment processes, stored as OCL files in Platform Hub's Git repo
- **Project templates** compose process templates into full project configurations (feature coming soon)
- Process templates live in `.octopus/process-templates/<slug>.ocl` in the Platform Hub Git repo
- Projects consume process templates via `process_template` blocks in their `deployment_process.ocl`
## OCL File Structure
### Process Template File
```hcl
name = "Deploy to Kubernetes - Helm"
description = "Standard Helm-based Kubernetes deployment with pre-validation and smoke tests"
# Parameters — values supplied by consuming projects
parameter "target_tags" {
display_settings = {
Octopus.ControlType = "TargetTags"
}
help_text = "Kubernetes target tags"
label = "Target Tags"
}
parameter "cloud_target" {
display_settings = {
Octopus.ControlType = "SingleLineText"
}
help_text = "Cloud provider (gcp, aws, azure)"
label = "Cloud Target"
value "gcp" {} # default value
}
# Steps — ordered deployment steps
step "deploy-helm" {
name = "Deploy via Helm"
properties = {
Octopus.Action.TargetRoles = "#{target_tags}"
}
action {
action_type = "Octopus.Script"
properties = {
Octopus.Action.Script.ScriptSource = "Inline"
Octopus.Action.Script.Syntax = "PowerShell"
Octopus.Action.Script.ScriptBody = "Write-Host 'Deploying...'"
}
worker_pool_variable = ""
}
}
```
### How Projects Consume Process Templates
In a project's `deployment_process.ocl`:
```hcl
process_template "deploy-app" {
name = "Deploy Application"
process_template_slug = "deploy-to-kubernetes-helm"
version_mask = "1.X" # auto-update minor/patch
parameter "target_tags" {
value = "kubernetes,production"
}
parameter "cloud_target" {
value = "aws"
}
}
```
## OCL Syntax Rules
From the EBNF grammar (https://github.com/OctopusDeploy/Ocl):
- **Name, `=`, and value** must be on the same line
- **Block name, labels, and `{`** must be on the same line
- **Closing `}`** must be on its own line (except empty blocks like `value "default" {}`)
- **Strings** use double quotes, cannot contain unescaped `"`
- **Multi-line strings** use heredoc: `<<-EOF` / `EOF` (indented variant)
- **Arrays** use `["item1", "item2"]`
- **Dictionaries** use `{ key = value }` (one entry per line inside braces)
## Referencing Step Templates
Step templates are referenced by **ID and version** in the action's properties, NOT by name:
```hcl
step "run-tests" {
name = "Run Unit Tests"
action {
# Reference a step template instead of action_type
properties = {
Octopus.Action.Template.Id = "ActionTemplates-62"
Octopus.Action.Template.Version = "1"
# Step template parameter values
Language = "go"
CloudTarget = "gcp"
}
worker_pool_variable = ""
}
}
```
When NOT using a step template, define `action_type` directly:
```hcl
action {
action_type = "Octopus.Script"
properties = { ... }
}
```
## Channel Scoping
Scope steps to specific channels using the `channels` attribute on the action block. Uses **channel slugs** (auto-generated from names):
```hcl
action {
action_type = "Octopus.Script"
channels = ["non-prod"] # only runs in non-prod channel
properties = { ... }
}
```
## Package References
```hcl
# Container image for worker execution
container {
feed = "registered-dockerhub" # feed slug, not ID
image = "octopusdeploy/worker-tools:ubuntu.22.04"
}
# Package reference in a step
packages "MyPackage" {
acquisition_location = "NotAcquired" # Server | ExecutionTarget | NotAcquired
feed = "platformhub-non-prod" # feed slug
package_id = "paul-oreilly-octopus/nonprod/listing-service"
properties = {
SelectionMode = "immediate"
}
}
```
## Parameter Types
| Control Type | `Octopus.ControlType` Value | Can Have Default |
|---|---|---|
| Single-line text | `SingleLineText` | Yes |
| Multi-line text | `MultiLineText` | Yes |
| Sensitive/password | `Sensitive` | Yes (encrypted) |
| Checkbox | `Checkbox` | Yes |
| Dropdown | `Select` | Yes |
| AWS/Azure/GCP Account | `AWSAccount` etc. | Yes |
| Worker Pool | (worker pool) | No |
| Package | (package) | No |
| Target Tags | `TargetTags` | No |
| Environments | (environments) | No |
| Channels | (channels) | No |
## Step Properties Reference
| Property | Type | Values | Default |
|---|---|---|---|
| `step.condition` | enum | `Success`, `Failure`, `Always`, `Variable` | `Success` |
| `step.start_trigger` | enum | `StartAfterPrevious`, `StartWithPrevious` | `StartAfterPrevious` |
| `step.package_requirement` | enum | `LetOctopusDecide`, `BeforePackageAcquisition`, `AfterPackageAcquisition` | `LetOctopusDecide` |
| `action.channels` | string[] | channel slugs | all |
| `action.environments` | string[] | environment slugs | all |
| `action.excluded_environments` | string[] | environment slugs | none |
| `action.is_disabled` | bool | | `False` |
| `action.is_required` | bool | | `False` |
| `action.notes` | string | step description | |
| `action.worker_pool` | string | worker pool slug | |
| `action.worker_pool_variable` | string | variable name | |
## Versioning
- Process templates use **semantic versioning** (major.minor.patch)
- `version_mask = "1.X"` in consuming projects auto-updates on minor/patch changes
- **Major version bumps** require explicit upgrade by consuming projects
- Keep major bumps for breaking changes (parameter renames, step removals)
- Minor/patch for new optional parameters, script improvements, bug fixes
## Gotchas
1. **Step templates are space-scoped.** Process templates in Platform Hub cannot reference `ActionTemplates-*` IDs from other spaces. If you need reusable steps, use inline `action_type = "Octopus.Script"` in the process template OCL. Step templates are useful within a single space's projects, but not for cross-space process templates.
2. **Process template names** cannot contain parentheses, slashes, or ampersands — only letters, numbers, periods, commas, dashes, underscores, and hashes.
3. **Heredoc for multi-line scripts** — use `<<-EOT` / `EOT` for PowerShell scripts that contain double quotes. The `-` prefix allows indented closing tags.
4. **Every step needs a worker pool.** Process templates must have a `worker_pool` parameter (type `WorkerPool`), and every action must set `worker_pool_variable = "worker_pool"` referencing it. Without this, the template will fail to parse with "A step must specify a worker pool parameter".
5. **Publishing and sharing is UI-only.** Process template sharing (which spaces can see/use a template) is stored in the Octopus database, not in Git/OCL. You must publish and share each template through the UI. There is no API or CLI for this currently.
## Best Practices
1. **One template per deployment pattern**, not per cloud. Use parameters to vary cloud-specific behaviour.
2. **Use step templates for reusable individual steps**, process templates for reusable multi-step workflows.
3. **Parameters should have sensible defaults** where possible — reduces friction for consuming projects.
4. **Use `TargetTags` parameter type** for Kubernetes target selection rather than hardcoding roles.
5. **Name templates with the action, not the technology**: "Deploy to Kubernetes" not "Helm Chart Deployer".
6. **Keep descriptions updated** — they appear in the UI when browsing templates.
7. **Process templates cannot reference the project's own Git repo** for scripts — use inline scripts or external URLs.
8. **Test templates** by creating a test project that consumes them before sharing widely.
## References
- OCL Syntax: https://octopus.com/docs/projects/version-control/ocl-file-format
- Config as Code Reference: https://octopus.com/docs/projects/version-control/config-as-code-reference
- Process Templates: https://octopus.com/docs/platform-hub/templates/process-templates
- Template Parameters: https://octopus.com/docs/platform-hub/templates/parameters
- Publishing & Sharing: https://octopus.com/docs/platform-hub/templates/publishing-and-sharing
- Best Practices: https://octopus.com/docs/platform-hub/templates/process-templates/best-practices
- Troubleshooting: https://octopus.com/docs/platform-hub/templates/process-templates/troubleshooting
- OCL Grammar (EBNF): https://github.com/OctopusDeploy/Ocl

56
scripting.md Normal file
View File

@@ -0,0 +1,56 @@
# Scripting Conventions
## Structure
- All scripts live in `scripts/` and run from the repository root
- Scripts should be idempotent and safe to re-run
- Exit non-zero on failure so `&&` chains work naturally
## Verification Scripts
- Automated checks confirming milestone or feature outcomes
- Use colour output (green/red) for pass/fail indicators
- Should be non-destructive and environment-resilient
- Avoid needing sudo — test from the accessible side of a connection instead
- Check for default/insecure credentials and print remediation instructions on failure
- Use `curl --resolve` to bypass DNS/proxy layers when testing direct connectivity
- **Never use `set -e` in verification scripts.** A verify script's job is to run ALL checks and report a summary. `set -e` exits on the first failure, hiding remaining issues. Use explicit conditional checks and a pass/fail counter instead. Note: `((var++))` under `set -e` is a classic bash trap — pre-increment of 0 evaluates to falsy, triggering errexit. Use `var=$((var + 1))`.
## Automation Triggers
If you run the same 3+ commands in sequence more than once, it should become a script. Look for:
- Repeated command sequences in conversation history
- Steps requiring careful ordering
- Multi-step manual processes that are error-prone
## Error Handling by Tool Purpose
Not all scripts need the same error handling strategy:
- **Destructive scripts** (deploy, configure, delete) should use `set -euo pipefail` — fail fast on any error.
- **Reporting/read-only scripts** (status dashboards, aggregation, monitoring) should start without `set -e` — complex data collection from multiple sources is hard to debug under errexit. Use explicit conditional checks instead.
- **The choice depends on the tool's purpose.** A script that writes to production needs strict error handling. A script that reads from 10 sources and aggregates results needs resilience.
## Dryrun Mode
Every script that modifies state should support `--dryrun` / `-n`:
- Makes the script self-documenting about its side effects
- Enables safe testing and review before execution
- Enables test harnesses that verify output without executing changes
- Dryrun output should show exactly what would happen, not a summary
## Shell Gotchas
- `((PASS++))` fails under `set -e` when PASS=0 — the expression evaluates to 0 (false), triggering errexit. Use `PASS=$((PASS + 1))` instead.
- `set -e` silently terminates complex pipelines and subshells with no output — makes debugging extremely difficult. Also kills command substitutions that capture non-zero exit codes (e.g., `result=$(grep "pattern" file)` exits if grep finds nothing).
- `grep` interprets option-like strings (starting with `-`) as flags — use `--` terminator before patterns or input that may start with dashes.
- Always quote variables in conditionals and file paths
- Use `trap` for cleanup of temp files and credentials
- **Use `git diff --numstat` for binary file detection** instead of `file`. The `file` command is unreliable (marks shell scripts as "executable"), while `git diff --numstat` shows `-` for binary files using git's robust binary detection heuristics.
- **Use `cat -A` to diagnose invisible character issues.** Reveals non-printing characters like em dashes, zero-width spaces, and smart quotes that look identical to correct characters but break YAML parsers, config files, and frontmatter. Essential when a file looks correct but tooling rejects it.
- `grep` returns exit code 1 when no lines match — under `set -e`, this kills the script even when zero matches is a valid outcome. Append `|| true` to `grep` commands in pipelines where empty results are expected.
- `while read` in a pipeline creates a subshell — variables modified inside the loop (counters, accumulators) are lost after the loop ends. Use process substitution (`while read line; do ...; done < <(command)`) or here-strings to keep the loop in the current shell.
- **Order matters in sed/regex transformation pipelines.** Process more specific patterns before general ones. For example, if both `![[image.png]]` and `[[page]]` are valid patterns, process the image embed first — otherwise the general wikilink regex matches the inner `[[image.png]]` and the `!` prefix is left orphaned.
## JSON Construction in Scripts
Use Python (not shell) for constructing JSON payloads. Multi-line prompts with quotes, backticks, and special characters break shell-based JSON construction (printf/sed/heredocs). Python's `json.dump` handles escaping correctly every time. For scripts that need to construct and submit JSON payloads, write the construction logic in Python even if the rest of the script is bash.

108
secrets-management.md Normal file
View File

@@ -0,0 +1,108 @@
# Secrets Management
## SOPS + age
SOPS with age encryption is the standard across all projects. A single `.sops.yaml` at the repo root defines path-based encryption rules.
### File Naming
- `.sops.yaml` path-based rules match specific filename patterns (e.g., `**/*secret*.yaml`)
- Non-secret files must NOT contain `secret` in their name, or the pre-commit hook will encrypt them
- KSOPS generator files should be named `ksops-generator.yaml`, not `secret-generator.yaml`
### encrypted_regex Gotcha
When using `encrypted_regex` for selective field encryption (e.g., Ansible group_vars), variable names must contain a keyword that matches the regex (e.g., `password|private_key|api_key|secret|token`). Arbitrary key names are silently left unencrypted.
### SOPS Vars Plugin
Each Ansible project needs `vars_plugins_enabled = host_group_vars,community.sops.sops` in `ansible.cfg`. Files in `group_vars/` must be named after a group (e.g., `all.sops.yaml`), not arbitrary names.
### Interactive Editor Pitfalls
- `sops <file>` opens an interactive editor — fails in non-interactive sessions
- `sops -e /tmp/file` fails when the temp path doesn't match `.sops.yaml` rules
- Multiple `sops --set` calls can corrupt files — use the interactive editor for multi-field edits
## Credential Handling
- **Never pass secrets via command-line arguments** — visible in `ps` output
- Use `@file` references, environment variables sourced at runtime, or stdin
- For Ansible, use temp files with `trap rm` cleanup: `-e "@${tmpfile}"`
- Read secrets at execution time and use them ephemerally — never cache or persist values
- Reference the **existence** of a secret file in docs, never its contents
## Bootstrap Secrets
Some secrets are chicken-and-egg (e.g., the age decryption key for ArgoCD's KSOPS). These must be created manually as a bootstrap step and documented clearly.
## Credential Lifecycle Management
- **Track credential expiry dates.** OAuth client secrets, API tokens, and certificates have expiry dates that can cause silent failures. Document expiry dates when creating credentials.
- **Set alerts before expiry.** For long-lived credentials (e.g., 720-day OAuth client secrets), set calendar reminders or automated monitoring alerts well before they expire.
- **Rotation plan.** Know the rotation procedure before you need it — some credential types (e.g., Azure app registrations) require coordinated updates across multiple systems.
## Multi-Field Secret Files
Secret files that contain multiple fields (e.g., repo URL, token, username) cannot be used as bare values. Consumers must parse individual fields (e.g., `grep + awk` or structured YAML/JSON parsing).
The multi-field format is preferable because it's self-documenting — all related credentials live together. But any automation reading the file needs extraction logic, not just `cat`.
## Generating Secrets with gen-secret
Use the `gen-secret` script (from `small-scripts`, symlinked to `~/sbin/gen-secret`) to generate cryptographically random strings that are safe for bash, YAML, and JSON without escaping.
### Workflow: Generate + SOPS Encrypt
1. **Generate the secret value** — use `gen-secret` with an appropriate length:
```bash
SESSION_SECRET=$(gen-secret 48) # 48-char session key
API_KEY=$(gen-secret 20) # 20-char access key
API_SECRET=$(gen-secret 40) # 40-char secret key
```
2. **Write plaintext YAML to the target path** — the file must be at the path matched by `.sops.yaml` rules (e.g., `**/*secret*.yaml`):
```bash
cat > path/to/my-secret.sops.yaml <<EOF
apiVersion: v1
kind: Secret
metadata:
name: my-credentials
namespace: my-namespace
type: Opaque
stringData:
SESSION_SECRET: ${SESSION_SECRET}
API_KEY: ${API_KEY}
EOF
```
3. **Encrypt in-place** — SOPS reads `.sops.yaml` to determine the encryption key and regex:
```bash
sops -e -i path/to/my-secret.sops.yaml
```
4. **Verify** — decrypt and confirm no placeholders remain:
```bash
sops -d path/to/my-secret.sops.yaml
```
### Key Points
- **Always encrypt at the target path.** `sops -e /tmp/file.yaml` fails because `/tmp/` doesn't match `.sops.yaml` path rules. Write the plaintext to the final location, then `sops -e -i` in-place.
- **Use shell variables, not files, for ephemeral secrets.** Generate into a variable (`SECRET=$(gen-secret 48)`), interpolate into the YAML, then encrypt. The plaintext never touches disk as a standalone file.
- **Appropriate lengths:** 32 chars is the default and sufficient for most use cases. Use 48+ for session secrets, 20 for access key IDs, 40 for secret keys (matching common API patterns).
- **For credentials from external systems** (e.g., Gitea API tokens, registry passwords), read them from `~/dev/claude/secrets/` at point of use — don't generate random replacements for values that must match an external system.
- **Clean up temp files** if you write plaintext to a temporary location. Use `trap` cleanup or `rm -f` after encryption.
### Replacing Placeholder Secrets
When SOPS-encrypted files contain placeholder values (e.g., `PLACEHOLDER_SESSION_SECRET`):
1. Decrypt: `sops -d secret.sops.yaml` — inspect current values
2. Write the corrected plaintext YAML to the same path (overwriting the encrypted file)
3. Re-encrypt: `sops -e -i secret.sops.yaml`
4. Verify: `sops -d secret.sops.yaml | grep -c PLACEHOLDER` — should return 0
## Backup Considerations
Backup plans must include encryption keys (age private keys, etc.) so that encrypted data in Git repos remains recoverable.

79
security-architecture.md Normal file
View File

@@ -0,0 +1,79 @@
# Security Architecture
## The Server Boundary Rule
**No server-side credential may cross the server boundary to the client. Ever.**
This is a hard line, not a guideline. The only credentials that cross the boundary between client and server are the client's own identity credentials (MFA tokens, login passwords, OIDC tokens, etc.) — and these flow from client to server, never the reverse.
### What this means in practice
- **API tokens stay server-side.** If a browser-based application needs to call a third-party API (Gitea, S3, database, etc.), it calls a backend proxy that holds the token. The token never appears in JavaScript, localStorage, cookies, or any client-accessible storage.
- **Service account credentials stay server-side.** Database passwords, S3 access keys, webhook secrets, SMTP credentials — these are mounted into server-side containers via Kubernetes Secrets or environment variables and never exposed to clients.
- **OAuth tokens for third-party services stay server-side.** If a user authenticates with Service A and the application needs to call Service B on their behalf, the application's backend holds the Service B credentials. The client only ever sees its own session token with the application.
- **Per-user tokens mapped server-side.** When individual users need distinct third-party access (e.g., per-user Gitea tokens for audit trails), the mapping from user identity to their token lives on the server. The client authenticates with its own identity (e.g., via Authelia MFA), and the server looks up the appropriate third-party token.
### The identity exception
The only credentials that legitimately cross from client to server:
- **Username + password** — the user's own login credentials
- **MFA tokens** — TOTP codes, WebAuthn assertions, security key responses
- **OIDC/OAuth tokens** — tokens that represent the user's identity with the application itself (not with third-party services)
- **Session cookies/JWTs** — issued by the server to represent an authenticated session
These all share the property: they are the user's own identity, flowing from client to server for authentication purposes.
### Architecture pattern: proxy with identity mapping
When a client-side application (SPA, CMS, admin UI) needs to interact with a backend service that requires credentials:
```
Client ──→ Auth Layer (MFA) ──→ API Proxy ──→ Backend Service
│
├── Reads user identity from auth headers
├── Looks up user's backend credential
├── Forwards request with backend credential
└── Returns response (without credential)
```
The proxy:
1. Sits behind the authentication layer (Authelia, OAuth2 Proxy, etc.)
2. Reads the authenticated user's identity from trusted headers (e.g., `Remote-User`)
3. Maps the identity to the appropriate backend credential
4. Makes the backend API call with the credential
5. Returns the response — never the credential
### Security layers (defense in depth)
A well-designed proxy architecture has multiple independent security layers:
1. **Authentication** — User must prove their identity (MFA, OIDC)
2. **Session validation** — Proxy validates the session is current and legitimate
3. **Authorization** — Proxy checks the user has access to the requested resource
4. **Backend permissions** — The backend service enforces its own access controls
5. **Branch/scope protection** — Fine-grained controls prevent privilege escalation (e.g., branch protection rules)
Each layer is independent — compromising one doesn't bypass the others.
### Anti-patterns
- **Passing API tokens to the browser via OAuth.** Even with PKCE, the token ends up in client-accessible storage. Use a backend proxy instead.
- **Shared service account tokens.** One token for all users means no audit trail and no granular revocation. Map per-user tokens server-side.
- **Embedding credentials in client-side config.** API keys in `config.js`, `.env` files served statically, or hardcoded in HTML — all violate the boundary rule.
- **Forwarding backend tokens via API responses.** Even "temporarily" returning a token in a response body breaks the rule. The client should never see it.
- **Using the same token for client auth and backend calls.** The user's session token with your application is distinct from any token your application uses to call backend services.
### When credentials must be client-side
Some scenarios genuinely require client-side credentials (e.g., direct S3 uploads for large files, WebRTC signaling). In these cases:
- Use **presigned URLs** or **temporary credentials** with the narrowest possible scope and shortest possible lifetime
- The presigning/credential-issuance happens server-side
- The temporary credential is scoped to exactly one operation (e.g., upload one file to one path)
- Log the issuance server-side for audit
### Real-world example: CMS editing
**Wrong:** CMS authenticates directly with Gitea via OAuth popup. Gitea token lands in browser localStorage. CMS makes API calls directly to Gitea with the token.
**Right:** CMS sits behind Authelia (MFA). A proxy service intercepts the OAuth flow, issues a proxy session token (containing only the user's identity), and forwards all API calls to Gitea using a per-user server-side Gitea token. The Gitea token never leaves the server.

58
skills-development.md Normal file
View File

@@ -0,0 +1,58 @@
# Claude Code Skills
## Skill Structure
- Each skill lives in `skills/<skill-name>/SKILL.md`
- Skills should be project-agnostic where possible — use dynamic context injection to adapt
- After adding a new skill, run the install script to register it
- Skills only useful for one project should live in that project's `.claude/skills/` instead
## Authoring Guidelines
- **Inline by default** — only use `context: fork` if the skill genuinely doesn't need conversation history
- **Pre-fetch context** with `!`command`` injection to reduce tool calls during execution
- **Restrict tools** with `allowed-tools` to the minimum needed — reduces permission prompts
- **Use $ARGUMENTS** for user input, `$0`, `$1` etc. for positional args
- Dynamic commands in `!`command`` run at skill load time, not during Claude's execution
## Portable Path Resolution
- **Use `CLAUDE_PROJECT_ROOT` env var** for cross-project path references in `!`command`` blocks. Hardcoded absolute paths (e.g., `~/dev/claude/...`) are user-specific. Relative paths (`../`) break depending on CWD and can trigger sandbox violations when they resolve outside allowed directories.
- **Add a detection fallback.** Include a "Step 0" in skill instructions that detects the project root by walking up the directory tree to find the highest `CLAUDE.md` if the env var isn't set. This makes skills work even without prior setup.
- **Keep config paths relative to the root.** Settings files should use paths relative to `CLAUDE_PROJECT_ROOT` (e.g., `projects_dir: projects`) rather than absolute paths, so they're portable across machines.
## Skill Discovery Timing
- **Skills are discovered at session start, not dynamically.** Creating or symlink a new skill mid-session requires restarting Claude Code to use it as a slash command.
- **Broken symlinks cause silent failures under `set -e`.** `readlink -f` on a broken symlink returns empty string. The install script should validate symlinks and remove stale ones.
## `!`command`` Gotchas
- **No `$()` command substitution** — the permission checker rejects commands containing `$()`
- **No complex shell pipelines relying on subshells** — keep commands simple and self-contained
- **`allowed-tools` patterns must match the command binary** — each binary used in `!`command`` blocks needs its own pattern
- **Prefer specific tool patterns over broad ones** — `Bash(git log *)` is safer than `Bash(git *)`
- **Fallback to tool instructions for dynamic paths** — if a command needs `$ARGUMENTS` to compute a path, use a plain-text instruction telling Claude to use the Read tool instead
- **Env var expansion works** — `${CLAUDE_PROJECT_ROOT}` expands in `!`command`` blocks because they run as shell commands. This is the recommended pattern for portable cross-project paths.
## Non-ASCII in YAML Frontmatter
Skills with em dashes (`—`), smart quotes (`"`), or other non-ASCII characters in the YAML frontmatter `description` field fail to load silently — the skill appears as "Unknown skill" with no error message. The markdown body below the frontmatter can contain any characters.
AI models commonly generate em dashes instead of regular dashes. Always validate skill files (e.g., with `cat -A` or a dedicated validator) before committing.
## Profile-Independent Skills Directories
Each Claude Code profile maintains a completely independent skills directory. Skills installed in one profile (e.g., default) are unavailable in other profiles (e.g., `.claude-octopus`). Install scripts must use the profile-aware config directory path rather than hardcoded paths like `~/.claude/skills/`.
## Research Failure History Before Building Validators
When building a tool that detects known problems (like a linter rule or a validator), research all historical failures first — session logs, git commit history, issue trackers. Documentation alone misses non-obvious failure patterns. The upfront research investment produces comprehensive coverage that incremental discovery cannot match.
## Task Decomposition for Independent Agents
When breaking work into tasks for independent agents (container-based or otherwise):
- **Task prompts must be fully self-contained** — agents have no conversation history from the decomposer
- **Include explicit "read these files first" instructions** in each task prompt
- **Balance granularity** — over-decomposing creates merge overhead; under-decomposing wastes parallelism potential
- **Scope each task to one deliverable** with clear reads (inputs) and writes (outputs) to minimise conflicts

250
spec-driven-development.md Normal file
View File

@@ -0,0 +1,250 @@
# Spec-Driven Development with AI Agents
Best practices for using structured specifications to coordinate AI agent implementation work. Extracted from real project experience (agent-runtimes) and industry research (OpenSpec, Codified Context paper, Addy Osmani's workflow guides).
## Why Specs Matter for AI Agents
AI agents trust documentation absolutely. A well-written spec gives an agent everything it needs to implement a subsystem without reading the entire codebase. A stale or vague spec causes silent failures where agents generate code that is structurally valid but architecturally wrong.
Specs serve three functions that CLAUDE.md alone cannot:
1. **Compressed context** — an agent reads one spec, not 300 lines of mixed concerns
2. **Testable contracts** — numbered requirements and scenarios translate directly to pytest
3. **Handoff boundaries** — an agent working on the dispatcher doesn't need to understand the entrypoint internals, just the interface between them
## Spec Structure
Each spec follows a consistent template. Sections are ordered so an agent can read top-down and build understanding progressively.
### Required Sections
1. **Overview** — What this subsystem does. 2-3 sentences. An agent should know if this spec is relevant after reading this.
2. **Responsibilities** — What this subsystem owns and what it delegates. Prevents scope creep during implementation.
3. **Dependencies** — Which other specs to read first. Keeps the reading list minimal.
4. **Data Model** — Types, schemas, state machines, interfaces. The concrete contract.
5. **Requirements** — Numbered functional requirements (e.g., E-1, E-2). Each must be independently testable.
6. **Scenarios** — Concrete given/when/then examples. These become test functions.
### Optional Sections
7. **Interface** — API surface, function signatures, HTTP endpoints. Include when the subsystem has an external-facing API.
8. **Extension Points** — How to add new capabilities without modifying existing code. Step-by-step instructions.
9. **Error Handling** — Failure modes and expected behaviour. Prevents agents from inventing their own error strategies.
### Writing Guidelines
- **Be specific, not comprehensive.** A spec that says "handle errors appropriately" is useless. A spec that says "return exit code 124 on timeout" is testable.
- **Include the why.** Design intent and constraints prevent agents from making structurally valid but architecturally wrong changes. Requirements without rationale are followed mechanically — agents can't judge edge cases or make trade-offs. Every constraint needs a "Why:" line. Example: `"Secrets never in payload"` needs `"because payloads may be logged and stored in task history"`.
- **Use concrete examples.** Every data model section should include a realistic JSON/code example, not just a schema.
- **Cross-reference, don't duplicate.** If two specs share a concept (e.g., the payload schema), one spec owns it and the other links to it.
- **Keep each spec self-contained.** An agent should be able to implement a subsystem by reading the target spec plus its listed dependencies. If it needs to read CLAUDE.md, the spec is incomplete.
## Requirement Numbering
Each spec uses a short prefix derived from its name, followed by a sequential number:
| Spec | Prefix | Example |
|---|---|---|
| payload.md | P | P-1, P-2 |
| entrypoint.md | E | E-1, E-2 |
| actions.md | A | A-1, A-2 |
| runners.md | R | R-1, R-2 |
| dispatcher.md | D | D-1, D-2 |
| control-plane.md | CP | CP-1, CP-2 |
Requirements must be:
- **Independently testable** — each maps to one or more test functions
- **Unambiguous** — an agent can determine pass/fail without human judgement
- **Stable** — changing a requirement number invalidates tests, so avoid renumbering
## Scenarios as Test Blueprints
Every scenario in a spec should be directly translatable to a test function. Use this format:
```markdown
### Scenario: Pre-action failure
**Given:** Payload with clone pre-action (invalid repo URL)
**When:** Clone fails (git returns non-zero)
**Then:** on_error actions run, container exits 1. Runner never executes.
```
This becomes:
```python
def test_scenario_preaction_failure_runs_on_error_and_exits_1(...):
"""Given clone fails, on_error runs and exits 1. Runner never executes."""
```
Guidelines:
- Each scenario tests one behaviour, not a combination
- Include both happy paths and error paths
- Name the scenario descriptively — it becomes the test function's docstring
- Include enough setup detail that an agent can write the test without guessing
## The Spec → Test → Code Workflow
This is the core development loop. Tests are written from the spec before code exists.
### 1. Write or Update the Spec
Define requirements and scenarios. Get them reviewed. The spec is the source of truth for what the system should do.
### 2. Write Tests from the Spec
Translate requirements and scenarios into pytest functions. Tests should:
- Map to requirement IDs in their names: `test_e4_preaction_failure_skips_remaining`
- Use the scenario's given/when/then as the test body structure
- Mock external dependencies (subprocess, HTTP, filesystem)
- Run fast (no Docker, no network, no real APIs)
### 3. Run the Tests — They Should All Fail
This confirms the tests are actually testing something. If a test passes before implementation, it's either testing the wrong thing or the feature already exists.
### 4. Implement Until Tests Pass
Write the minimum code to make tests pass. The spec defines what, the tests verify it, the code implements it.
### 5. Update Spec if Implementation Reveals Issues
Sometimes implementation reveals that a requirement is unworkable or incomplete. Update the spec, update the test, then update the code. The spec stays authoritative.
## Spec Maintenance
### Preventing Drift
Specs drift from code when they're treated as planning documents that are "done" after implementation. They must be treated as living contracts.
**Rules:**
- **Spec changes require test changes.** If a requirement changes, its test must change in the same commit.
- **Code changes that affect interfaces require spec changes.** If a function signature, API endpoint, or data schema changes, the relevant spec must be updated in the same commit.
- **New features require spec-first.** Add the requirement and scenario to the spec, write the test, then implement.
### CI Enforcement
Enforce spec hygiene with automated checks:
1. **Pre-commit hook** — run pytest, block commit on failure (already implemented)
2. **Spec coverage check** — a script that verifies every numbered requirement has at least one test function referencing it
3. **Orphan test detection** — tests referencing requirement IDs that no longer exist in specs
Before completing any milestone, manually walk through every requirement ID (e.g., CP-1..CP-20, TH-1..TH-13) and verify a corresponding test exists. Automated spec coverage checks catch this in CI, but a manual audit before milestone completion catches gaps that the automation might miss (stubs, placeholder tests, tests that reference the ID but don't actually test the requirement).
### Review Checklist
When reviewing a PR that touches a spec subsystem:
- [ ] Spec updated if interface or behaviour changed
- [ ] Test added/updated for new/changed requirements
- [ ] Cross-references still valid
- [ ] No requirements removed without deprecation note
## Context Architecture for Agents
Based on the Codified Context paper (108k-line system, 283 sessions), structure project knowledge in three tiers:
### Tier 1: Hot Context (Always Loaded)
CLAUDE.md — conventions, env vars, repo structure, scripts. Loaded every session. Keep under ~300 lines by linking to details elsewhere.
### Tier 2: Spec Context (Per-Task)
`spec/` files — loaded based on what the agent is working on. An agent implementing a new action reads `spec/actions.md` + `spec/payload.md`. An agent working on the dispatcher reads `spec/dispatcher.md` + `spec/container-backends.md`.
The spec index (SPEC.md) has a "read this when..." column to guide selection.
### Tier 3: Cold Context (On-Demand)
`memory/` files — gotchas, reflections, decisions. Loaded only when relevant. An agent hitting a weird Cilium issue checks `memory/gotchas-cilium.md`.
### Routing Context to Agents
When launching an agent to work on a subsystem:
1. Point it at the relevant spec(s) via its prompt
2. Include CLAUDE.md for conventions
3. Let it pull from memory/ on-demand if it hits issues
Don't load everything — agents perform better with focused context than with a 50-page dump.
## Testing Depth
The spec→test→code workflow defines *when* to write tests. For *how* to write comprehensive tests — edge case discovery, property-based testing, mutation testing, AI agent testing patterns — see [Test-Driven Development](test-driven-development.md).
## Post-Write Spec Audit
After writing specs, audit them against best practices before implementation. Common gap categories:
1. **Missing rationale** — Constraints without "Why:" lines. Agents follow them mechanically but can't judge edge cases.
2. **Missing error/failure scenarios** — Happy paths are covered but failure modes aren't specified.
3. **Cross-spec interface misalignment** — Two specs describe the same interface differently.
4. **Vague requirements** — "Handle errors appropriately" instead of specific error codes and behaviours.
5. **Missing specs for discovered subsystems** — Implementation reveals components that weren't planned for.
Write-then-audit is more productive than trying to get specs perfect on the first pass. The audit step catches systematic gaps across all specs at once.
## Planning Session Limits
Architecture decisions, infrastructure research, and spec refinement each get one planning session. After three sessions of planning, start implementation. Specs are hypotheses that need code to validate them — extended planning without implementation produces diminishing returns and theoretical designs that don't survive contact with reality.
## Categorize Findings Before Acting
When a spec review or audit produces many findings, categorize them by priority (high/medium/low) before making changes. Present the categorized list for alignment before editing. Starting edits without prioritization leads to scope creep — low-priority cosmetic fixes consume time that should go to high-priority structural gaps.
## Multi-Agent Orchestration Practices
### Commit WIP Before Decomposing Tasks
Untracked and uncommitted files are NOT available in git worktrees. If agents work in worktrees (or container-mounted worktrees), they won't see specs, plans, or dependency outputs that haven't been committed. Commit to a staging branch before decomposition — this eliminates the dominant overhead of manually copying files into each worktree.
### Agents Must Self-Verify with Tests
Add "Run tests and fix any failures" to every implementation agent prompt. Agents that write code without running tests produce bugs that only surface during assembly. Self-verification catches issues while the agent still has full context of what it wrote.
### State Import and Style Conventions Explicitly
Agents default to standard language conventions (e.g., relative Python imports, standard packaging). If the project uses non-standard patterns (bare imports, specific naming conventions, module-level structure), state them explicitly in the prompt. A single line like "Use `from harness import X`, not `from .harness import X`" prevents import mismatches during assembly.
### Budget for Assembly Fixups
Parallel agent work produces ~3 fixups per orchestration run, each under 5 minutes. Common fixup categories: import conventions, module-level side effects, SDK exception constructor signatures, validator patterns. This is the expected cost of parallel work, not a failure. Budget 15-20 minutes for assembly and fixup after each orchestration run.
### Two-Phase Orchestration: Specs First, Then Implementation
When orchestrating multi-agent work for a milestone, decompose in two phases:
1. **Phase 1:** Spec-writing agents produce the contracts (using the plan as input).
2. **Review:** Human reviews specs for cross-spec consistency before proceeding.
3. **Phase 2:** Implementation agents receive actual spec files (not plan descriptions).
This works significantly better than defining all tasks upfront because spec agents validate the plan against reality, the review step catches cross-spec inconsistencies, and implementation agents work from concrete contracts rather than plan summaries.
### Include an Integration Verification Task After Orchestration
Agent orchestration leaves integration gaps at component boundaries. Each agent completes its assigned scope correctly, but nobody owns the integration points between them (e.g., stub comments, ORM mapping methods not updated for new fields). After every orchestration run, include an explicit integration verification step that checks cross-component contracts — call sites, shared data models, and handoff points.
### Decompose Along File Boundaries
When splitting work into parallel agent tasks, ensure each task writes to distinct files. When two agents must modify the same file, make the shared changes small and predictable — identify the conflict point upfront so the merge is trivial. File-boundary decomposition produces zero-conflict assemblies.
### Choose Manual Implementation for Tightly-Coupled Cross-Component Work
When changes are small per file (5-15 lines) but tightly coupled across many files (each change depends on the previous), skip agent orchestration and implement manually. The assembly overhead exceeds the implementation time. Agent orchestration excels when tasks are independent and substantial; manual implementation excels when work is sequential and interconnected.
## Anti-Patterns
### Specs as documentation, not contracts
**Symptom:** Specs describe what was built, updated after the fact. Tests don't reference spec IDs.
**Fix:** Write specs before code. Tests reference requirement IDs. Specs are the input, not the output.
### Mega-spec
**Symptom:** One large spec covering the entire system. Agents must read thousands of lines to find what they need.
**Fix:** Split by subsystem. Each spec should be readable in under 5 minutes.
### Spec without scenarios
**Symptom:** Requirements are abstract ("handle errors gracefully"). No concrete examples.
**Fix:** Every requirement needs at least one given/when/then scenario with specific inputs and outputs.
### Implementation details in specs
**Symptom:** Spec dictates variable names, algorithm choices, internal data structures.
**Fix:** Specs define what and why, not how. The interface is specified; the implementation is free.
### Untested requirements
**Symptom:** Requirements exist in the spec but no test references them. They drift without anyone noticing.
**Fix:** Spec coverage check in CI. Every requirement ID must appear in at least one test function name.

486
test-driven-development.md Normal file
View File

@@ -0,0 +1,486 @@
# Test-Driven Development for Spec-Based Projects
Best practices for writing comprehensive, regression-catching tests in projects that use structured specifications. Focuses on maximising test value (catching real bugs) rather than test volume (inflating coverage numbers). Extracted from industry research, academic papers (TDAD, Codified Context), and practitioner experience.
## Core Principle: Tests Are the Spec's Enforcement Layer
In a spec-driven project, the spec defines *what* and the tests *prove it*. A requirement without a test is an aspiration. A test without a requirement is undocumented behaviour. Keep them tightly coupled:
- Every numbered requirement (P-1, E-3) has at least one test
- Every test function name includes its requirement ID: `test_e3_preaction_failure_exits_1`
- Spec changes and test changes ship in the same commit
## Deriving Tests from Specs
### Requirements to Tests
Each spec requirement becomes one or more test functions. The mapping isn't always 1:1 — a requirement like "must respect timeout" needs tests for: default timeout, explicit timeout, timeout=0 (no limit), timeout exceeded.
```python
# From spec: R-4: Runners must respect runtime.timeout.
# Default 3600s. Value of 0 means no timeout.
def test_r4_default_timeout_is_3600():
"""R-4: When timeout not specified, default is 3600s."""
def test_r4_explicit_timeout_is_honoured():
"""R-4: When timeout=60, process killed after 60s."""
def test_r4_zero_timeout_means_no_limit():
"""R-4: When timeout=0, no timeout is applied."""
def test_r4_timeout_returns_exit_code_124():
"""R-4 + R-5: Timeout produces exit code 124."""
```
### Scenarios to Tests
GIVEN/WHEN/THEN scenarios translate directly to Arrange/Act/Assert:
```python
def test_scenario_preaction_failure_runs_on_error():
"""Given clone fails, on_error runs and exits 1. Runner never executes."""
# GIVEN — arrange
payload = make_payload(pre_actions=[{"action": "clone", "repo": "bad-url"}])
mock_clone = Mock(side_effect=subprocess.CalledProcessError(128, "git"))
# WHEN — act
exit_code = run_entrypoint(payload, clone_handler=mock_clone)
# THEN — assert
assert exit_code == 1
mock_runner.assert_not_called()
mock_on_error.assert_called_once()
```
### Parameterised Tests from Spec Enumerations
When a spec lists multiple valid values, use `@pytest.mark.parametrize`:
```python
# From spec: task states are pending, assigned, running, succeeded, failed, timed_out, cancelled
@pytest.mark.parametrize("terminal_state", ["succeeded", "failed", "timed_out", "cancelled"])
def test_cp_terminal_state_cannot_be_overwritten(terminal_state):
"""CP: Terminal states reject further transitions with 409."""
```
## Systematic Edge Case Discovery
~80% of bugs cluster at boundaries. Use these techniques to find edge cases systematically rather than by intuition.
### Boundary Value Analysis
For every input parameter, test at the edges of its valid range:
| Input type | Test values |
|---|---|
| Integer (range 1-100) | 0, 1, 2, 99, 100, 101, -1, MAX_INT |
| String | `""`, `"a"`, max-length string, max+1, unicode (`"\u0000"`, emoji), whitespace-only |
| List/Array | `[]`, `[single]`, many items, duplicates, `None` |
| Dict/Map | `{}`, missing required keys, extra unknown keys, `None` values |
| Timeout (seconds) | 0, 1, -1, very large (999999), `None`/missing |
| Base64 | valid, invalid chars, empty, padding variants (`=`, `==`, none) |
### Equivalence Partitioning
Group inputs into classes where all members should behave identically. Test one from each class:
```python
# Payload validation: prompt field
# Class 1: valid string → accepted
# Class 2: empty string → rejected (spec says prompt is required)
# Class 3: missing key → rejected
# Class 4: wrong type (int, list, None) → rejected
# Class 5: very long string → accepted (no length limit in spec)
@pytest.mark.parametrize("prompt,should_pass", [
("Fix the bug", True), # Class 1: valid
("", False), # Class 2: empty
(None, False), # Class 3: missing/None
(42, False), # Class 4: wrong type
("x" * 100_000, True), # Class 5: long string
])
def test_p_prompt_validation(prompt, should_pass):
...
```
### State Transition Coverage
For state machines (task states, dispatcher states), test:
1. **Every valid transition:** `pending → assigned → running → succeeded`
2. **Every invalid transition:** `succeeded → running` (should be rejected)
3. **Initial state:** newly created tasks start in `pending`
4. **Terminal states:** `succeeded`, `failed`, `timed_out`, `cancelled` cannot transition further
5. **Re-entrant transitions:** same state → same state (should be idempotent or rejected, per spec)
```python
VALID_TRANSITIONS = [
("pending", "assigned"),
("assigned", "running"),
("running", "succeeded"),
("running", "failed"),
("running", "timed_out"),
("assigned", "cancelled"),
("running", "cancelled"),
]
INVALID_TRANSITIONS = [
("succeeded", "failed"),
("failed", "running"),
("cancelled", "pending"),
("timed_out", "running"),
]
@pytest.mark.parametrize("from_state,to_state", VALID_TRANSITIONS)
def test_valid_state_transition(from_state, to_state):
...
@pytest.mark.parametrize("from_state,to_state", INVALID_TRANSITIONS)
def test_invalid_state_transition_rejected(from_state, to_state):
...
```
### The Edge Case Checklist
Walk through this for every function under test:
1. **Empty/null inputs** — what happens when required fields are missing?
2. **Boundary values** — min, max, zero, negative, off-by-one
3. **Type mismatches** — string where int expected, list where dict expected
4. **Malformed input** — invalid JSON, bad base64, truncated data
5. **Concurrent operations** — two tasks claiming the same resource
6. **Ordering** — actions that depend on sequence (pre-action before runner)
7. **Idempotency** — calling the same operation twice (kill an already-killed container)
8. **Resource exhaustion** — at capacity, disk full, timeout expired
9. **Partial failure** — first action succeeds, second fails (cleanup?)
## Property-Based Testing with Hypothesis
Instead of specifying individual test cases, define *properties* that must hold for all inputs. Hypothesis generates hundreds of inputs including edge cases you'd never think of.
### When to Use Property-Based Testing
- **Serialisation roundtrips:** encode → decode returns original
- **Parsers:** should never crash on any input
- **Data transformations:** invariants that hold regardless of input
- **Validators:** valid inputs accepted, invalid inputs rejected (never crash)
### When NOT to Use It
- Tests where generating valid inputs is harder than the code itself
- Tests where the "property" just restates the implementation
- UI or integration tests
### Patterns
```python
from hypothesis import given, strategies as st, assume, settings
from hypothesis import example
# Roundtrip: base64 encode/decode preserves payload
@given(st.text())
def test_base64_roundtrip(payload_str):
encoded = base64.b64encode(payload_str.encode()).decode()
decoded = base64.b64decode(encoded).decode()
assert decoded == payload_str
# Invariant: payload validation never crashes (may reject, never exception)
@given(st.dictionaries(st.text(), st.text() | st.integers() | st.none()))
def test_payload_validation_never_crashes(raw_payload):
# Should return True/False or raise ValidationError — never unhandled exception
try:
validate_payload(raw_payload)
except ValidationError:
pass # Expected for invalid input
# Pin known edge cases alongside random generation
@example("") # empty string
@example("\x00") # null byte
@example("a" * 10**6) # very long
@given(st.text())
def test_prompt_handling(prompt):
...
# Composite strategies for domain objects
@st.composite
def valid_payloads(draw):
return {
"task_id": draw(st.uuids()).hex,
"prompt": draw(st.text(min_size=1)),
"runtime": {"cli": draw(st.sampled_from(["claude", "codex"]))},
}
@given(valid_payloads())
def test_valid_payload_always_accepted(payload):
assert validate_payload(payload) is True
```
### Stateful Testing for State Machines
Hypothesis can generate sequences of operations and check invariants after each step:
```python
from hypothesis.stateful import RuleBasedStateMachine, rule, precondition
class TaskStateMachine(RuleBasedStateMachine):
def __init__(self):
super().__init__()
self.task = Task(state="pending")
@rule()
@precondition(lambda self: self.task.state == "pending")
def assign(self):
self.task.transition("assigned")
assert self.task.state == "assigned"
@rule()
@precondition(lambda self: self.task.state == "running")
def complete(self):
self.task.transition("succeeded")
assert self.task.state == "succeeded"
# Invariant: terminal states never change
@invariant()
def terminal_states_are_final(self):
if self.task.state in ("succeeded", "failed", "cancelled"):
with pytest.raises(InvalidTransition):
self.task.transition("running")
TestTaskStates = TaskStateMachine.TestCase
```
## Mutation Testing
Mutation testing answers: "If someone introduced a bug, would our tests catch it?"
Tools make small code changes (replacing `>` with `>=`, `True` with `False`, deleting statements) and check if tests still pass. Surviving mutants = test gaps.
### Setup with mutmut
```toml
# pyproject.toml
[tool.mutmut]
paths_to_mutate = "entrypoint/"
tests_dir = "tests/"
runner = "python -m pytest tests/ -x -q"
```
```bash
# Run mutation testing
mutmut run
# See surviving mutants
mutmut results
# Inspect a specific mutant
mutmut show 42
```
### Practical Guidance
- **Target: mutation score above 80%.** Scores above 90% have diminishing returns (equivalent mutants).
- **Focus on business logic** — validators, state machines, parsers. Skip glue code.
- **Use mutation testing to audit AI-generated tests.** This is the most powerful combination: AI writes tests from spec, mutation testing verifies those tests catch real faults.
- **Run on changed files only in CI** (full suite is slow). Full run nightly or pre-release.
## Test Architecture
### The Testing Pyramid for Spec-Driven Projects
| Layer | Proportion | Speed | What it catches |
|---|---|---|---|
| Unit tests | 60-70% | <1ms each | Logic errors, boundary violations, state machine bugs |
| Property-based | 10-15% | ~10ms each | Edge cases humans miss, roundtrip failures, crash inputs |
| Integration | 15-20% | ~100ms each | Component interaction bugs, mock/reality divergence |
| E2E / acceptance | 5-10% | ~1s+ each | Full-chain failures, deployment config issues |
### Test Isolation Principles
- **No test depends on another test's state.** Each test sets up its own preconditions.
- **No test depends on execution order.** `pytest-randomly` catches order dependencies.
- **No test touches the real filesystem outside `tmp_path`.** Monkeypatch paths that default to production locations (like `/workspace`).
- **No test makes network calls.** Mock HTTP, subprocess, and socket calls.
- **Integration tests are marked** (`@pytest.mark.integration`) and excluded by default.
### Fixture Architecture
```python
# conftest.py — shared fixtures, not test logic
@pytest.fixture
def minimal_payload():
"""Smallest valid payload — tests shouldn't need more unless testing specific fields."""
return {"task_id": "test-123", "prompt": "do something", "runtime": {"cli": "claude"}}
@pytest.fixture
def encode_payload():
"""Helper: dict → base64 string (how the dispatcher passes payloads)."""
def _encode(d):
return base64.b64encode(json.dumps(d).encode()).decode()
return _encode
# Per-module conftest for module-specific fixtures
# tests/test_dispatcher/conftest.py
@pytest.fixture
def mock_backend():
"""Fake container backend that records calls without Docker."""
...
```
### Negative Tests Are as Important as Positive Tests
For every "this works" test, write at least one "this fails correctly" test:
```python
# Positive: valid payload accepted
def test_p1_valid_payload_loads():
...
# Negative: missing required field rejected
def test_p3_missing_prompt_raises():
...
# Negative: wrong type rejected
def test_p_prompt_wrong_type_raises():
...
# Negative: extra unknown fields are ignored (not rejected)
def test_p_unknown_fields_ignored():
...
```
## AI Agent Testing Patterns
### The Two-Phase Rule
**Never let the same agent write both tests and implementation in one pass.** An agent that writes tests and code together will unconsciously write tests that verify its own broken assumptions.
The workflow:
1. **Phase 1:** Agent reads spec → writes tests. Human reviews tests against spec.
2. **Phase 2:** Agent (or different agent) reads spec + tests → writes implementation until tests pass.
### Hidden Test Splits
Hold back some tests that the implementing agent never sees. Use them as a final validation:
```python
# tests/test_payload.py — agent sees these during development
def test_p1_load_from_env_var(): ...
def test_p2_missing_payload_exits_1(): ...
# tests/test_payload_hidden.py — agent never sees these, run post-implementation
# (Marked with a custom marker, excluded from default run)
@pytest.mark.hidden
def test_p1_load_from_file_fallback(): ...
@pytest.mark.hidden
def test_p_concurrent_payload_loads(): ...
```
### Regression Tests from Real Bugs
Every bug found in production or during integration testing becomes a permanent test case:
```python
def test_regression_crlf_corruption():
"""Regression: smtp-oauth-relay converted \\r\\n to \\n, breaking quoted-printable.
Fixed by as_bytes(policy=email_policy.SMTP). See memory/gotchas-email-relay.md."""
...
```
These are the highest-value tests because they catch proven failure modes.
## Test Quality Metrics
### What to Measure
| Metric | Target | Why |
|---|---|---|
| Spec coverage | 100% | Every numbered requirement has at least one test |
| Mutation score | >80% | Tests catch real faults, not just inflate coverage |
| Line coverage | >90% | Baseline hygiene (necessary but not sufficient) |
| Test speed | <10s total | Fast enough for pre-commit hooks |
| Assertion density | >1 per test | Tests that don't assert don't catch anything |
### What NOT to Measure
- **100% line coverage as a goal.** Chasing 100% leads to tests that exercise code paths without meaningful assertions.
- **Test count.** 50 well-targeted tests beat 200 shallow ones.
- **Test-to-code ratio.** The ratio depends on the module's complexity, not a universal number.
## CI Integration
### Pre-commit (Every Commit)
```bash
pytest tests/ -x -q --tb=short -m "not integration"
```
### PR Validation (Every Push)
```bash
# Unit + property-based tests
pytest tests/ -q --tb=short -m "not integration"
# Mutation testing on changed files only
mutmut run --paths-to-mutate="$(git diff --name-only main... | grep '.py$' | tr '\n' ',')"
```
### Nightly
```bash
# Full mutation testing
mutmut run
# Integration tests (requires Docker)
pytest tests/ -m integration
# Hidden test validation
pytest tests/ -m hidden
```
## Python Testing Gotchas
### `subprocess.run(check=True)` Is Invisible to Mocks
When you mock `subprocess.run`, the mock replaces the entire function — including the `check=True` logic that raises `CalledProcessError`. A mock returning `CompletedProcess(returncode=1)` won't trigger the exception even though the real code uses `check=True`. To test failure paths, use `side_effect=CalledProcessError(...)` explicitly.
### Use Routing Callables for Multi-Call Subprocess Mocks
When a function calls `subprocess.run` multiple times (e.g., git config, add, diff, commit, push), a fixed `side_effect` list is fragile and breaks when call order changes. Instead, use a routing callable that inspects the command: `mock_run.side_effect = lambda cmd, **kw: route_by_command(cmd)`. Clearer, more maintainable, and self-documenting.
### Pydantic v2 `@field_validator` Doesn't Fire for Default Values
`@field_validator('field_name')` never runs when the field takes its default value (e.g., `None`). Cross-field validation logic (e.g., "if type is X then field Y is required") silently passes when the dependent field is omitted. Use `@model_validator(mode='after')` for any validation that depends on multiple fields or needs to fire even when fields take defaults.
### Never `sys.exit()` at Module Level
`sys.exit()` in an `except ImportError` block at module level kills pytest collection entirely — all tests fail, not just the ones for that module. Use a flag pattern instead: `_HAS_DEPENDENCY = False` in the except block, then check `if not _HAS_DEPENDENCY: return 1` inside the function. This allows the module to be imported and mocked even when the optional dependency is missing.
### Use `pytest.importorskip` for Optional Dependency Tests
When test files import optional packages (e.g., `sqlalchemy`, `psycopg`) at module level, pytest collection fails for the entire test suite — not just the tests that need that package. Use `mod = pytest.importorskip("sqlalchemy")` and then attribute access (`mod.text`). Also guard transitive imports: `pytest.importorskip("myapp.db.postgres_store")` if the module itself imports the optional package at module level.
### Patch Individual Functions, Not Whole Modules
Patching an entire module (e.g., `patch("mod.kubernetes.config")`) replaces exception classes with MagicMock objects. `except SomeException` then catches `MagicMock` instead of the real exception, causing tests to pass the wrong code path. Patch individual functions (`load_incluster_config`, `load_kube_config`) and leave exception classes intact so `except` clauses work correctly.
### Async Migration Requires Full Test Conversion
When migrating a codebase from sync to async, helper functions get converted but test functions are often left as sync `def`. Every test that calls an async function needs `async def` + `@pytest.mark.asyncio` + `await`. After any async migration, run tests and grep for `RuntimeWarning: coroutine '...' was never awaited` to find remaining sync-to-async gaps.
## Anti-Patterns
### Tests that mirror implementation
**Symptom:** Test asserts that function calls happen in a specific order, using mock.assert_has_calls with exact sequences. Breaks on any refactor.
**Fix:** Test behaviour (inputs → outputs), not implementation details.
### Tests without assertions
**Symptom:** `test_it_runs()` calls the function and checks it doesn't crash. No assertion on the result.
**Fix:** Every test must assert something specific about the output, side effects, or raised exceptions.
### Overmocking
**Symptom:** Every dependency is mocked. Tests pass but integration fails because mocks don't match real behaviour.
**Fix:** Mock at the boundary (subprocess, HTTP, filesystem), not between your own modules. Use real objects for internal dependencies.
### Fragile tests
**Symptom:** Tests break when unrelated code changes. Usually caused by asserting on implementation details, shared mutable state, or execution order.
**Fix:** Test the public interface. Use fixtures for setup. Isolate each test completely.
### Testing private methods
**Symptom:** Tests import `_internal_helper` and test it directly. These break on any refactor.
**Fix:** Test through the public API. If a private method is complex enough to need its own tests, it should probably be a separate module with a public interface.

80
validation.md Normal file
View File

@@ -0,0 +1,80 @@
# Validation & Deployment
## Validate Locally, Deploy Once
The single biggest time sink across projects is "deploy first, validate later." Real-world stats from a 9-milestone infrastructure project showed 50-60% of commits were fixes that could have been caught locally.
**Always validate before pushing:**
- `helm template` for Helm chart values
- `kustomize build` (or `kubectl kustomize`) for Kustomize apps
- `kubectl apply --dry-run=server` for K8s naming/schema issues
- `docker run <app> validate-configuration` for apps that support it (Authelia, Homepage, etc.)
- `docker inspect` for unfamiliar container images before writing init containers
- Lint/typecheck/test for application code
**Batch fixes locally, push once.** Each push-sync-crash-fix cycle wastes minutes and clutters Git history.
## Test the Full Chain Immediately
After wiring up any new service or endpoint, test end-to-end from the user's perspective right away. Don't assume intermediate steps working means the whole chain works.
- `curl --resolve domain:443:<ip> https://domain` to test bypassing DNS/proxy layers
- Test from the actual consumer (not same-namespace test pods for network policies)
- Test DNS resolution after deploying FQDN-based policies
## Pre-Flight Checks
Before starting a deploy or automation phase:
- Verify SSH keys are loaded (`ssh -T git@<host>`)
- Confirm environment variables and credentials are available
- Check that the target environment is in the expected state
- Verify DNS records resolve as expected
## Scripts That Change Config Must Self-Verify
After updating and restarting a service, the script should test that the change actually took effect (e.g., curl an API endpoint, check a config value). A "success" message without verification hides failures.
## Check Container Image Runtime Requirements First
Before writing deployment manifests (StatefulSets, Deployments, init containers), check the image's runtime expectations: UID it runs as, writable directories it needs, filesystem layout. Use `docker inspect` or image documentation.
Modern images often run as non-root with specific writable directory requirements that aren't obvious from docs alone. Discovering these at deploy time wastes an entire push-crash-fix cycle per missed requirement.
## Verify Counts and Summaries Mechanically
After editing specification or documentation files that include summary counts (e.g., "14 requirements"), verify them with `grep` or `wc` rather than counting manually. Manual counting of dozens of items is error-prone and produces incorrect summaries that erode trust in the documentation.
## Test Pre-Commit Hooks Manually After Adding Dependencies
Run `bash .githooks/pre-commit` (or your hook path) manually after adding new dependencies or changing test imports. Hidden virtual environments (`.venv/`) that the hook discovers before system Python can cause `ModuleNotFoundError` at commit time even though tests pass from the terminal. Discovering hook failures during a real commit wastes debugging effort on environment issues rather than code issues. After adding a dependency, check all Python environments: `find . -name "activate" -o -name "pytest"` to discover venvs, and install into each.
## Order Multi-Step Migrations Carefully
When performing multi-step changes on remote systems (port changes, firewall rules, service migrations), plan explicit ordering to avoid lockout:
1. Open the new path first (new port, new firewall rule)
2. Migrate the service to use the new path
3. Add redirects or backward-compatibility rules
4. Remove the old path
Doing all steps at once risks losing access if any step fails. Plan the ordering upfront, not mid-deploy.
## Smoke-Test Service Images Locally Before CI
Before pushing Dockerfile or service configuration changes, run `docker compose up` locally with a real database and real service images. Unit tests cannot catch deployment-category bugs: Dockerfile CMD syntax, import path errors, env var prefix mismatches, URL encoding issues, factory patterns, startup ordering. A single local `docker compose up` catches these in seconds vs. the 3+ minute CI cycle per fix.
## Stream Secrets from Source Files, Never from Context
When piping secrets into commands (base64 encoding, kubectl create secret, etc.), always stream from the source file in the same pipeline: `cat /path/to/secret | base64`. Never reconstruct a secret value from conversation context or memory — single-character typos in tokens cause authentication failures that are extremely difficult to diagnose. Save generated secrets to `local_secrets/` immediately upon creation, then reference that file for all subsequent uses.
## Verify Container Image Tags Before Writing References
Always verify that a container image tag exists before writing it into manifests, scripts, CI configs, or templates. Use `docker manifest inspect <image>:<tag>` or query the registry API directly. AI agents and cached knowledge frequently produce outdated or incorrect tag formats (e.g., `ci-0.159.0` instead of `0.159.0`, `v0.6.0` when only `v0.5.0` exists). A 5-second manifest inspect catches it immediately vs. a full push-sync-crash-fix cycle.
## Front-Load Decision Questions with Recommended Defaults
When starting a milestone with multiple architectural choices, batch all decision questions into a single set with recommended defaults for each. This gets answers in one round and prevents mid-implementation direction changes. Example: "Domain? (recommend X) | Manifest location? (recommend Y) | Database approach? (recommend Z)" — all answered at once, zero backtracking.
## Explore the Target Environment Before Planning
For infrastructure-heavy work, research the target environment's actual state before making design decisions. This means checking: what ingress controller is in use, how DNS resolves, what TLS strategy exists, what storage backends are available, what auth middleware is configured. Discovering these facts during planning (not implementation) prevents architectural surprises. In agent-orchestrated workflows, dedicated exploration agents that survey the target environment pay for themselves by eliminating implementation detours.