Add best practices, hooks, memory files, and scripts from recent sessions

Includes: spec-driven and test-driven development best practices,
reproduce-before-fixing debugging workflow, require-plan-file hook,
find-project-root script, session logs, memory files for decisions/
gotchas/process-lessons, and updates to existing best practice topics.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

memory/decisions.md

@@ -0,0 +1,53 @@
+# Architecture & Design Decisions
+
+## Knowledge Pipeline: Separate /reflect-logs from /reflect
+
+`/reflect-logs` handles continuous log processing; `/reflect` handles milestone reflections. Different purpose, different cadence — combining them would overcomplicate the milestone skill.
+
+## Knowledge Pipeline: MD5 for reflection state, git SHAs for distill state
+
+Log files may not be committed when reflected on, so MD5 of file content is the right identity. `/distill-best-practices` works across committed repos, so git SHAs are appropriate there.
+
+## Knowledge Pipeline: Pruning happens in /log, not /reflect-logs
+
+`/log` runs most frequently (every session end), so it naturally keeps the log directory clean as a side effect.
+
+## Knowledge Pipeline: /distill-best-practices is interactive
+
+Cross-project convention changes need human judgment. The skill proposes updates and waits for approval before writing.
+
+## Session Logs: Timestamp-based IDs (HHMMSS)
+
+Human-readable, naturally sorted, no external dependencies. Format: `YYYY-MM-DD.HHMMSS.md`.
+
+## Linting: Formatter exit codes vs hook exit codes
+
+Formatter scripts exit 1 on lint errors. The dispatcher hook decides the final exit code (exit 2 for PostToolUse feedback). This separates formatter logic from hook semantics — same scripts work for both PostToolUse and pre-commit.
+
+## Linting: Checkpoint via git hash-object with .pre-lint sidecar
+
+Fast (~1ms), no commits or stash needed, orphan blobs auto-GC'd. Falls back to `cp` outside git repos.
+
+## Linting: Project opt-in via formatter symlinks
+
+Projects opt in by having a `formatters/` directory with symlinks back to canonical scripts. Zero-config, visible in `ls`, no parsing needed. The hook walks up the directory tree to find `formatters/`.
+
+## CLAUDE.md: Remove technology-specific sections from root
+
+Ansible and Helm sections removed from root CLAUDE.md — already covered with more detail in `best-practices/ansible.md` and `best-practices/helm.md`. Technology-specific practices belong in best-practices, not root guidelines.
+
+## context-load: Walk upward collecting context files
+
+Walks from cwd upward collecting CLAUDE.md, CONTEXT.md, MEMORY.md, BESTPRACTICES.md at each level. Gives hierarchical context inheritance — highest ancestor provides global guidelines, project dir provides specifics.
+
+## context-load: Dedup via readlink -f
+
+Root CLAUDE.md is a symlink to claude-foundations. Without dedup it would load twice. `readlink -f` resolves all symlinks before comparison.
+
+## context-load: Tree depth 3
+
+Deep enough to show project structure without overwhelming output. Applied at every CLAUDE.md location.
+
+## CONTEXT.md follows MEMORY.md pattern
+
+Thin index + `context/` folder. Consistency with MEMORY.md. CONTEXT.md focuses on active work for agent orientation; MEMORY.md on accumulated learnings.

memory/gotchas-skills.md

@@ -0,0 +1,13 @@
+# Skills Gotchas
+
+## Broken symlinks cause silent failures under set -e
+
+**Symptom:** install.sh fails with exit 1 on a broken symlink.
+**Cause:** `readlink -f` on a broken symlink returns an empty string, causing comparison failure under `set -e`.
+**Fix:** Remove stale symlinks before re-running install. When skills move directories (e.g., from `~/dev/claude/custom-claude-skills/` to `~/dev/claude/projects/custom-claude-skills/`), old symlinks break.
+
+## Skills created mid-session are not available as slash commands
+
+**Symptom:** A newly created skill doesn't appear when you type `/skillname`.
+**Cause:** Skills are discovered at session start, not dynamically during the session.
+**Fix:** Start a new Claude Code session to pick up newly created skills.

memory/log/2026-03-13.115251.md

@@ -0,0 +1,25 @@
+# Session Log — 2026-03-13
+
+## Summary
+Cleaned up the root CLAUDE.md (removed duplicated Ansible/Helm sections, consolidated validation guidance, fixed best-practices references) and built a `context-load` / `start-claude` script pair for automated session context gathering. Introduced the CONTEXT.md pattern for future independent agent work.
+
+## Decisions
+- Decision: Remove Ansible and Helm sections from root CLAUDE.md — Rationale: already covered with more detail in `best-practices/ansible.md` and `best-practices/helm.md`; technology-specific practices belong in best-practices, not the root guidelines
+- Decision: Fold "Validate Before Deploying" into Process Principles — Rationale: was duplicated content; the examples fit naturally in the existing bullet point
+- Decision: context-load walks upward from cwd collecting CLAUDE.md, CONTEXT.md, MEMORY.md, BESTPRACTICES.md — Rationale: gives hierarchical context inheritance; highest ancestor provides global guidelines, project dir provides specifics
+- Decision: Tree depth 3 from every CLAUDE.md location — Rationale: deep enough to show project structure without overwhelming output
+- Decision: Dedup loaded files via `readlink -f` — Rationale: root CLAUDE.md is a symlink to claude-foundations; without dedup it would load twice
+- Decision: CONTEXT.md follows MEMORY.md pattern (thin index + `context/` folder) — Rationale: consistency; CONTEXT.md focuses on active work for agent orientation, MEMORY.md on accumulated learnings
+
+## Key Context
+- `~/dev/claude/CLAUDE.md` is a symlink to `~/dev/claude/projects/claude-foundations/CLAUDE.md` — this is intentional, claude-foundations is the canonical source
+- `context-load` output is passed via `--append-system-prompt` by the `start-claude` wrapper
+- Both scripts symlinked into `~/sbin/`
+- CONTEXT.md is intended for future Docker-based independent agent operation — each agent gets full context load, starting prompt points to relevant CONTEXT.md entry
+- git-status-report output is ANSI-stripped before inclusion in context
+- Also committed the previously uncommitted linting system (18 files) from the earlier session
+
+## Process Notes
+- Session was efficient — cleanups and script creation done in parallel with minimal iteration
+- The context-load smoke tests from different directories caught the symlink dedup working correctly
+- Previous session's linting work was uncommitted — worth running `git-status-report` at session start to catch this pattern

memory/log/2026-03-15.225345.md

@@ -0,0 +1,22 @@
+# Session Log — 2026-03-15
+
+## Summary
+Fixed the `/distill-best-practices` skill which was broken due to hardcoded and relative paths in `!`command`` blocks. Replaced all paths with `CLAUDE_PROJECT_ROOT` env var for portability, added a `find-project-root` helper script, and updated `settings.yaml` to use relative paths. Also fixed the same relative-path issue in `/log` and `/reflect-logs` skills.
+
+## Decisions
+- Decision: Use `CLAUDE_PROJECT_ROOT` env var for all cross-project path resolution in skills — Rationale: Makes skills shareable with colleagues; hardcoded `~/dev/claude/` paths are user-specific and relative `../` paths break depending on CWD
+- Decision: Add Step 0 (detect project root) as runtime fallback in distill skill — Rationale: Skills should degrade gracefully if env var isn't set; Claude can walk up the directory tree to find highest CLAUDE.md
+- Decision: `settings.yaml` paths relative to CLAUDE_PROJECT_ROOT, not absolute — Rationale: Portability; `projects_dir: projects` instead of `~/dev/claude/projects`
+- Decision: Added `extra_projects` section to settings.yaml for projects outside `projects_dir` — Rationale: `small-scripts` lives at root level, not under `projects/`
+
+## Gotchas Discovered
+- **[skills]** Symptom: `/distill-best-practices` failed with sandbox error — `cat ../claude-foundations/...` resolved to `/home/paul/dev/claude-foundations/` (outside sandbox) when CWD was `~/dev/claude/` — Fix: Replace all relative and hardcoded paths with `${CLAUDE_PROJECT_ROOT}` env var
+- **[skills]** Symptom: `!`command`` blocks can't use `$()` command substitution — Fix: Use env var expansion (`${CLAUDE_PROJECT_ROOT}`) which works, and fall back to runtime detection in skill instructions
+- **[skills]** Symptom: Claude Code Bash tool doesn't persist `export` across `;`-separated commands in the same invocation when the variable is used in file path arguments — Fix: Use `bash -c '...'` wrapper or ensure var is in shell profile
+
+## Key Context
+- `settings.yaml` now tracks 6 projects under `distill.projects` plus `small-scripts` under `extra_projects`
+- Added projects: `agent-runtimes`, `claude-foundations`, `cluster-apps/octopus-deploy`, `hugo-accelerator`
+- `find-project-root` script created at `claude-foundations/scripts/find-project-root` — walks up from CWD to find highest CLAUDE.md
+- `CLAUDE_PROJECT_ROOT` export added to `~/.bashrc`
+- Three skills updated: `distill-best-practices` (full rewrite of paths), `log` and `reflect-logs` (settings fallback path)

memory/process-lessons.md

@@ -0,0 +1,29 @@
+# Process Lessons
+
+## Always run git-status-report at session start
+
+Previous sessions may leave uncommitted work. Running `git-status-report` (or checking git status) at session start catches this pattern before it compounds.
+
+## Use plan mode for architectural tasks
+
+Plan mode (Shift+Tab twice) works well for designing systems before implementing. Explore existing patterns, design the architecture, get approval, then execute. Implementation is straightforward when the plan is thorough.
+
+## PostToolUse exit code 2 feeds errors back to Claude
+
+Exit code 2 from a PostToolUse hook sends stderr content back to Claude as feedback without blocking the edit. Exit 0 = silent success. Exit 1 = hard block.
+
+## Agent-type hooks are read-only
+
+Hooks with `type: "command"` cannot use Edit/Write tools. Lint fixing from hooks must happen via the Agent tool subagent, not directly in hooks.
+
+## All hooks in a matcher array run in parallel
+
+Multiple hooks registered for the same matcher execute concurrently, not sequentially. Design hooks to be independent.
+
+## SSH key for ai_enablement is password-protected
+
+Needs ssh-agent loaded before git push to Gitea. If push hangs, check that the key is added to the agent.
+
+## Batch parallel file creation for efficiency
+
+Creating many independent files in a single Write batch (e.g., 11 best-practices files at once) is significantly faster than sequential creation.

memory/runbook-new-argocd-app.md

@@ -0,0 +1,222 @@
+# Runbook: Adding a New ArgoCD-Driven App to the Cluster
+
+Step-by-step procedure for deploying a new application to the homelab Kubernetes cluster via ArgoCD GitOps. Derived from deploying Octopus Deploy (2026-03-13) and existing apps (Authelia, Homepage, Email Relay).
+
+## Decision: In-repo vs Separate Repo
+
+| Approach | When to use | Example |
+|----------|-------------|---------|
+| **In cluster-bootstrap** (`platform/<app>/`) | Tightly coupled to cluster lifecycle, simple apps | Authelia, Homepage, Email Relay |
+| **Separate Gitea repo** | Independent lifecycle, external contributors, large/complex apps | Octopus Deploy |
+
+Most apps go in cluster-bootstrap under `platform/`. Use a separate repo only when there's a clear reason.
+
+## Step 1: Create Manifests
+
+### Directory structure (in-repo)
+
+```
+platform/<app>/
+├── kustomization.yaml
+├── namespace.yaml
+├── statefulset.yaml or deployment.yaml
+├── service.yaml
+├── ingressroute.yaml          # If externally accessible
+├── ksops-generator.yaml       # If app has secrets
+└── <name>-secret.sops.yaml    # SOPS-encrypted secrets
+```
+
+### Directory structure (separate repo)
+
+```
+<app>/
+├── .sops.yaml                 # SOPS encryption rules (same age key as cluster-bootstrap)
+├── .gitignore                 # local_secrets/
+├── kustomization.yaml
+├── namespace.yaml
+├── <component>/               # Subdirectories per component
+│   ├── statefulset.yaml
+│   └── service.yaml
+├── ksops-generator.yaml
+├── *-secret.sops.yaml
+├── local_secrets/             # Gitignored plaintext secrets for setup
+└── scripts/
+    └── setup.sh               # Secret generation + SOPS encryption
+```
+
+### Kustomization pattern
+
+```yaml
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - namespace.yaml
+  - <component>/statefulset.yaml
+  - <component>/service.yaml
+  - ingressroute.yaml
+generators:
+  - ksops-generator.yaml       # Only if secrets exist
+```
+
+### KSOPS generator pattern
+
+```yaml
+apiVersion: viaduct.ai/v1
+kind: ksops
+metadata:
+  name: <app>-secret-generator
+  annotations:
+    config.kubernetes.io/function: |
+      exec:
+        path: ksops
+files:
+  - ./<name>-secret.sops.yaml
+```
+
+### SOPS config (separate repo only)
+
+```yaml
+creation_rules:
+  - path_regex: .*secret.*\.yaml$
+    encrypted_regex: "^(data|stringData)$"
+    age: >-
+      age1edc9agzzs8cngd2rsvfhm8aeucnlq2clmj36jh0rrkwuj073fyssr0u4x9
+```
+
+In-repo apps inherit from the cluster-bootstrap root `.sops.yaml`.
+
+## Step 2: Namespace Checklist
+
+```yaml
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: <app>
+  labels:
+    # Only if app needs privileged containers (DinD, host networking, etc):
+    pod-security.kubernetes.io/enforce: privileged
+    pod-security.kubernetes.io/audit: privileged
+    pod-security.kubernetes.io/warn: privileged
+```
+
+**Always check:** Does any container need `securityContext.privileged: true` or host-level access? If yes, add PodSecurity labels. Forgetting this causes silent pod creation failures.
+
+## Step 3: Secrets
+
+1. Create plaintext secret YAML in `local_secrets/` (gitignored)
+2. Encrypt with SOPS: `sops --encrypt local_secrets/<name>-secret.yaml > <name>-secret.sops.yaml`
+3. Reference in `ksops-generator.yaml`
+4. Secret filenames **must** contain `secret` (triggers SOPS rules)
+5. Non-secret files must **not** contain `secret` in their name
+
+## Step 4: Traefik IngressRoute (if externally accessible)
+
+```yaml
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: <app>
+  namespace: <app>
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`<app>.oreillyit.nz`)
+      kind: Rule
+      services:
+        - name: <app>
+          port: 80
+```
+
+If the app needs Authelia protection, add a middleware reference. If the app handles its own auth, omit it.
+
+## Step 5: ArgoCD Application
+
+Create `bootstrap/apps/<app>.yaml` in cluster-bootstrap:
+
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: <app>
+  namespace: argocd
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io
+spec:
+  project: default
+  source:
+    # In-repo:
+    repoURL: http://gitea.bootstrap.homelab.internal/homelab/cluster-bootstrap.git
+    targetRevision: main
+    path: platform/<app>
+    # Separate repo:
+    # repoURL: http://gitea.bootstrap.homelab.internal/homelab/<app>.git
+    # targetRevision: main
+    # path: .
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: <app>
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - ServerSideApply=true
+      - CreateNamespace=true
+```
+
+After pushing, the root app may not pick it up immediately. Force refresh:
+```bash
+kubectl annotate application root -n argocd argocd.argoproj.io/refresh=normal --overwrite
+```
+
+## Step 6: Caddy Reverse Proxy (if externally accessible)
+
+Add to `external/vps/inventory.yml` under `caddy_sites`:
+
+```yaml
+- domain: "<app>.oreillyit.nz"
+  type: public
+  upstream: "https://10.111.1.100:443"   # Traefik VIP
+  upstream_tls_insecure: true             # Traefik uses self-signed certs
+  comment: "<App description>"
+```
+
+Deploy:
+```bash
+ansible-playbook -i external/vps/inventory.yml external/vps/playbook.yml --tags compose
+```
+
+Caddy automatically obtains a Let's Encrypt certificate on first request.
+
+## Step 7: DNS
+
+Add a Cloudflare A record for `<app>.oreillyit.nz` pointing to the VPS IP (`43.224.182.153`). Use orange cloud (proxied) for most services.
+
+## Step 8: Verify
+
+```bash
+# ArgoCD status
+kubectl get application <app> -n argocd
+
+# Pod health
+kubectl get pods -n <app>
+
+# PVC status (if applicable)
+kubectl get pvc -n <app>
+
+# External access
+curl -sk -o /dev/null -w "%{http_code}" https://<app>.oreillyit.nz/
+```
+
+## Common Pitfalls
+
+| Pitfall | Symptom | Prevention |
+|---------|---------|------------|
+| Missing PodSecurity labels | `violates PodSecurity "baseline"` in StatefulSet events | Always check if any container needs privileged mode |
+| Container runs as non-root | Permission denied on writable dirs | Check image docs / `docker inspect` before writing manifests |
+| ArgoCD polling delay | New app doesn't appear after push | Annotate root app with `refresh=normal` |
+| StatefulSet CrashLoopBackOff | Updated spec not applied to pod | Delete the crashlooping pod manually |
+| Gitea repo permissions | `not authorized to write` on push | Grant collaborator access before pushing |
+| Proxmox CSI lock contention | PVC provisioning failures with lock timeout | Transient — CSI retries automatically |
+| Caddy not configured | Connection refused or SSL error on domain | Add site to inventory.yml and deploy with Ansible |

memory/script-install-hooks.md

@@ -19,7 +19,7 @@ One-time setup. Re-run after adding new hooks.
 
 1. Iterates over `hooks/*.sh`
 2. Creates symlinks in `~/.claude/hooks/` (force-overwrites existing)
-3. Prints the JSON config for `~/.claude/settings.json` covering PreCompact and PostToolUse matchers
+3. Prints the JSON config for `~/.claude/settings.json` covering PreCompact, PostToolUse, and PreToolUse matchers
 
 ## Gotchas
 

memory/script-require-plan-file.md

@@ -0,0 +1,45 @@
+---
+name: require-plan-file
+description: PreToolUse hook that blocks ExitPlanMode unless a *-PLAN.md file exists in the current project root
+type: reference
+---
+
+# script: require-plan-file
+
+**Location:** `hooks/require-plan-file.sh`
+**Symlinked to:** `~/.claude/hooks/require-plan-file.sh`
+
+## Purpose
+
+Enforces the plan-file convention: Claude cannot exit plan mode until it has written a `[MILESTONE]-[PURPOSE]-PLAN.md` file in the project root.
+
+## How it works
+
+Fires as a `PreToolUse` hook on `ExitPlanMode`. Reads `cwd` from the JSON input on stdin, checks for any `*-PLAN.md` file in that directory:
+
+- **File found** → exits 0 (allows ExitPlanMode to proceed)
+- **No file found** → exits 2 (blocks ExitPlanMode, stderr message injected into Claude's context)
+
+The exit 2 message tells Claude exactly what to do, so it self-corrects and writes the file before retrying.
+
+## Settings.json config
+
+```json
+"PreToolUse": [
+  {
+    "matcher": "ExitPlanMode",
+    "hooks": [{ "type": "command", "command": "~/.claude/hooks/require-plan-file.sh" }]
+  }
+]
+```
+
+## Naming convention
+
+`[MILESTONE]-[PURPOSE]-PLAN.md` — e.g. `M2-auth-PLAN.md`, `M3-monitoring-PLAN.md`, `initial-setup-PLAN.md`
+
+Plan files are committed to the project repo as a persistent record of planning decisions.
+
+## Gotchas
+
+- `cwd` in the hook input is the Claude session's working directory, not necessarily the repo root — works correctly as long as Claude is `cd`'d into the project root (the standard pattern).
+- If `cwd` cannot be parsed from stdin, the hook exits 0 (fails open) to avoid blocking legitimate use.

memory/skill-context-load.md

@@ -0,0 +1,28 @@
+# skill: /context-load
+
+**Location:** `custom-claude-skills/skills/context-load/SKILL.md`
+**Symlinked to:** `~/.claude/skills/context-load`
+
+## Purpose
+
+Reloads project context into the conversation after `/clear` or when context has been lost mid-session. Equivalent to the context that `start-claude` injects at session start via `--append-system-prompt`.
+
+## Usage
+
+```
+/context-load
+```
+
+Run from any project directory. The skill will gather context from cwd upward, just like the `context-load` script does at launch.
+
+## How it works
+
+- Uses `!`context-load`` to run the `scripts/context-load` script at skill load time
+- The script output (CLAUDE.md files, trees, CONTEXT.md, MEMORY.md, BESTPRACTICES.md, git status) is injected directly into the skill prompt
+- Claude reads and internalizes the output, then confirms what it loaded
+
+## Gotchas
+
+- Depends on `context-load` being on `$PATH` (symlinked to `~/sbin/context-load`)
+- Output size scales with the number of projects in the directory hierarchy — deep nesting or large index files may use significant tokens
+- Only loads index files, not topic files from `memory/` or `context/` — Claude must use Read tool for those if needed