claude-foundations/CLAUDE.md

# CLAUDE.md — General Project Guidelines

## Session Start

1. **Immediately** (without waiting for user input) list the project directories under `~/dev/claude/` (excluding `secrets/`) and present a numbered menu like:

   > What are we working on today?
   >
   > 1. **cluster-bootstrap** — Kubernetes homelab cluster
   > 2. **custom-claude-skills** — Reusable Claude Code skills
   > ...
   > N-1. **No project right now** — just chat
   > N. **New project!** — start something new

   Scan the directories at runtime so the list is always current. Include a brief description if the project has a CLAUDE.md or README.md you can glean one from.

2. Based on the user's choice:
   - **Existing project**: `cd` into the directory, read all `.md` files, and read `~/dev/claude/secrets/` (read-only reference — review every file to refresh context). Then read the [Best Practices Index](best-practices/INDEX.md) in the `claude-foundations` repo and load any topic files relevant to the selected project's technology stack. Ask clarifying questions if anything is unclear or incomplete, and note context in MEMORY.md.
   - **No project right now**: Do nothing further — just respond normally.
   - **New project!**: Follow the "New Projects" section below. Also read `~/dev/claude/secrets/` as above. Read the [Best Practices Index](best-practices/INDEX.md) in the `claude-foundations` repo and load topic files relevant to the new project's technology stack.

## Secrets (`~/dev/claude/secrets/`)

**CRITICAL — treat this folder with extreme paranoia:**

- Files in `~/dev/claude/secrets/` are **read-only**. Never edit them.
- **Never** copy, echo, write, or reproduce secret values into any other file — not MEMORY.md, not CLAUDE.md, not commit messages, not scripts, not tool output, nowhere.
- **Never** include secret values in git commits, diffs, or changelogs of any project.
- **Never** pass secret values as command-line arguments (visible in `ps` output). Use `@file` references, environment variables sourced at runtime, or stdin.
- When a task requires a secret, read it at execution time from the secrets folder and use it ephemerally. Do not cache or persist the value.
- It is acceptable to reference the **existence** of a secret file (e.g., "credentials are in `~/dev/claude/secrets/gitea/ai_enablement`") but never its contents.

## New Projects

If this is a new project:

1. Create a new directory under `~/dev/claude/<project-name>/`
2. **Ask the user** which Gitea user/org the repo should be created under (e.g., `homelab`, a personal user, etc.) before setting up the remote
3. Create the initial standard files:
   - **CLAUDE.md** — Project-specific architecture, conventions, repo structure, and working instructions for Claude
   - **MEMORY.md** — Persistent learnings, gotchas, reflections, and process improvements
   - **FUTURE.md** — Ideas and improvements not on the active roadmap (Problem/Idea/Open questions/Depends on format)
   - **README.md** — Human-readable overview, quick start, milestones, and scripts reference
4. Parse the other CLAUDE.md files from sibling project folders in `~/dev/claude/`, and based on the type of project being considered, bring over related practices, guidelines, and learnings

## Source Control

- All projects are hosted on **Gitea** (`gitea.oreillyit.nz`) as the primary remote — prefer this hostname over `gitea.homelab.internal` (same instance, but the public name enables external access)
- Migrate existing remotes from `gitea.homelab.internal` to `gitea.oreillyit.nz` when convenient
- AI-focused projects go under the **`skynet`** org; infrastructure projects under **`homelab`**
- SSH workflows preferred. SSH config uses host aliases per Gitea user:
  - `gitea.oreillyit.nz-homelab` → authenticates as `cluster-administrator` (key: `~/.ssh/gitea-cluster-admin`)
  - `gitea.oreillyit.nz-ai-enablement` → authenticates as `ai_enablement` (key: `~/.ssh/gitea.ai-enablement`)
- Git remote URLs use the alias: `git@gitea.oreillyit.nz-<user>:<org>/<repo>.git`
  - Example: `git@gitea.oreillyit.nz-ai-enablement:skynet/custom-claude-skills.git`
- Optionally push-mirror to GitHub for public visibility
- Use meaningful commit messages; prefer small, focused commits over large batches
- Enable pre-commit hooks where appropriate (secret detection, linting, formatting)
- Never commit secrets in plaintext — use SOPS + age or equivalent encryption

## Documentation Standards

Every project maintains four core markdown files:

### CLAUDE.md (per-project)
The primary reference for Claude sessions. Should contain:
- Project overview and architecture
- Repository structure (keep updated as the project evolves)
- Key design decisions with rationale
- Conventions and coding standards
- Environment details (IPs, URLs, credentials references)
- Common operations / how-to recipes

### MEMORY.md (Tiered Memory System)
Long-running projects accumulate significant context. To keep MEMORY.md useful rather than bloated, use a **tiered memory** structure:

**MEMORY.md** is a **thin index only** — one-line descriptions with links to topic files in `memory/`. No content lives in MEMORY.md itself. Think of it as a card catalog. Keep it under ~50 lines.

**memory/** contains the actual content, split by topic:
- `memory/project-status.md` — Current milestone, what's next, blockers
- `memory/network.md` — IPs, VIPs, subnets, topology
- `memory/gotchas-<topic>.md` — Gotchas grouped by technology (e.g., `gotchas-cilium.md`, `gotchas-authelia.md`)
- `memory/process-lessons.md` — How-to-work-with-this-repo lessons for Claude
- `memory/m<N>-reflection.md` — One file per milestone reflection (these are time-bound, so per-file is natural)
- `memory/decisions.md` — Architecture and design decisions made during planning

**Principles:**
- **Split by topic, not by time.** A Cilium gotcha belongs in `gotchas-cilium.md` whether discovered in M5 or M8.
- **Milestone reflections are the exception** — inherently time-bound, one file per milestone.
- **Index descriptions matter.** They're used to decide what to read. "Cilium L2/LB gotchas and externalTrafficPolicy quirks" beats "cluster stuff".
- **Prune aggressively.** If a gotcha was fixed (e.g., chart version upgraded past the bug), delete it. Stale memory is worse than no memory.
- **Each memory file should be self-contained and greppable.** Include enough context that the file makes sense on its own.
- **Deduplicate with CLAUDE.md.** Conventions and patterns that are stable should live in CLAUDE.md. Memory files are for learnings, gotchas, and reflections that accumulate over time. If something in memory has graduated to a stable convention, move it to CLAUDE.md and remove it from memory.

**When reading memory at session start:** Read MEMORY.md (the index), then selectively read topic files relevant to the current task. Don't read all memory files unless doing a broad review.

**When writing memory after a milestone:** Create the reflection file, update any affected topic files (new gotchas, updated status), and update the index.

### FUTURE.md
Backlog of improvement ideas, each with:
- **Problem:** What's painful or manual today
- **Idea:** What the improvement looks like
- **Open questions:** Unknowns to research before starting
- **Depends on:** Other items or milestones that should come first

### README.md
Human-readable project documentation:
- Architecture summary
- Quick start / setup instructions
- Milestone table with status
- Scripts section listing every script with purpose and usage

## Knowledge Distillation Pipeline

Three skills form a continuous learning pipeline across projects:

1. **`/log`** — Run at end of session. Captures decisions, gotchas, open questions to `memory/log/YYYY-MM-DD.<HHMMSS>.md` in the current project. Also prunes old reflected logs.
2. **`/reflect-logs`** — Run periodically. Processes unprocessed session logs into topic memory files (`memory/gotchas-*.md`, `memory/process-lessons.md`, etc.). Flags stale entries. Tracks state in `.reflection-state.json`.
3. **`/distill-best-practices`** — Run from any project. Reads changed memory files across all tracked projects and proposes updates to the `claude-foundations` repo's `best-practices/` folder. Tracks state in `best-practices/.distill-state.json`.

### State Files
- **`.reflection-state.json`** — Per-project, tracks which logs have been reflected on (md5 hashes of log file content)
- **`best-practices/.distill-state.json`** — In claude-foundations, tracks git SHAs per project at time of last distillation
- **`settings.yaml`** — In claude-foundations, configures log retention (default 7 days), max logs per reflection run, and tracked project list

### Log Format
Session logs use structured markdown with parseable section headers: Summary, Decisions, Gotchas Discovered (tagged with `[topic]` for routing), Open Questions, Key Context, Process Notes. Empty sections are omitted.

### Pruning
- Reflected logs older than `log.retention_days` (default: 7) are automatically deleted by `/log`
- Unreflected logs older than `log.warn_unreflected_days` (default: 14) trigger a warning instead of deletion
- `/reflect-logs` flags stale memory entries (version-specific bugs that have been fixed, manual processes that have been automated)

## Milestones

Break projects into numbered milestones (M1, M2, ...). Every milestone completion MUST include:

1. **Verification script** (`scripts/verify-m<N>.sh`) — automated checks confirming all milestone outcomes. Scripts should be idempotent, non-destructive, and return non-zero on failure. Use colour output (green/red) for pass/fail.
2. **Milestone reflection** in `memory/m<N>-reflection.md` — review the entire conversation and capture:
   - Process improvements (what slowed us down, wrong assumptions, backtracking)
   - Key knowledge for reproduction (gotchas, version quirks, debugging detours)
   - Scripts and automation opportunities (repeated command sequences → scripts)
   - Future improvement ideas (add to FUTURE.md)
   - Update affected topic files in `memory/` (new gotchas, updated status) and the MEMORY.md index
3. **Updated README.md** — scripts section, milestone table, any new setup steps
4. **Updated CLAUDE.md** — repo structure, conventions, new patterns discovered

## Version Management

- Use the latest stable version of dependencies unless pinned for a reason
- Verify versions from live sources (`helm search repo`, upstream docs, package registries) — don't rely on memory
- Document the reason in a comment if a version is intentionally pinned below latest
- Check compatibility matrices before upgrading (e.g., Talos ↔ Kubernetes, framework ↔ runtime)

## Secrets Management

- SOPS + age is the standard encryption tool across all projects
- The `.sops.yaml` at the repo root defines path-based encryption rules
- Filenames containing `secret` trigger SOPS encryption via pre-commit hooks
- Non-secret files must NOT contain `secret` in their name
- Keep unencrypted secrets in `local_secrets/` (gitignored)

## Scripting Conventions

- All scripts live in `scripts/` and run from the repository root
- Scripts should be idempotent and safe to re-run
- Use colour output for pass/fail indicators in verification scripts
- Verification scripts should check for default/insecure credentials and print remediation instructions on failure
- Scripts should exit non-zero on failure so `&&` chains work naturally

## Process Principles

These are hard-won lessons from real project work:

- **Validate locally, deploy once.** Don't use the live environment as a test bed. Catch errors with local validation tools before pushing (`helm template`, `kustomize build`, lint/typecheck, `docker inspect`, etc.). Batch fixes locally, push once.
- **Check before you act.** Before writing firewall/network rules, check actual routing (`ip route get`). Before running config management with variables, ensure values are real, not placeholders. Before assuming a container has a shell, `docker inspect` it.
- **Test the full chain immediately.** After wiring up a new service or endpoint, test end-to-end from the user's perspective right away. Don't assume intermediate steps working means the whole chain works.
- **Verify scripts should be environment-resilient.** Avoid needing sudo or special access. Test from the accessible side of a connection. Use `curl --resolve` to bypass DNS/proxy layers when testing direct connectivity.
- **Automate repeated sequences.** If you run the same 3+ commands in sequence more than once, it should become a script.
- **Reflect after milestones.** Don't just finish — review what happened, what went wrong, what can be improved. Write it down so future sessions benefit.