Add best practices, hooks, memory files, and scripts from recent sessions
Includes: spec-driven and test-driven development best practices, reproduce-before-fixing debugging workflow, require-plan-file hook, find-project-root script, session logs, memory files for decisions/ gotchas/process-lessons, and updates to existing best practice topics. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Paul O'Reilly
222
memory/runbook-new-argocd-app.md
Normal file
222
memory/runbook-new-argocd-app.md
Normal file
@@ -0,0 +1,222 @@
|
||||
# Runbook: Adding a New ArgoCD-Driven App to the Cluster
|
||||
|
||||
Step-by-step procedure for deploying a new application to the homelab Kubernetes cluster via ArgoCD GitOps. Derived from deploying Octopus Deploy (2026-03-13) and existing apps (Authelia, Homepage, Email Relay).
|
||||
|
||||
## Decision: In-repo vs Separate Repo
|
||||
|
||||
| Approach | When to use | Example |
|
||||
|----------|-------------|---------|
|
||||
| **In cluster-bootstrap** (`platform/<app>/`) | Tightly coupled to cluster lifecycle, simple apps | Authelia, Homepage, Email Relay |
|
||||
| **Separate Gitea repo** | Independent lifecycle, external contributors, large/complex apps | Octopus Deploy |
|
||||
|
||||
Most apps go in cluster-bootstrap under `platform/`. Use a separate repo only when there's a clear reason.
|
||||
|
||||
## Step 1: Create Manifests
|
||||
|
||||
### Directory structure (in-repo)
|
||||
|
||||
```
|
||||
platform/<app>/
|
||||
├── kustomization.yaml
|
||||
├── namespace.yaml
|
||||
├── statefulset.yaml or deployment.yaml
|
||||
├── service.yaml
|
||||
├── ingressroute.yaml # If externally accessible
|
||||
├── ksops-generator.yaml # If app has secrets
|
||||
└── <name>-secret.sops.yaml # SOPS-encrypted secrets
|
||||
```
|
||||
|
||||
### Directory structure (separate repo)
|
||||
|
||||
```
|
||||
<app>/
|
||||
├── .sops.yaml # SOPS encryption rules (same age key as cluster-bootstrap)
|
||||
├── .gitignore # local_secrets/
|
||||
├── kustomization.yaml
|
||||
├── namespace.yaml
|
||||
├── <component>/ # Subdirectories per component
|
||||
│ ├── statefulset.yaml
|
||||
│ └── service.yaml
|
||||
├── ksops-generator.yaml
|
||||
├── *-secret.sops.yaml
|
||||
├── local_secrets/ # Gitignored plaintext secrets for setup
|
||||
└── scripts/
|
||||
└── setup.sh # Secret generation + SOPS encryption
|
||||
```
|
||||
|
||||
### Kustomization pattern
|
||||
|
||||
```yaml
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- namespace.yaml
|
||||
- <component>/statefulset.yaml
|
||||
- <component>/service.yaml
|
||||
- ingressroute.yaml
|
||||
generators:
|
||||
- ksops-generator.yaml # Only if secrets exist
|
||||
```
|
||||
|
||||
### KSOPS generator pattern
|
||||
|
||||
```yaml
|
||||
apiVersion: viaduct.ai/v1
|
||||
kind: ksops
|
||||
metadata:
|
||||
name: <app>-secret-generator
|
||||
annotations:
|
||||
config.kubernetes.io/function: |
|
||||
exec:
|
||||
path: ksops
|
||||
files:
|
||||
- ./<name>-secret.sops.yaml
|
||||
```
|
||||
|
||||
### SOPS config (separate repo only)
|
||||
|
||||
```yaml
|
||||
creation_rules:
|
||||
- path_regex: .*secret.*\.yaml$
|
||||
encrypted_regex: "^(data|stringData)$"
|
||||
age: >-
|
||||
age1edc9agzzs8cngd2rsvfhm8aeucnlq2clmj36jh0rrkwuj073fyssr0u4x9
|
||||
```
|
||||
|
||||
In-repo apps inherit from the cluster-bootstrap root `.sops.yaml`.
|
||||
|
||||
## Step 2: Namespace Checklist
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: <app>
|
||||
labels:
|
||||
# Only if app needs privileged containers (DinD, host networking, etc):
|
||||
pod-security.kubernetes.io/enforce: privileged
|
||||
pod-security.kubernetes.io/audit: privileged
|
||||
pod-security.kubernetes.io/warn: privileged
|
||||
```
|
||||
|
||||
**Always check:** Does any container need `securityContext.privileged: true` or host-level access? If yes, add PodSecurity labels. Forgetting this causes silent pod creation failures.
|
||||
|
||||
## Step 3: Secrets
|
||||
|
||||
1. Create plaintext secret YAML in `local_secrets/` (gitignored)
|
||||
2. Encrypt with SOPS: `sops --encrypt local_secrets/<name>-secret.yaml > <name>-secret.sops.yaml`
|
||||
3. Reference in `ksops-generator.yaml`
|
||||
4. Secret filenames **must** contain `secret` (triggers SOPS rules)
|
||||
5. Non-secret files must **not** contain `secret` in their name
|
||||
|
||||
## Step 4: Traefik IngressRoute (if externally accessible)
|
||||
|
||||
```yaml
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: IngressRoute
|
||||
metadata:
|
||||
name: <app>
|
||||
namespace: <app>
|
||||
spec:
|
||||
entryPoints:
|
||||
- websecure
|
||||
routes:
|
||||
- match: Host(`<app>.oreillyit.nz`)
|
||||
kind: Rule
|
||||
services:
|
||||
- name: <app>
|
||||
port: 80
|
||||
```
|
||||
|
||||
If the app needs Authelia protection, add a middleware reference. If the app handles its own auth, omit it.
|
||||
|
||||
## Step 5: ArgoCD Application
|
||||
|
||||
Create `bootstrap/apps/<app>.yaml` in cluster-bootstrap:
|
||||
|
||||
```yaml
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: <app>
|
||||
namespace: argocd
|
||||
finalizers:
|
||||
- resources-finalizer.argocd.argoproj.io
|
||||
spec:
|
||||
project: default
|
||||
source:
|
||||
# In-repo:
|
||||
repoURL: http://gitea.bootstrap.homelab.internal/homelab/cluster-bootstrap.git
|
||||
targetRevision: main
|
||||
path: platform/<app>
|
||||
# Separate repo:
|
||||
# repoURL: http://gitea.bootstrap.homelab.internal/homelab/<app>.git
|
||||
# targetRevision: main
|
||||
# path: .
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
namespace: <app>
|
||||
syncPolicy:
|
||||
automated:
|
||||
prune: true
|
||||
selfHeal: true
|
||||
syncOptions:
|
||||
- ServerSideApply=true
|
||||
- CreateNamespace=true
|
||||
```
|
||||
|
||||
After pushing, the root app may not pick it up immediately. Force refresh:
|
||||
```bash
|
||||
kubectl annotate application root -n argocd argocd.argoproj.io/refresh=normal --overwrite
|
||||
```
|
||||
|
||||
## Step 6: Caddy Reverse Proxy (if externally accessible)
|
||||
|
||||
Add to `external/vps/inventory.yml` under `caddy_sites`:
|
||||
|
||||
```yaml
|
||||
- domain: "<app>.oreillyit.nz"
|
||||
type: public
|
||||
upstream: "https://10.111.1.100:443" # Traefik VIP
|
||||
upstream_tls_insecure: true # Traefik uses self-signed certs
|
||||
comment: "<App description>"
|
||||
```
|
||||
|
||||
Deploy:
|
||||
```bash
|
||||
ansible-playbook -i external/vps/inventory.yml external/vps/playbook.yml --tags compose
|
||||
```
|
||||
|
||||
Caddy automatically obtains a Let's Encrypt certificate on first request.
|
||||
|
||||
## Step 7: DNS
|
||||
|
||||
Add a Cloudflare A record for `<app>.oreillyit.nz` pointing to the VPS IP (`43.224.182.153`). Use orange cloud (proxied) for most services.
|
||||
|
||||
## Step 8: Verify
|
||||
|
||||
```bash
|
||||
# ArgoCD status
|
||||
kubectl get application <app> -n argocd
|
||||
|
||||
# Pod health
|
||||
kubectl get pods -n <app>
|
||||
|
||||
# PVC status (if applicable)
|
||||
kubectl get pvc -n <app>
|
||||
|
||||
# External access
|
||||
curl -sk -o /dev/null -w "%{http_code}" https://<app>.oreillyit.nz/
|
||||
```
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
| Pitfall | Symptom | Prevention |
|
||||
|---------|---------|------------|
|
||||
| Missing PodSecurity labels | `violates PodSecurity "baseline"` in StatefulSet events | Always check if any container needs privileged mode |
|
||||
| Container runs as non-root | Permission denied on writable dirs | Check image docs / `docker inspect` before writing manifests |
|
||||
| ArgoCD polling delay | New app doesn't appear after push | Annotate root app with `refresh=normal` |
|
||||
| StatefulSet CrashLoopBackOff | Updated spec not applied to pod | Delete the crashlooping pod manually |
|
||||
| Gitea repo permissions | `not authorized to write` on push | Grant collaborator access before pushing |
|
||||
| Proxmox CSI lock contention | PVC provisioning failures with lock timeout | Transient — CSI retries automatically |
|
||||
| Caddy not configured | Connection refused or SSL error on domain | Add site to inventory.yml and deploy with Ansible |
|
||||
Reference in New Issue
Block a user