claude-foundations/memory/runbook-new-argocd-app.md

# Runbook: Adding a New ArgoCD-Driven App to the Cluster

Step-by-step procedure for deploying a new application to the homelab Kubernetes cluster via ArgoCD GitOps. Derived from deploying Octopus Deploy (2026-03-13) and existing apps (Authelia, Homepage, Email Relay).

## Decision: In-repo vs Separate Repo

| Approach | When to use | Example |
|----------|-------------|---------|
| **In cluster-bootstrap** (`platform/<app>/`) | Tightly coupled to cluster lifecycle, simple apps | Authelia, Homepage, Email Relay |
| **Separate Gitea repo** | Independent lifecycle, external contributors, large/complex apps | Octopus Deploy |

Most apps go in cluster-bootstrap under `platform/`. Use a separate repo only when there's a clear reason.

## Step 1: Create Manifests

### Directory structure (in-repo)

```
platform/<app>/
├── kustomization.yaml
├── namespace.yaml
├── statefulset.yaml or deployment.yaml
├── service.yaml
├── ingressroute.yaml          # If externally accessible
├── ksops-generator.yaml       # If app has secrets
└── <name>-secret.sops.yaml    # SOPS-encrypted secrets
```

### Directory structure (separate repo)

```
<app>/
├── .sops.yaml                 # SOPS encryption rules (same age key as cluster-bootstrap)
├── .gitignore                 # local_secrets/
├── kustomization.yaml
├── namespace.yaml
├── <component>/               # Subdirectories per component
│   ├── statefulset.yaml
│   └── service.yaml
├── ksops-generator.yaml
├── *-secret.sops.yaml
├── local_secrets/             # Gitignored plaintext secrets for setup
└── scripts/
    └── setup.sh               # Secret generation + SOPS encryption
```

### Kustomization pattern

```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - <component>/statefulset.yaml
  - <component>/service.yaml
  - ingressroute.yaml
generators:
  - ksops-generator.yaml       # Only if secrets exist
```

### KSOPS generator pattern

```yaml
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: <app>-secret-generator
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops
files:
  - ./<name>-secret.sops.yaml
```

### SOPS config (separate repo only)

```yaml
creation_rules:
  - path_regex: .*secret.*\.yaml$
    encrypted_regex: "^(data|stringData)$"
    age: >-
      age1edc9agzzs8cngd2rsvfhm8aeucnlq2clmj36jh0rrkwuj073fyssr0u4x9
```

In-repo apps inherit from the cluster-bootstrap root `.sops.yaml`.

## Step 2: Namespace Checklist

```yaml
apiVersion: v1
kind: Namespace
metadata:
  name: <app>
  labels:
    # Only if app needs privileged containers (DinD, host networking, etc):
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/warn: privileged
```

**Always check:** Does any container need `securityContext.privileged: true` or host-level access? If yes, add PodSecurity labels. Forgetting this causes silent pod creation failures.

## Step 3: Secrets

1. Create plaintext secret YAML in `local_secrets/` (gitignored)
2. Encrypt with SOPS: `sops --encrypt local_secrets/<name>-secret.yaml > <name>-secret.sops.yaml`
3. Reference in `ksops-generator.yaml`
4. Secret filenames **must** contain `secret` (triggers SOPS rules)
5. Non-secret files must **not** contain `secret` in their name

## Step 4: Traefik IngressRoute (if externally accessible)

```yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: <app>
  namespace: <app>
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`<app>.oreillyit.nz`)
      kind: Rule
      services:
        - name: <app>
          port: 80
```

If the app needs Authelia protection, add a middleware reference. If the app handles its own auth, omit it.

## Step 5: ArgoCD Application

Create `bootstrap/apps/<app>.yaml` in cluster-bootstrap:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: <app>
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    # In-repo:
    repoURL: http://gitea.bootstrap.homelab.internal/homelab/cluster-bootstrap.git
    targetRevision: main
    path: platform/<app>
    # Separate repo:
    # repoURL: http://gitea.bootstrap.homelab.internal/homelab/<app>.git
    # targetRevision: main
    # path: .
  destination:
    server: https://kubernetes.default.svc
    namespace: <app>
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true
      - CreateNamespace=true
```

After pushing, the root app may not pick it up immediately. Force refresh:
```bash
kubectl annotate application root -n argocd argocd.argoproj.io/refresh=normal --overwrite
```

## Step 6: Caddy Reverse Proxy (if externally accessible)

Add to `external/vps/inventory.yml` under `caddy_sites`:

```yaml
- domain: "<app>.oreillyit.nz"
  type: public
  upstream: "https://10.111.1.100:443"   # Traefik VIP
  upstream_tls_insecure: true             # Traefik uses self-signed certs
  comment: "<App description>"
```

Deploy:
```bash
ansible-playbook -i external/vps/inventory.yml external/vps/playbook.yml --tags compose
```

Caddy automatically obtains a Let's Encrypt certificate on first request.

## Step 7: DNS

Add a Cloudflare A record for `<app>.oreillyit.nz` pointing to the VPS IP (`43.224.182.153`). Use orange cloud (proxied) for most services.

## Step 8: Verify

```bash
# ArgoCD status
kubectl get application <app> -n argocd

# Pod health
kubectl get pods -n <app>

# PVC status (if applicable)
kubectl get pvc -n <app>

# External access
curl -sk -o /dev/null -w "%{http_code}" https://<app>.oreillyit.nz/
```

## Common Pitfalls

| Pitfall | Symptom | Prevention |
|---------|---------|------------|
| Missing PodSecurity labels | `violates PodSecurity "baseline"` in StatefulSet events | Always check if any container needs privileged mode |
| Container runs as non-root | Permission denied on writable dirs | Check image docs / `docker inspect` before writing manifests |
| ArgoCD polling delay | New app doesn't appear after push | Annotate root app with `refresh=normal` |
| StatefulSet CrashLoopBackOff | Updated spec not applied to pod | Delete the crashlooping pod manually |
| Gitea repo permissions | `not authorized to write` on push | Grant collaborator access before pushing |
| Proxmox CSI lock contention | PVC provisioning failures with lock timeout | Transient — CSI retries automatically |
| Caddy not configured | Connection refused or SSL error on domain | Add site to inventory.yml and deploy with Ansible |