claude-foundations/memory/runbook-new-argocd-app.md

Runbook: Adding a New ArgoCD-Driven App to the Cluster

Step-by-step procedure for deploying a new application to the homelab Kubernetes cluster via ArgoCD GitOps. Derived from deploying Octopus Deploy (2026-03-13) and existing apps (Authelia, Homepage, Email Relay).

Decision: In-repo vs Separate Repo

Approach	When to use	Example
In cluster-bootstrap (platform/<app>/)	Tightly coupled to cluster lifecycle, simple apps	Authelia, Homepage, Email Relay
Separate Gitea repo	Independent lifecycle, external contributors, large/complex apps	Octopus Deploy

Most apps go in cluster-bootstrap under platform/. Use a separate repo only when there's a clear reason.

Step 1: Create Manifests

Directory structure (in-repo)

platform/<app>/
├── kustomization.yaml
├── namespace.yaml
├── statefulset.yaml or deployment.yaml
├── service.yaml
├── ingressroute.yaml          # If externally accessible
├── ksops-generator.yaml       # If app has secrets
└── <name>-secret.sops.yaml    # SOPS-encrypted secrets

Directory structure (separate repo)

<app>/
├── .sops.yaml                 # SOPS encryption rules (same age key as cluster-bootstrap)
├── .gitignore                 # local_secrets/
├── kustomization.yaml
├── namespace.yaml
├── <component>/               # Subdirectories per component
│   ├── statefulset.yaml
│   └── service.yaml
├── ksops-generator.yaml
├── *-secret.sops.yaml
├── local_secrets/             # Gitignored plaintext secrets for setup
└── scripts/
    └── setup.sh               # Secret generation + SOPS encryption

Kustomization pattern

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - <component>/statefulset.yaml
  - <component>/service.yaml
  - ingressroute.yaml
generators:
  - ksops-generator.yaml       # Only if secrets exist

KSOPS generator pattern

apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: <app>-secret-generator
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops
files:
  - ./<name>-secret.sops.yaml

SOPS config (separate repo only)

creation_rules:
  - path_regex: .*secret.*\.yaml$
    encrypted_regex: "^(data|stringData)$"
    age: >-
      age1edc9agzzs8cngd2rsvfhm8aeucnlq2clmj36jh0rrkwuj073fyssr0u4x9

In-repo apps inherit from the cluster-bootstrap root .sops.yaml.

Step 2: Namespace Checklist

apiVersion: v1
kind: Namespace
metadata:
  name: <app>
  labels:
    # Only if app needs privileged containers (DinD, host networking, etc):
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/warn: privileged

Always check: Does any container need securityContext.privileged: true or host-level access? If yes, add PodSecurity labels. Forgetting this causes silent pod creation failures.

Step 3: Secrets

1. Create plaintext secret YAML in local_secrets/ (gitignored)
2. Encrypt with SOPS: sops --encrypt local_secrets/<name>-secret.yaml > <name>-secret.sops.yaml
3. Reference in ksops-generator.yaml
4. Secret filenames must contain secret (triggers SOPS rules)
5. Non-secret files must not contain secret in their name

Step 4: Traefik IngressRoute (if externally accessible)

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: <app>
  namespace: <app>
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`<app>.oreillyit.nz`)
      kind: Rule
      services:
        - name: <app>
          port: 80

If the app needs Authelia protection, add a middleware reference. If the app handles its own auth, omit it.

Step 5: ArgoCD Application

Create bootstrap/apps/<app>.yaml in cluster-bootstrap:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: <app>
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    # In-repo:
    repoURL: http://gitea.bootstrap.homelab.internal/homelab/cluster-bootstrap.git
    targetRevision: main
    path: platform/<app>
    # Separate repo:
    # repoURL: http://gitea.bootstrap.homelab.internal/homelab/<app>.git
    # targetRevision: main
    # path: .
  destination:
    server: https://kubernetes.default.svc
    namespace: <app>
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true
      - CreateNamespace=true

After pushing, the root app may not pick it up immediately. Force refresh:

kubectl annotate application root -n argocd argocd.argoproj.io/refresh=normal --overwrite

Step 6: Caddy Reverse Proxy (if externally accessible)

Add to external/vps/inventory.yml under caddy_sites:

- domain: "<app>.oreillyit.nz"
  type: public
  upstream: "https://10.111.1.100:443"   # Traefik VIP
  upstream_tls_insecure: true             # Traefik uses self-signed certs
  comment: "<App description>"

Deploy:

ansible-playbook -i external/vps/inventory.yml external/vps/playbook.yml --tags compose

Caddy automatically obtains a Let's Encrypt certificate on first request.

Step 7: DNS

Add a Cloudflare A record for <app>.oreillyit.nz pointing to the VPS IP (43.224.182.153). Use orange cloud (proxied) for most services.

Step 8: Verify

# ArgoCD status
kubectl get application <app> -n argocd

# Pod health
kubectl get pods -n <app>

# PVC status (if applicable)
kubectl get pvc -n <app>

# External access
curl -sk -o /dev/null -w "%{http_code}" https://<app>.oreillyit.nz/

Common Pitfalls

Pitfall	Symptom	Prevention
Missing PodSecurity labels	violates PodSecurity "baseline" in StatefulSet events	Always check if any container needs privileged mode
Container runs as non-root	Permission denied on writable dirs	Check image docs / docker inspect before writing manifests
ArgoCD polling delay	New app doesn't appear after push	Annotate root app with refresh=normal
StatefulSet CrashLoopBackOff	Updated spec not applied to pod	Delete the crashlooping pod manually
Gitea repo permissions	not authorized to write on push	Grant collaborator access before pushing
Proxmox CSI lock contention	PVC provisioning failures with lock timeout	Transient — CSI retries automatically
Caddy not configured	Connection refused or SSL error on domain	Add site to inventory.yml and deploy with Ansible