Add database selection best practice: SQLite is not a production database

Hard rule: any service with a FQDN, multiple consumers, or concurrent
access MUST use PostgreSQL from day one. Documents the cost of "we'll
migrate later" based on the Gitea SQLite→PostgreSQL migration that cost
nearly a full day of productivity.

Extracted from cluster-bootstrap gitea-scaling session (2026-03-28).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

best-practices/database-selection.md

@@ -0,0 +1,100 @@
+# Database Selection
+
+## The Rule: SQLite Is Not a Production Database
+
+**Any service that meets ANY of the following criteria MUST use PostgreSQL (or equivalent server-grade database) from day one:**
+
+- Attached to a FQDN (has a real domain name, even internal)
+- Serves traffic from more than one process (API consumers, CI runners, webhooks, polling)
+- Backs infrastructure that other systems depend on (Git hosting, container registries, auth providers)
+- Will be accessed concurrently by automated systems (ArgoCD, CI runners, cron jobs)
+
+**Do not use SQLite for these workloads. Not temporarily. Not "to start with." Not "we'll migrate later."**
+
+SQLite uses file-level locking — only one writer at a time, and writes block reads. Under concurrent access, requests queue up waiting for the write lock, causing cascading timeouts. The failure mode is insidious: the service appears to work fine under light load but becomes intermittently unresponsive under real workloads. By the time you notice, everything that depends on it is also failing.
+
+## The Cost of "We'll Migrate Later"
+
+The Gitea SQLite→PostgreSQL migration (2026-03-28) cost nearly a full day of productivity:
+
+- **Hours of accumulated unresponsiveness** across multiple projects before root cause was identified
+- **Planning and implementation** of the migration itself
+- **Migration complexity** that didn't need to exist: Gitea 1.23 has no `restore` command, `doctor convert` only handles charset conversion, `docker cp` corrupted PostgreSQL directory permissions, SSH authorized_keys weren't regenerated
+- **Downstream impact** on ArgoCD (20 apps polling a locked database), CI runners (continuous 500 errors), container registry pulls (timeouts)
+
+The PostgreSQL container takes 5 minutes to add to a Docker Compose stack at initial setup time. The migration took a day. Always pay the 5 minutes upfront.
+
+## When SQLite Is Acceptable
+
+SQLite is fine for:
+- Local development databases (single developer, single process)
+- Embedded application data stores (mobile apps, desktop apps, CLI tools)
+- Read-heavy workloads with rare writes and a single writer process
+- Test fixtures and throwaway data
+- Configuration stores read at startup (not at request time)
+
+## Implementation Pattern
+
+For Docker Compose services that need a database:
+
+```yaml
+services:
+  postgres:
+    image: postgres:17-alpine
+    restart: unless-stopped
+    environment:
+      POSTGRES_DB: myapp
+      POSTGRES_USER: myapp
+      POSTGRES_PASSWORD: {{ db_password }}
+    volumes:
+      - /opt/postgres-myapp:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U myapp -d myapp"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+      start_period: 30s
+    networks:
+      - app_internal
+    deploy:
+      resources:
+        limits:
+          memory: 1G
+
+  myapp:
+    depends_on:
+      postgres:
+        condition: service_healthy
+    networks:
+      - app_internal
+      - external_network
+
+networks:
+  app_internal:
+    driver: bridge
+    internal: true
+```
+
+Key points:
+- PostgreSQL on an **internal bridge network** (no external access needed)
+- Application **depends on PostgreSQL health** before starting
+- **Resource limits** to prevent runaway memory usage
+- **Separate data directory** per application (`/opt/postgres-myapp`, not shared)
+- PostgreSQL container UID is **999** (not 1000) — set directory ownership accordingly
+
+## For Kubernetes Deployments
+
+Use the application's Helm chart PostgreSQL subchart, or deploy a standalone PostgreSQL instance:
+- Bitnami PostgreSQL Helm chart for simple deployments
+- CloudNativePG operator for production-grade PostgreSQL with HA, backups, and failover
+- Never use SQLite with `emptyDir` or even PVC-backed volumes in multi-replica deployments
+
+## Checklist for New Service Deployment
+
+Before deploying any new service, check:
+
+1. What database does the default configuration use?
+2. If SQLite: does the service support PostgreSQL? (Almost all do — Gitea, Authelia, Headscale, Zulip, etc.)
+3. Switch to PostgreSQL **before the first deployment**, not after problems appear
+4. Add the database password to SOPS-encrypted secrets
+5. Verify the database connection works before adding consumers