From 25918bf028de4df3018811c9fd35a9c5977a6f26 Mon Sep 17 00:00:00 2001 From: Paul O'Reilly Date: Sat, 28 Mar 2026 17:23:04 +1300 Subject: [PATCH] Add database selection best practice: SQLite is not a production database MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Hard rule: any service with a FQDN, multiple consumers, or concurrent access MUST use PostgreSQL from day one. Documents the cost of "we'll migrate later" based on the Gitea SQLite→PostgreSQL migration that cost nearly a full day of productivity. Extracted from cluster-bootstrap gitea-scaling session (2026-03-28). Co-Authored-By: Claude Opus 4.6 (1M context) --- BESTPRACTICES.md | 2 + best-practices/database-selection.md | 100 +++++++++++++++++++++++++++ 2 files changed, 102 insertions(+) create mode 100644 best-practices/database-selection.md diff --git a/BESTPRACTICES.md b/BESTPRACTICES.md index 27ddaec..b267526 100644 --- a/BESTPRACTICES.md +++ b/BESTPRACTICES.md @@ -25,5 +25,7 @@ Generalised best practices extracted from real project work. Each topic file is - [Test-Driven Development](best-practices/test-driven-development.md) — Edge case discovery, property-based testing, mutation testing, AI agent testing patterns, test architecture - [Networking & Infrastructure](best-practices/networking.md) — nftables safety, systemd socket activation, Docker forwarding, TLS SNI vs Host header, wildcard certs - [Docker UID Matching](best-practices/docker-uid-matching.md) — UID wrapper entrypoint for mounted volumes, gosu pattern, when to use vs K8s securityContext +- [Database Selection](best-practices/database-selection.md) — SQLite is not a production database; always use PostgreSQL for services with FQDNs, multiple consumers, or concurrent access +- [Docker](best-practices/docker.md) — gosu PID 1, GIT_SSH_COMMAND scope, slim image health checks, buildx local images, default users, TTY flags, UID resolution - [Octopus Process Templates](best-practices/octopus-process-templates.md) — OCL syntax, step template references, channel scoping, parameters, versioning, Platform Hub patterns - [Platform Hub Knowledge Base](PLATFORMHUB.md) — Comprehensive guide: architecture, OCL syntax, template patterns, gotchas, API reference diff --git a/best-practices/database-selection.md b/best-practices/database-selection.md new file mode 100644 index 0000000..2e34d15 --- /dev/null +++ b/best-practices/database-selection.md @@ -0,0 +1,100 @@ +# Database Selection + +## The Rule: SQLite Is Not a Production Database + +**Any service that meets ANY of the following criteria MUST use PostgreSQL (or equivalent server-grade database) from day one:** + +- Attached to a FQDN (has a real domain name, even internal) +- Serves traffic from more than one process (API consumers, CI runners, webhooks, polling) +- Backs infrastructure that other systems depend on (Git hosting, container registries, auth providers) +- Will be accessed concurrently by automated systems (ArgoCD, CI runners, cron jobs) + +**Do not use SQLite for these workloads. Not temporarily. Not "to start with." Not "we'll migrate later."** + +SQLite uses file-level locking — only one writer at a time, and writes block reads. Under concurrent access, requests queue up waiting for the write lock, causing cascading timeouts. The failure mode is insidious: the service appears to work fine under light load but becomes intermittently unresponsive under real workloads. By the time you notice, everything that depends on it is also failing. + +## The Cost of "We'll Migrate Later" + +The Gitea SQLite→PostgreSQL migration (2026-03-28) cost nearly a full day of productivity: + +- **Hours of accumulated unresponsiveness** across multiple projects before root cause was identified +- **Planning and implementation** of the migration itself +- **Migration complexity** that didn't need to exist: Gitea 1.23 has no `restore` command, `doctor convert` only handles charset conversion, `docker cp` corrupted PostgreSQL directory permissions, SSH authorized_keys weren't regenerated +- **Downstream impact** on ArgoCD (20 apps polling a locked database), CI runners (continuous 500 errors), container registry pulls (timeouts) + +The PostgreSQL container takes 5 minutes to add to a Docker Compose stack at initial setup time. The migration took a day. Always pay the 5 minutes upfront. + +## When SQLite Is Acceptable + +SQLite is fine for: +- Local development databases (single developer, single process) +- Embedded application data stores (mobile apps, desktop apps, CLI tools) +- Read-heavy workloads with rare writes and a single writer process +- Test fixtures and throwaway data +- Configuration stores read at startup (not at request time) + +## Implementation Pattern + +For Docker Compose services that need a database: + +```yaml +services: + postgres: + image: postgres:17-alpine + restart: unless-stopped + environment: + POSTGRES_DB: myapp + POSTGRES_USER: myapp + POSTGRES_PASSWORD: {{ db_password }} + volumes: + - /opt/postgres-myapp:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U myapp -d myapp"] + interval: 10s + timeout: 5s + retries: 5 + start_period: 30s + networks: + - app_internal + deploy: + resources: + limits: + memory: 1G + + myapp: + depends_on: + postgres: + condition: service_healthy + networks: + - app_internal + - external_network + +networks: + app_internal: + driver: bridge + internal: true +``` + +Key points: +- PostgreSQL on an **internal bridge network** (no external access needed) +- Application **depends on PostgreSQL health** before starting +- **Resource limits** to prevent runaway memory usage +- **Separate data directory** per application (`/opt/postgres-myapp`, not shared) +- PostgreSQL container UID is **999** (not 1000) — set directory ownership accordingly + +## For Kubernetes Deployments + +Use the application's Helm chart PostgreSQL subchart, or deploy a standalone PostgreSQL instance: +- Bitnami PostgreSQL Helm chart for simple deployments +- CloudNativePG operator for production-grade PostgreSQL with HA, backups, and failover +- Never use SQLite with `emptyDir` or even PVC-backed volumes in multi-replica deployments + +## Checklist for New Service Deployment + +Before deploying any new service, check: + +1. What database does the default configuration use? +2. If SQLite: does the service support PostgreSQL? (Almost all do — Gitea, Authelia, Headscale, Zulip, etc.) +3. Switch to PostgreSQL **before the first deployment**, not after problems appear +4. Add the database password to SOPS-encrypted secrets +5. Verify the database connection works before adding consumers