best-practices/database-selection.md

Database Selection

The Rule: SQLite Is Not a Production Database

Any service that meets ANY of the following criteria MUST use PostgreSQL (or equivalent server-grade database) from day one:

• Attached to a FQDN (has a real domain name, even internal)
• Serves traffic from more than one process (API consumers, CI runners, webhooks, polling)
• Backs infrastructure that other systems depend on (Git hosting, container registries, auth providers)
• Will be accessed concurrently by automated systems (ArgoCD, CI runners, cron jobs)

Do not use SQLite for these workloads. Not temporarily. Not "to start with." Not "we'll migrate later."

SQLite uses file-level locking — only one writer at a time, and writes block reads. Under concurrent access, requests queue up waiting for the write lock, causing cascading timeouts. The failure mode is insidious: the service appears to work fine under light load but becomes intermittently unresponsive under real workloads. By the time you notice, everything that depends on it is also failing.

The Cost of "We'll Migrate Later"

The Gitea SQLite→PostgreSQL migration (2026-03-28) cost nearly a full day of productivity:

• Hours of accumulated unresponsiveness across multiple projects before root cause was identified
• Planning and implementation of the migration itself
• Migration complexity that didn't need to exist: Gitea 1.23 has no restore command, doctor convert only handles charset conversion, docker cp corrupted PostgreSQL directory permissions, SSH authorized_keys weren't regenerated
• Downstream impact on ArgoCD (20 apps polling a locked database), CI runners (continuous 500 errors), container registry pulls (timeouts)

The PostgreSQL container takes 5 minutes to add to a Docker Compose stack at initial setup time. The migration took a day. Always pay the 5 minutes upfront.

When SQLite Is Acceptable

SQLite is fine for:

• Local development databases (single developer, single process)
• Embedded application data stores (mobile apps, desktop apps, CLI tools)
• Read-heavy workloads with rare writes and a single writer process
• Test fixtures and throwaway data
• Configuration stores read at startup (not at request time)

Implementation Pattern

For Docker Compose services that need a database:

services:
  postgres:
    image: postgres:17-alpine
    restart: unless-stopped
    environment:
      POSTGRES_DB: myapp
      POSTGRES_USER: myapp
      POSTGRES_PASSWORD: {{ db_password }}
    volumes:
      - /opt/postgres-myapp:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U myapp -d myapp"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 30s
    networks:
      - app_internal
    deploy:
      resources:
        limits:
          memory: 1G

  myapp:
    depends_on:
      postgres:
        condition: service_healthy
    networks:
      - app_internal
      - external_network

networks:
  app_internal:
    driver: bridge
    internal: true

Key points:

• PostgreSQL on an internal bridge network (no external access needed)
• Application depends on PostgreSQL health before starting
• Resource limits to prevent runaway memory usage
• Separate data directory per application (/opt/postgres-myapp, not shared)
• PostgreSQL container UID is 999 (not 1000) — set directory ownership accordingly

For Kubernetes Deployments

Use the application's Helm chart PostgreSQL subchart, or deploy a standalone PostgreSQL instance:

• Bitnami PostgreSQL Helm chart for simple deployments
• CloudNativePG operator for production-grade PostgreSQL with HA, backups, and failover
• Never use SQLite with emptyDir or even PVC-backed volumes in multi-replica deployments

Checklist for New Service Deployment

Before deploying any new service, check:

1. What database does the default configuration use?
2. If SQLite: does the service support PostgreSQL? (Almost all do — Gitea, Authelia, Headscale, Zulip, etc.)
3. Switch to PostgreSQL before the first deployment, not after problems appear
4. Add the database password to SOPS-encrypted secrets
5. Verify the database connection works before adding consumers

ON CONFLICT DO NOTHING Requires a Real Unique Constraint

SQLAlchemy on_conflict_do_nothing() and raw ON CONFLICT DO NOTHING only work when there is a matching unique constraint or unique index. Without one, the statement either silently does nothing or inserts a duplicate, depending on the exact phrasing.

Trap: "URL" looks like a natural unique key for a crawl/ingest table, so the LLM or developer adds UNIQUE(url) to make the upsert work. But URL content changes over time — two rows for the same URL at different timestamps are semantically distinct. The fake unique constraint then corrupts the model (or blocks legitimate re-ingestion).

Rule:

• If the natural key is truly unique (user ID, slug, message hash), add the unique constraint and use on_conflict_do_nothing()
• If the natural key is not unique over time (URL, title, filename), use query-before-insert in a transaction, not a fake unique constraint
• Don't invent unique constraints to make ON CONFLICT work — you're encoding a false invariant into the schema

Alembic Multi-Schema Migrations

To run Alembic against multiple Postgres schemas in one database:

1. Set include_schemas=True in env.py so autogenerate sees non-default schemas
2. Add an include_name filter so autogenerate only tracks the schemas you manage (otherwise it tries to "fix" information_schema, pg_catalog, etc.)
3. Issue CREATE SCHEMA IF NOT EXISTS <name> before run_migrations() — otherwise the first migration fails on a missing schema

def include_name(name, type_, parent_names):
    if type_ == "schema":
        return name in {"app", "audit", "reporting"}
    return True

def run_migrations_online():
    connectable = engine_from_config(...)
    with connectable.connect() as connection:
        for schema in ("app", "audit", "reporting"):
            connection.execute(text(f'CREATE SCHEMA IF NOT EXISTS "{schema}"'))
        context.configure(
            connection=connection,
            include_schemas=True,
            include_name=include_name,
            ...
        )
        with context.begin_transaction():
            context.run_migrations()

Also set version_table_schema on context.configure if the Alembic version table should live in a specific schema rather than public.