best-practices/docker.md

# Docker Best Practices

## Use gosu for Entrypoint Privilege Dropping

`su -c "command"` and `sudo -u agent command` create child processes. The real command is not PID 1, so Docker signals (SIGTERM on stop) don't reach it. Use `gosu agent command` which execs directly — the command becomes PID 1 with proper signal handling.

## GIT_SSH_COMMAND Only Affects Git-Invoked SSH

`GIT_SSH_COMMAND` only applies when git invokes SSH (clone, push, fetch). Direct `ssh` calls need explicit flags: `ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null`. Don't assume setting `GIT_SSH_COMMAND` fixes all SSH operations in a container.

## Use Python urllib for Health Checks in Slim Images

Service images based on `python:3.12-slim` don't include curl. For in-container health checks, use `python3 -c "import urllib.request; urllib.request.urlopen('http://...')"`. This applies to verification scripts using `kubectl exec` and to Kubernetes liveness/readiness probes that exec into containers.

## Buildx Docker-Container Driver Can't See Local Images

When using buildx with the docker-container driver, `FROM local-image:latest` tries Docker Hub because the builder runs in a separate container that can't see locally-loaded images. Always use the full registry path in Dockerfiles. In CI, split into sequential jobs so base images are pushed to the registry before dependent images build.

## Delete Conflicting Default Users at Build Time

Ubuntu 24.04 base images ship with a `ubuntu` user at UID 1000 — the most common host UID. This causes `usermod -u 1000` conflicts and can trigger non-deterministic hangs (e.g., `newgrp ubuntu` waiting for a password on stdin). Delete the default user in the Dockerfile: `RUN userdel -r ubuntu`.

## Service Images Should Use Minimal Base Images

Service images (API servers, background workers) should use `python:3.12-slim` or equivalent, not the agent base image. Agent base images include CLIs, Node.js, and other tooling that bloats service images unnecessarily. Keep agent tooling in agent images only.

## Platform-Specific Native Binaries

Never mount host `node_modules` into a Docker container when the build uses platform-specific native binaries (e.g., Tailwind CSS, esbuild, SWC). Always run `npm install` inside the same container that runs the build. The native binary is compiled for the platform where `npm install` runs — host and container may differ in libc, architecture, or OS.

**Symptom:** `Cannot find native binding` or `Cannot find module '@tailwindcss/oxide-linux-x64-gnu'`
**Fix:** Run `npm install` inside the container, not on the host.

## Docker Wrapper Scripts and TTY Flags

Docker wrapper scripts (e.g., `~/sbin/hugo` calling `docker run -it ...`) fail with `the input device is not a TTY` in non-interactive contexts (CI pipelines, Claude Code, cron jobs, scripts).

**Fix:** Only pass `-t` when stdin is a terminal: `[ -t 0 ] && TTY_FLAG="-t" || TTY_FLAG=""`. Or omit `-t` entirely and let callers add it when needed.

## Three-Tier UID Resolution

The UID wrapper should resolve the target UID/GID using this priority:
1. **Environment variables** (`AGENT_UID`/`AGENT_GID`) — injected by the orchestrator/dispatcher. Preferred because it's explicit and deployment-specific.
2. **stat the mount point** — detect the UID/GID of the mounted directory. Works when no env vars are set.
3. **Skip** — if not root or no mount point exists, run as the default container user.

This makes UID matching a deployment concern (varies per host), not a configuration concern (baked into images). See [Docker UID Matching](docker-uid-matching.md) for the full UID wrapper pattern.

## network_mode: service:* Breaks on Parent Container Restart

When a container shares another's network namespace via `network_mode: "service:<parent>"`, restarting the parent recreates the namespace. The dependent container keeps stale socket bindings — its listeners are bound to a namespace that no longer exists. TCP connections fail while the dependent container appears healthy. Fix: add a healthcheck to the parent and use `depends_on: condition: service_healthy` on the dependent. Alternatively, use an internal bridge network instead of namespace sharing.

## docker compose restart Is Concurrent, Not Ordered

`docker compose restart svc1 svc2 svc3` restarts all named services concurrently, ignoring `depends_on` ordering. Dependent services may start before their dependencies are ready. Use `docker compose up -d` (which respects `depends_on`) or restart in explicit stages: stop dependents, restart the dependency, wait for healthy, then start dependents.

## docker cp Can Corrupt Container Filesystem Ownership

`docker cp` runs as root and can change ownership of parent directories in the container's filesystem layer. This is particularly dangerous for database containers (e.g., PostgreSQL UID 999) — copying a file into `/tmp/` can corrupt the data directory ownership, causing "Permission denied" errors. After any `docker cp` into a stateful container, verify and fix ownership: `chown -R <expected-uid>:<expected-gid> <data-dir>`.

## Prefer Internal Bridge Networks Over Namespace Sharing

For sidecar-style containers that need to communicate (e.g., app + database, app + TLS proxy), prefer an internal Docker bridge network over `network_mode: "service:<parent>"`. Bridge networks allow proper `depends_on` ordering with health checks, independent restart of each container, and clear network isolation. Namespace sharing couples container lifecycles — restarting the parent invalidates the dependent's network stack.

## Verify Dockerfile COPY After Creating New Files

After creating a file intended for a Docker image (scripts, configs, wrappers), immediately add the COPY line to the Dockerfile and verify with `docker run --entrypoint sh <image> -c "ls /path/to/file"`. Files placed in image source directories are not automatically included — they need explicit COPY instructions. This class of bug can remain latent until the code path is first exercised.

## docker-compose.override.yaml Merges Lists Additively

Docker Compose V2 merges list fields (ports, volumes, environment) by appending, not replacing. An override file with a different port mapping adds a second binding rather than replacing the original, causing conflicts. Modify the base `docker-compose.yaml` directly or use `!override` for list replacement.

## Volume Source Paths Must Be Absolute

Docker interprets relative paths in volume mount source fields as named volumes, not bind mounts. Use `os.path.abspath()` or equivalent when constructing volume source paths programmatically. The error message ("includes invalid characters for a local volume name") is misleading — the real issue is that the path is relative.

## Init Scripts Needing Root Must Run Before gosu/exec Privilege Drop

In Docker entrypoints that drop privileges via `gosu <user> "$@"` or `exec gosu <user> command`, any setup that requires root (creating directories, setting ownership, writing to system paths) must happen before the `gosu` call. Once `exec gosu` runs, the process is replaced with the unprivileged user — subsequent commands in the same shell context run as that user. Structure entrypoints as: (1) root-level setup, (2) `exec gosu <user> "$@"`.

## Payload Size Limits at Multiple Layers

Large payloads embedded in environment variables or command-line arguments hit hard limits at multiple layers:
- **OS `ARG_MAX`** (~128KB on Linux): the kernel limit on aggregate environment size. Exceeded values cause `Argument list too long` errors.
- **K8s env var limit** (~228KB base64 per variable): containers crash with exit 255 and zero logs.
- **CLI argv**: even when payload delivery fits via env var or file mount, many CLIs (`claude --print <prompt>`, shell wrappers) pass the prompt on argv, which hits `ARG_MAX` independently. When a prompt exceeds ~64KB, write it to a temp file and pipe via stdin instead of passing it positionally.

Use mounted files (ConfigMaps, Secrets, host bind mounts) for any payload that approaches these limits. For inter-task artifact passing, use git branches or mounted volumes — not env var payloads. See also the [Kubernetes Patterns](kubernetes.md) entry on env var size limits.

**Meta-rule:** after fixing an exec-arg limit at one layer (env, argv, file mount), immediately check adjacent layers for the same pattern before declaring it done. The same payload often flows through multiple chokepoints.

## `rm -f` on Bind-Mounted Files Fails Under `set -e`

Attempting `rm -f /path/to/bind-mounted-file` when the file is a bind mount (e.g., a host file mounted read-only into a container) fails with "Device or resource busy" even with the `-f` flag. Under `set -e` this exits the script immediately. Use `rm -f path 2>/dev/null || true` to suppress the error and continue, or check whether the path is a bind mount before attempting deletion.

## Compose `profiles:` Blocks On-Demand Lifecycle Managers

Services gated by `profiles:` in `docker-compose.yml` are not created until the profile is activated. On-demand lifecycle managers — Sablier, autoheal-style wake-up tools, CI runners that start/stop existing containers on request — cannot manage what does not exist. The container has no Docker record for them to act on.

**Fix:** drop `profiles:` for services managed by an external lifecycle tool, and use `docker compose create` (not `up`) to materialise the containers in a stopped state. The lifecycle manager can then start them on demand.

## GPU-Agnostic Base Compose with Provider Override Files

Keep the base `docker-compose.yml` free of hardware-specific runtime config. Put NVIDIA/AMD/Apple GPU or accelerator runtime settings in separate override files composed in with `-f`:

```
docker-compose.yml              # GPU-agnostic base
docker-compose.nvidia.yml       # NVIDIA runtime, device reservations
docker-compose.apple.yml        # Apple Silicon / Metal settings
docker-compose.amd.yml          # ROCm overrides
```

Deploy with `docker compose -f docker-compose.yml -f docker-compose.nvidia.yml up -d`. The same stack template deploys unchanged across heterogeneous hosts; adding a new accelerator type is an override file, not a fork of the base compose.

## Snap-Packaged Docker Breaks After Unclean Shutdown

**Symptom:** containers fail with "read-only file system" on `/opt/` or other snap-confined paths after a power cut or forced reboot, even though the disk is healthy.

**Cause:** snap auto-refresh or AppArmor profile corruption during unclean shutdown leaves Docker's snap confinement in an inconsistent state.

**Fix:** `snap restart docker` is usually enough; if not, `snap revert docker`. On production hosts, prefer distro-packaged or upstream Docker (`docker-ce` from Docker's apt repo) to avoid snap confinement entirely.

## Set PYTHONPATH in Dockerfiles for `src/` Layout Projects

Python projects using a `src/` layout must set `ENV PYTHONPATH=/app/src` in every Dockerfile that does `COPY src/ ./src/`. Local development hides the problem because `pip install -e .` wires up imports via the editable install — the container has no such install, so imports fail only at runtime inside the container.

Set `PYTHONPATH` when scaffolding the Dockerfile, not after the first failed container run. Same rule applies to any multi-stage build where the final stage copies `src/` without re-running `pip install`.

## Write CLI Entrypoints Alongside the Module They Run

When a module will run inside a container, create its `_cli.py` entry point and verify the Dockerfile `ENTRYPOINT` in the same phase/commit as the module itself. Deferring CLI shims to "a later phase" repeatedly leads to containers that build cleanly but have no runnable entry point — the build passes, the deployment looks healthy, and the first invocation fails with `No module named ...` or an `ENTRYPOINT` that points at nothing.

Treat "module + CLI shim + ENTRYPOINT verification" as one atomic unit of work.

## PostgreSQL Alpine Image Runs as UID 70, Not 999

The `postgres:*-alpine` images run postgres as **UID 70**, not the 999 used by Debian-based `postgres:*` images. Setting data-directory ownership to 999 (or 1000) via host `chown` or Ansible `file` modules silently breaks access — `pg_isready` may still pass while internal operations fail with "Permission denied" on WAL writes, replication slots, or extension installs.

**Always verify before setting ownership:**
```
docker exec --user postgres <container> id
```

Match the automation to the actual UID. If a project mixes Alpine and Debian Postgres images across environments, treat the UID as per-environment config, not a hardcoded constant.