best-practices/docker.md

Docker Best Practices

Use gosu for Entrypoint Privilege Dropping

su -c "command" and sudo -u agent command create child processes. The real command is not PID 1, so Docker signals (SIGTERM on stop) don't reach it. Use gosu agent command which execs directly — the command becomes PID 1 with proper signal handling.

GIT_SSH_COMMAND Only Affects Git-Invoked SSH

GIT_SSH_COMMAND only applies when git invokes SSH (clone, push, fetch). Direct ssh calls need explicit flags: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null. Don't assume setting GIT_SSH_COMMAND fixes all SSH operations in a container.

Use Python urllib for Health Checks in Slim Images

Service images based on python:3.12-slim don't include curl. For in-container health checks, use python3 -c "import urllib.request; urllib.request.urlopen('http://...')". This applies to verification scripts using kubectl exec and to Kubernetes liveness/readiness probes that exec into containers.

Buildx Docker-Container Driver Can't See Local Images

When using buildx with the docker-container driver, FROM local-image:latest tries Docker Hub because the builder runs in a separate container that can't see locally-loaded images. Always use the full registry path in Dockerfiles. In CI, split into sequential jobs so base images are pushed to the registry before dependent images build.

Delete Conflicting Default Users at Build Time

Ubuntu 24.04 base images ship with a ubuntu user at UID 1000 — the most common host UID. This causes usermod -u 1000 conflicts and can trigger non-deterministic hangs (e.g., newgrp ubuntu waiting for a password on stdin). Delete the default user in the Dockerfile: RUN userdel -r ubuntu.

Service Images Should Use Minimal Base Images

Service images (API servers, background workers) should use python:3.12-slim or equivalent, not the agent base image. Agent base images include CLIs, Node.js, and other tooling that bloats service images unnecessarily. Keep agent tooling in agent images only.

Platform-Specific Native Binaries

Never mount host node_modules into a Docker container when the build uses platform-specific native binaries (e.g., Tailwind CSS, esbuild, SWC). Always run npm install inside the same container that runs the build. The native binary is compiled for the platform where npm install runs — host and container may differ in libc, architecture, or OS.

Symptom: Cannot find native binding or Cannot find module '@tailwindcss/oxide-linux-x64-gnu' Fix: Run npm install inside the container, not on the host.

Docker Wrapper Scripts and TTY Flags

Docker wrapper scripts (e.g., ~/sbin/hugo calling docker run -it ...) fail with the input device is not a TTY in non-interactive contexts (CI pipelines, Claude Code, cron jobs, scripts).

Fix: Only pass -t when stdin is a terminal: [ -t 0 ] && TTY_FLAG="-t" || TTY_FLAG="". Or omit -t entirely and let callers add it when needed.

Three-Tier UID Resolution

The UID wrapper should resolve the target UID/GID using this priority:

1. Environment variables (AGENT_UID/AGENT_GID) — injected by the orchestrator/dispatcher. Preferred because it's explicit and deployment-specific.
2. stat the mount point — detect the UID/GID of the mounted directory. Works when no env vars are set.
3. Skip — if not root or no mount point exists, run as the default container user.

This makes UID matching a deployment concern (varies per host), not a configuration concern (baked into images). See Docker UID Matching for the full UID wrapper pattern.

network_mode: service:* Breaks on Parent Container Restart

When a container shares another's network namespace via network_mode: "service:<parent>", restarting the parent recreates the namespace. The dependent container keeps stale socket bindings — its listeners are bound to a namespace that no longer exists. TCP connections fail while the dependent container appears healthy. Fix: add a healthcheck to the parent and use depends_on: condition: service_healthy on the dependent. Alternatively, use an internal bridge network instead of namespace sharing.

docker compose restart Is Concurrent, Not Ordered

docker compose restart svc1 svc2 svc3 restarts all named services concurrently, ignoring depends_on ordering. Dependent services may start before their dependencies are ready. Use docker compose up -d (which respects depends_on) or restart in explicit stages: stop dependents, restart the dependency, wait for healthy, then start dependents.

docker cp Can Corrupt Container Filesystem Ownership

docker cp runs as root and can change ownership of parent directories in the container's filesystem layer. This is particularly dangerous for database containers (e.g., PostgreSQL UID 999) — copying a file into /tmp/ can corrupt the data directory ownership, causing "Permission denied" errors. After any docker cp into a stateful container, verify and fix ownership: chown -R <expected-uid>:<expected-gid> <data-dir>.

Prefer Internal Bridge Networks Over Namespace Sharing

For sidecar-style containers that need to communicate (e.g., app + database, app + TLS proxy), prefer an internal Docker bridge network over network_mode: "service:<parent>". Bridge networks allow proper depends_on ordering with health checks, independent restart of each container, and clear network isolation. Namespace sharing couples container lifecycles — restarting the parent invalidates the dependent's network stack.

Verify Dockerfile COPY After Creating New Files

After creating a file intended for a Docker image (scripts, configs, wrappers), immediately add the COPY line to the Dockerfile and verify with docker run --entrypoint sh <image> -c "ls /path/to/file". Files placed in image source directories are not automatically included — they need explicit COPY instructions. This class of bug can remain latent until the code path is first exercised.

docker-compose.override.yaml Merges Lists Additively

Docker Compose V2 merges list fields (ports, volumes, environment) by appending, not replacing. An override file with a different port mapping adds a second binding rather than replacing the original, causing conflicts. Modify the base docker-compose.yaml directly or use !override for list replacement.

Volume Source Paths Must Be Absolute

Docker interprets relative paths in volume mount source fields as named volumes, not bind mounts. Use os.path.abspath() or equivalent when constructing volume source paths programmatically. The error message ("includes invalid characters for a local volume name") is misleading — the real issue is that the path is relative.

Init Scripts Needing Root Must Run Before gosu/exec Privilege Drop

In Docker entrypoints that drop privileges via gosu <user> "$@" or exec gosu <user> command, any setup that requires root (creating directories, setting ownership, writing to system paths) must happen before the gosu call. Once exec gosu runs, the process is replaced with the unprivileged user — subsequent commands in the same shell context run as that user. Structure entrypoints as: (1) root-level setup, (2) exec gosu <user> "$@".

Payload Size Limits at Multiple Layers

Large payloads embedded in environment variables or command-line arguments hit hard limits at multiple layers:

• OS ARG_MAX (~128KB on Linux): the kernel limit on aggregate environment size. Exceeded values cause Argument list too long errors.
• K8s env var limit (~228KB base64 per variable): containers crash with exit 255 and zero logs.
• CLI argv: even when payload delivery fits via env var or file mount, many CLIs (claude --print <prompt>, shell wrappers) pass the prompt on argv, which hits ARG_MAX independently. When a prompt exceeds ~64KB, write it to a temp file and pipe via stdin instead of passing it positionally.

Use mounted files (ConfigMaps, Secrets, host bind mounts) for any payload that approaches these limits. For inter-task artifact passing, use git branches or mounted volumes — not env var payloads. See also the Kubernetes Patterns entry on env var size limits.

Meta-rule: after fixing an exec-arg limit at one layer (env, argv, file mount), immediately check adjacent layers for the same pattern before declaring it done. The same payload often flows through multiple chokepoints.

rm -f on Bind-Mounted Files Fails Under set -e

Attempting rm -f /path/to/bind-mounted-file when the file is a bind mount (e.g., a host file mounted read-only into a container) fails with "Device or resource busy" even with the -f flag. Under set -e this exits the script immediately. Use rm -f path 2>/dev/null || true to suppress the error and continue, or check whether the path is a bind mount before attempting deletion.

Compose profiles: Blocks On-Demand Lifecycle Managers

Services gated by profiles: in docker-compose.yml are not created until the profile is activated. On-demand lifecycle managers — Sablier, autoheal-style wake-up tools, CI runners that start/stop existing containers on request — cannot manage what does not exist. The container has no Docker record for them to act on.

Fix: drop profiles: for services managed by an external lifecycle tool, and use docker compose create (not up) to materialise the containers in a stopped state. The lifecycle manager can then start them on demand.

GPU-Agnostic Base Compose with Provider Override Files

Keep the base docker-compose.yml free of hardware-specific runtime config. Put NVIDIA/AMD/Apple GPU or accelerator runtime settings in separate override files composed in with -f:

docker-compose.yml              # GPU-agnostic base
docker-compose.nvidia.yml       # NVIDIA runtime, device reservations
docker-compose.apple.yml        # Apple Silicon / Metal settings
docker-compose.amd.yml          # ROCm overrides

Deploy with docker compose -f docker-compose.yml -f docker-compose.nvidia.yml up -d. The same stack template deploys unchanged across heterogeneous hosts; adding a new accelerator type is an override file, not a fork of the base compose.

Snap-Packaged Docker Breaks After Unclean Shutdown

Symptom: containers fail with "read-only file system" on /opt/ or other snap-confined paths after a power cut or forced reboot, even though the disk is healthy.

Cause: snap auto-refresh or AppArmor profile corruption during unclean shutdown leaves Docker's snap confinement in an inconsistent state.

Fix: snap restart docker is usually enough; if not, snap revert docker. On production hosts, prefer distro-packaged or upstream Docker (docker-ce from Docker's apt repo) to avoid snap confinement entirely.

Set PYTHONPATH in Dockerfiles for src/ Layout Projects

Python projects using a src/ layout must set ENV PYTHONPATH=/app/src in every Dockerfile that does COPY src/ ./src/. Local development hides the problem because pip install -e . wires up imports via the editable install — the container has no such install, so imports fail only at runtime inside the container.

Set PYTHONPATH when scaffolding the Dockerfile, not after the first failed container run. Same rule applies to any multi-stage build where the final stage copies src/ without re-running pip install.

Write CLI Entrypoints Alongside the Module They Run

When a module will run inside a container, create its _cli.py entry point and verify the Dockerfile ENTRYPOINT in the same phase/commit as the module itself. Deferring CLI shims to "a later phase" repeatedly leads to containers that build cleanly but have no runnable entry point — the build passes, the deployment looks healthy, and the first invocation fails with No module named ... or an ENTRYPOINT that points at nothing.

Treat "module + CLI shim + ENTRYPOINT verification" as one atomic unit of work.

PostgreSQL Alpine Image Runs as UID 70, Not 999

The postgres:*-alpine images run postgres as UID 70, not the 999 used by Debian-based postgres:* images. Setting data-directory ownership to 999 (or 1000) via host chown or Ansible file modules silently breaks access — pg_isready may still pass while internal operations fail with "Permission denied" on WAL writes, replication slots, or extension installs.

Always verify before setting ownership:

docker exec --user postgres <container> id

Match the automation to the actual UID. If a project mixes Alpine and Debian Postgres images across environments, treat the UID as per-environment config, not a hardcoded constant.