best-practices/docker.md

Docker Best Practices

Use gosu for Entrypoint Privilege Dropping

su -c "command" and sudo -u agent command create child processes. The real command is not PID 1, so Docker signals (SIGTERM on stop) don't reach it. Use gosu agent command which execs directly — the command becomes PID 1 with proper signal handling.

GIT_SSH_COMMAND Only Affects Git-Invoked SSH

GIT_SSH_COMMAND only applies when git invokes SSH (clone, push, fetch). Direct ssh calls need explicit flags: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null. Don't assume setting GIT_SSH_COMMAND fixes all SSH operations in a container.

Use Python urllib for Health Checks in Slim Images

Service images based on python:3.12-slim don't include curl. For in-container health checks, use python3 -c "import urllib.request; urllib.request.urlopen('http://...')". This applies to verification scripts using kubectl exec and to Kubernetes liveness/readiness probes that exec into containers.

Buildx Docker-Container Driver Can't See Local Images

When using buildx with the docker-container driver, FROM local-image:latest tries Docker Hub because the builder runs in a separate container that can't see locally-loaded images. Always use the full registry path in Dockerfiles. In CI, split into sequential jobs so base images are pushed to the registry before dependent images build.

Delete Conflicting Default Users at Build Time

Ubuntu 24.04 base images ship with a ubuntu user at UID 1000 — the most common host UID. This causes usermod -u 1000 conflicts and can trigger non-deterministic hangs (e.g., newgrp ubuntu waiting for a password on stdin). Delete the default user in the Dockerfile: RUN userdel -r ubuntu.

Service Images Should Use Minimal Base Images

Service images (API servers, background workers) should use python:3.12-slim or equivalent, not the agent base image. Agent base images include CLIs, Node.js, and other tooling that bloats service images unnecessarily. Keep agent tooling in agent images only.

Platform-Specific Native Binaries

Never mount host node_modules into a Docker container when the build uses platform-specific native binaries (e.g., Tailwind CSS, esbuild, SWC). Always run npm install inside the same container that runs the build. The native binary is compiled for the platform where npm install runs — host and container may differ in libc, architecture, or OS.

Symptom: Cannot find native binding or Cannot find module '@tailwindcss/oxide-linux-x64-gnu' Fix: Run npm install inside the container, not on the host.

Docker Wrapper Scripts and TTY Flags

Docker wrapper scripts (e.g., ~/sbin/hugo calling docker run -it ...) fail with the input device is not a TTY in non-interactive contexts (CI pipelines, Claude Code, cron jobs, scripts).

Fix: Only pass -t when stdin is a terminal: [ -t 0 ] && TTY_FLAG="-t" || TTY_FLAG="". Or omit -t entirely and let callers add it when needed.

Three-Tier UID Resolution

The UID wrapper should resolve the target UID/GID using this priority:

1. Environment variables (AGENT_UID/AGENT_GID) — injected by the orchestrator/dispatcher. Preferred because it's explicit and deployment-specific.
2. stat the mount point — detect the UID/GID of the mounted directory. Works when no env vars are set.
3. Skip — if not root or no mount point exists, run as the default container user.

This makes UID matching a deployment concern (varies per host), not a configuration concern (baked into images). See Docker UID Matching for the full UID wrapper pattern.

network_mode: service:* Breaks on Parent Container Restart

When a container shares another's network namespace via network_mode: "service:<parent>", restarting the parent recreates the namespace. The dependent container keeps stale socket bindings — its listeners are bound to a namespace that no longer exists. TCP connections fail while the dependent container appears healthy. Fix: add a healthcheck to the parent and use depends_on: condition: service_healthy on the dependent. Alternatively, use an internal bridge network instead of namespace sharing.

docker compose restart Is Concurrent, Not Ordered

docker compose restart svc1 svc2 svc3 restarts all named services concurrently, ignoring depends_on ordering. Dependent services may start before their dependencies are ready. Use docker compose up -d (which respects depends_on) or restart in explicit stages: stop dependents, restart the dependency, wait for healthy, then start dependents.

docker cp Can Corrupt Container Filesystem Ownership

docker cp runs as root and can change ownership of parent directories in the container's filesystem layer. This is particularly dangerous for database containers (e.g., PostgreSQL UID 999) — copying a file into /tmp/ can corrupt the data directory ownership, causing "Permission denied" errors. After any docker cp into a stateful container, verify and fix ownership: chown -R <expected-uid>:<expected-gid> <data-dir>.

Prefer Internal Bridge Networks Over Namespace Sharing

For sidecar-style containers that need to communicate (e.g., app + database, app + TLS proxy), prefer an internal Docker bridge network over network_mode: "service:<parent>". Bridge networks allow proper depends_on ordering with health checks, independent restart of each container, and clear network isolation. Namespace sharing couples container lifecycles — restarting the parent invalidates the dependent's network stack.

Verify Dockerfile COPY After Creating New Files

After creating a file intended for a Docker image (scripts, configs, wrappers), immediately add the COPY line to the Dockerfile and verify with docker run --entrypoint sh <image> -c "ls /path/to/file". Files placed in image source directories are not automatically included — they need explicit COPY instructions. This class of bug can remain latent until the code path is first exercised.

docker-compose.override.yaml Merges Lists Additively

Docker Compose V2 merges list fields (ports, volumes, environment) by appending, not replacing. An override file with a different port mapping adds a second binding rather than replacing the original, causing conflicts. Modify the base docker-compose.yaml directly or use !override for list replacement.

Volume Source Paths Must Be Absolute

Docker interprets relative paths in volume mount source fields as named volumes, not bind mounts. Use os.path.abspath() or equivalent when constructing volume source paths programmatically. The error message ("includes invalid characters for a local volume name") is misleading — the real issue is that the path is relative.