best-practices/ci-container-builds.md

CI Container Build Best Practices

Practices for fast, reliable container image builds in CI — particularly Gitea Actions with Docker-in-Docker (DinD) runners, but applicable to any ephemeral CI runner environment.

Enable Registry Cache with Inline Metadata

DinD runners have ephemeral Docker daemons — the local layer cache is lost between builds. Use Docker's inline cache to embed cache metadata in pushed images, then read it back on subsequent builds:

- uses: docker/build-push-action@v5
  with:
    cache-from: type=registry,ref=registry.example.com/org/image:latest
    cache-to: type=inline
    tags: registry.example.com/org/image:latest

Key details:

• cache-to: type=inline embeds cache metadata in the image being pushed. No separate cache image needed.
• cache-from must reference the actual image tag (e.g., :latest), not a separate :buildcache tag. With inline caching, the cache metadata lives inside the image itself.
• type=inline only caches final-stage layers, not intermediate build stages. For single-stage Dockerfiles (most service images), this is equivalent to full caching.
• A missing cache-from reference (e.g., first build before any image exists) is a silent cache miss, not an error.
• This approach works with any OCI-compliant registry (Gitea Packages, GHCR, Docker Hub, etc.).

Why not type=registry for cache-to? It requires the docker-container buildx driver, which can stall in DinD (Docker-in-Docker-in-Docker). Inline is simpler and works with the docker driver.

Use the Docker Buildx Driver in DinD

The default buildx driver (docker-container) spawns a BuildKit container inside the DinD daemon — Docker-in-Docker-in-Docker. This can silently stall or hang.

- uses: docker/setup-buildx-action@v3
  with:
    driver: docker

The docker driver uses the DinD daemon's built-in BuildKit support directly. It's less feature-rich (no multi-platform builds, no cache-to: type=registry) but reliable in DinD environments.

When to use docker-container instead: If you need multi-platform builds (platforms: linux/amd64,linux/arm64) or cache-to: type=registry. Test thoroughly in your DinD setup first.

Separate Dependency Layers from Application Code

Dependency installation (pip, npm, apt) is the most expensive CI step. Structure Dockerfiles so dependency layers cache independently from source code changes.

For projects with pyproject.toml (setuptools):

COPY pyproject.toml .
# Create stub so setuptools can resolve deps without real source
RUN mkdir -p src/my_package && touch src/my_package/__init__.py
RUN pip install .

# Copy real source — only this layer busts on code changes
COPY src/ src/
RUN pip install . --no-deps

The stub package trick lets pip resolve and install all dependencies from pyproject.toml alone. The final --no-deps reinstall just links the real source without re-downloading anything.

For projects with explicit dependency lists:

# Dependencies layer — only busts when this list changes
RUN pip install fastapi sqlalchemy httpx pydantic

WORKDIR /app
COPY src/ ./src/

For projects where pip install . works without source (e.g., some hatchling/flit backends that only need pyproject.toml for dependency resolution):

COPY pyproject.toml .
RUN pip install ".[postgres]"

COPY controlplane/ /app/controlplane/

Test whether your build backend needs source present. setuptools does; hatchling may not.

Don't Use --no-cache-dir with pip

pip install --no-cache-dir forces pip to re-download and rebuild wheels from source on every run. With Docker layer caching, this is unnecessary — if inputs haven't changed, the entire RUN layer is skipped.

When a layer cache miss does occur (e.g., a new dependency added), pip's local cache provides faster wheel reuse. The trade-off is ~300-500MB larger image layers, but build time savings of 2-4x on cache misses.

# Before — slow on cache miss
RUN pip install --no-cache-dir .

# After — faster rebuilds
RUN pip install .

Pre-Build Base Images on a Schedule

Base images with heavy, infrequently-changing layers (OS packages, language runtimes, CLI tools) should be built on a schedule rather than on every push:

on:
  schedule:
    - cron: '0 15 * * *'  # Daily at 3am NZST
  push:
    branches: [main]
    paths:
      - "images/base/**"   # Only rebuild on actual base image changes

Downstream images reference the base via registry tag (FROM registry/org/base:latest). Application pushes skip the expensive base layers entirely.

Tag strategy for daily builds:

• :base-daily — rolling tag for the latest scheduled build
• :latest — always the most recent build (scheduled or push-triggered)
• :sha-<short> — commit-pinned for reproducibility

Measured Impact

Real-world results from a Gitea Actions setup with 8 DinD runner replicas (1-2 CPU, 2-4GB RAM each):

Image	Before (avg)	After (cached)	Improvement
controlplane (pip deps)	3.7m	2.2m	41% faster
dispatcher (pip + SOPS/age)	4.2m	1.7m	60% faster
agent-base (apt + pip + binaries)	5.8m	3.9m	33% faster
agent-claude (npm install)	3.8m	4.0m	~same

Why agent-claude didn't improve: npm install -g @anthropic-ai/claude-code is a single monolithic layer. Any change to the base image (which it depends on) busts this layer. npm doesn't benefit from inline cache the way pip does because there's no equivalent layer separation.

Total wall-clock for a 4-image push: ~10m → ~8m (3 parallel + 1 sequential).

First build after enabling caching is slower (cache-seeding). Subsequent builds see the improvement.

Common Pitfalls

Cache reference must match the actual tag

With cache-to: type=inline, the cache is embedded in the pushed image. If you push :latest and :sha-abc, the cache lives in both. But cache-from: type=registry,ref=image:buildcache fails silently because no :buildcache tag exists. Always reference the tag you actually push:

cache-from: type=registry,ref=my-image:latest   # correct
cache-from: type=registry,ref=my-image:buildcache  # wrong — tag doesn't exist

pip install . needs source with setuptools

pip install . with setuptools requires the package directory to exist. Copying only pyproject.toml and running pip install . will fail. Use the stub package pattern (empty __init__.py) or switch to a backend that resolves deps from metadata alone.

Ephemeral DinD means no persistent local cache

DinD sidecars use ephemeral storage. Docker's local layer cache is lost when the runner pod restarts. Registry cache is the only reliable persistence mechanism. Don't rely on local type=local cache paths — they'll be empty on every build.

Path filters and workflow-only changes

CI workflows with path filters (e.g., paths: ["src/**", "Dockerfile"]) won't trigger when only the workflow file itself changes. This means cache configuration changes require a matching source change to trigger a build. Push a trivial change to a matched path to test.

CI Path Filters Must Include All COPY'd Directories and Runtime-Mounted Paths

When a Dockerfile COPYs from a directory (e.g., harnesses/, models/), that directory must be in the CI workflow's paths: trigger filter. Otherwise, changes to those directories won't trigger image rebuilds, leaving deployed images stale. Always cross-check CI path triggers against Dockerfile COPY sources.

This also applies to directories that are mounted at runtime (not COPY'd) but whose contents affect the container's behaviour — e.g., a config/ directory bind-mounted into a container via Compose or K8s volume mount. A change to mounted config doesn't change the image, but it may require a rolling restart or cache invalidation step that the CI workflow should trigger. Include these paths in the workflow trigger and add a separate step (e.g., kubectl rollout restart) rather than assuming a build is required.

CI Image Tagging Strategy: Short SHA + Full SHA + Latest

Tag container images with three tags: sha-<7char> (human-readable in kubectl output), <full-sha> (exact traceability), and latest (local dev convenience). The sha- prefix distinguishes commit tags from version tags. Pin deploy manifests to commit SHAs via Kustomize images: blocks — git blame on the kustomization shows exactly when each version was deployed.

Multi-Registry / Tier-Separated Image Publishing

When separating image tiers by registry (e.g., org-nonprod/app on push-to-main, org-prod/app on v* tag):

• Give CI a service account that is a member of both registry orgs, and store one credential per registry (do not share a single token across tiers).
• Keep tag formats distinct per tier (e.g., sha-<8> for non-prod, prod-sha-<8> for prod) so downstream systems — Octopus channels, Kustomize overlays, audit tooling — can reason about provenance from the tag alone.
• Gate the prod publish workflow on v* tags or an explicit release event, never on main-branch pushes.