best-practices/ci-container-builds.md

CI Container Build Best Practices

Practices for fast, reliable container image builds in CI — particularly Gitea Actions with Docker-in-Docker (DinD) runners, but applicable to any ephemeral CI runner environment.

Enable Registry Cache with Inline Metadata

DinD runners have ephemeral Docker daemons — the local layer cache is lost between builds. Use Docker's inline cache to embed cache metadata in pushed images, then read it back on subsequent builds:

- uses: docker/build-push-action@v5
  with:
    cache-from: type=registry,ref=registry.example.com/org/image:latest
    cache-to: type=inline
    tags: registry.example.com/org/image:latest

Key details:

• cache-to: type=inline embeds cache metadata in the image being pushed. No separate cache image needed.
• cache-from must reference the actual image tag (e.g., :latest), not a separate :buildcache tag. With inline caching, the cache metadata lives inside the image itself.
• type=inline only caches final-stage layers, not intermediate build stages. For single-stage Dockerfiles (most service images), this is equivalent to full caching.
• A missing cache-from reference (e.g., first build before any image exists) is a silent cache miss, not an error.
• This approach works with any OCI-compliant registry (Gitea Packages, GHCR, Docker Hub, etc.).

Why not type=registry for cache-to? It requires the docker-container buildx driver, which can stall in DinD (Docker-in-Docker-in-Docker). Inline is simpler and works with the docker driver.

Use the Docker Buildx Driver in DinD

The default buildx driver (docker-container) spawns a BuildKit container inside the DinD daemon — Docker-in-Docker-in-Docker. This can silently stall or hang.

- uses: docker/setup-buildx-action@v3
  with:
    driver: docker

The docker driver uses the DinD daemon's built-in BuildKit support directly. It's less feature-rich (no multi-platform builds, no cache-to: type=registry) but reliable in DinD environments.

When to use docker-container instead: If you need multi-platform builds (platforms: linux/amd64,linux/arm64) or cache-to: type=registry. Test thoroughly in your DinD setup first.

Separate Dependency Layers from Application Code

Dependency installation (pip, npm, apt) is the most expensive CI step. Structure Dockerfiles so dependency layers cache independently from source code changes.

For projects with pyproject.toml (setuptools):

COPY pyproject.toml .
# Create stub so setuptools can resolve deps without real source
RUN mkdir -p src/my_package && touch src/my_package/__init__.py
RUN pip install .

# Copy real source — only this layer busts on code changes
COPY src/ src/
RUN pip install . --no-deps

The stub package trick lets pip resolve and install all dependencies from pyproject.toml alone. The final --no-deps reinstall just links the real source without re-downloading anything.

For projects with explicit dependency lists:

# Dependencies layer — only busts when this list changes
RUN pip install fastapi sqlalchemy httpx pydantic

WORKDIR /app
COPY src/ ./src/

For projects where pip install . works without source (e.g., some hatchling/flit backends that only need pyproject.toml for dependency resolution):

COPY pyproject.toml .
RUN pip install ".[postgres]"

COPY controlplane/ /app/controlplane/

Test whether your build backend needs source present. setuptools does; hatchling may not.

Don't Use --no-cache-dir with pip

pip install --no-cache-dir forces pip to re-download and rebuild wheels from source on every run. With Docker layer caching, this is unnecessary — if inputs haven't changed, the entire RUN layer is skipped.

When a layer cache miss does occur (e.g., a new dependency added), pip's local cache provides faster wheel reuse. The trade-off is ~300-500MB larger image layers, but build time savings of 2-4x on cache misses.

# Before — slow on cache miss
RUN pip install --no-cache-dir .

# After — faster rebuilds
RUN pip install .

Pre-Build Base Images on a Schedule

Base images with heavy, infrequently-changing layers (OS packages, language runtimes, CLI tools) should be built on a schedule rather than on every push:

on:
  schedule:
    - cron: '0 15 * * *'  # Daily at 3am NZST
  push:
    branches: [main]
    paths:
      - "images/base/**"   # Only rebuild on actual base image changes

Downstream images reference the base via registry tag (FROM registry/org/base:latest). Application pushes skip the expensive base layers entirely.

Tag strategy for daily builds:

• :base-daily — rolling tag for the latest scheduled build
• :latest — always the most recent build (scheduled or push-triggered)
• :sha-<short> — commit-pinned for reproducibility

Measured Impact

Real-world results from a Gitea Actions setup with 8 DinD runner replicas (1-2 CPU, 2-4GB RAM each):

Image	Before (avg)	After (cached)	Improvement
controlplane (pip deps)	3.7m	2.2m	41% faster
dispatcher (pip + SOPS/age)	4.2m	1.7m	60% faster
agent-base (apt + pip + binaries)	5.8m	3.9m	33% faster
agent-claude (npm install)	3.8m	4.0m	~same

Why agent-claude didn't improve: npm install -g @anthropic-ai/claude-code is a single monolithic layer. Any change to the base image (which it depends on) busts this layer. npm doesn't benefit from inline cache the way pip does because there's no equivalent layer separation.

Total wall-clock for a 4-image push: ~10m → ~8m (3 parallel + 1 sequential).

First build after enabling caching is slower (cache-seeding). Subsequent builds see the improvement.

Common Pitfalls

Cache reference must match the actual tag

With cache-to: type=inline, the cache is embedded in the pushed image. If you push :latest and :sha-abc, the cache lives in both. But cache-from: type=registry,ref=image:buildcache fails silently because no :buildcache tag exists. Always reference the tag you actually push:

cache-from: type=registry,ref=my-image:latest   # correct
cache-from: type=registry,ref=my-image:buildcache  # wrong — tag doesn't exist

pip install . needs source with setuptools

pip install . with setuptools requires the package directory to exist. Copying only pyproject.toml and running pip install . will fail. Use the stub package pattern (empty __init__.py) or switch to a backend that resolves deps from metadata alone.

Ephemeral DinD means no persistent local cache

DinD sidecars use ephemeral storage. Docker's local layer cache is lost when the runner pod restarts. Registry cache is the only reliable persistence mechanism. Don't rely on local type=local cache paths — they'll be empty on every build.

Path filters and workflow-only changes

CI workflows with path filters (e.g., paths: ["src/**", "Dockerfile"]) won't trigger when only the workflow file itself changes. This means cache configuration changes require a matching source change to trigger a build. Push a trivial change to a matched path to test.

CI Path Filters Must Include All COPY'd Directories and Runtime-Mounted Paths

When a Dockerfile COPYs from a directory (e.g., harnesses/, models/), that directory must be in the CI workflow's paths: trigger filter. Otherwise, changes to those directories won't trigger image rebuilds, leaving deployed images stale. Always cross-check CI path triggers against Dockerfile COPY sources.

This also applies to directories that are mounted at runtime (not COPY'd) but whose contents affect the container's behaviour — e.g., a config/ directory bind-mounted into a container via Compose or K8s volume mount. A change to mounted config doesn't change the image, but it may require a rolling restart or cache invalidation step that the CI workflow should trigger. Include these paths in the workflow trigger and add a separate step (e.g., kubectl rollout restart) rather than assuming a build is required.

CI Image Tagging Strategy: Short SHA + Full SHA + Latest

Tag container images with three tags: sha-<7char> (human-readable in kubectl output), <full-sha> (exact traceability), and latest (local dev convenience). The sha- prefix distinguishes commit tags from version tags. Pin deploy manifests to commit SHAs via Kustomize images: blocks — git blame on the kustomization shows exactly when each version was deployed.

Multi-Registry / Tier-Separated Image Publishing

When separating image tiers by registry (e.g., org-nonprod/app on push-to-main, org-prod/app on v* tag):

• Give CI a service account that is a member of both registry orgs, and store one credential per registry (do not share a single token across tiers).
• Keep tag formats distinct per tier (e.g., sha-<8> for non-prod, prod-sha-<8> for prod) so downstream systems — Octopus channels, Kustomize overlays, audit tooling — can reason about provenance from the tag alone.
• Gate the prod publish workflow on v* tags or an explicit release event, never on main-branch pushes.

Codegen-Freshness Scripts Must Be Observation-Only and Sandboxed

CI scripts that regenerate generated code from manifests and compare it to committed output (e.g., "the OpenAPI client matches the spec", "the generated K8s manifests match the Helm chart") must obey two rules:

1. Regenerate into a per-job temp directory — use ${RUNNER_TEMP} or mktemp -d inside the job's workspace. Never /tmp on shared multi-tenant runners — other jobs can plant conftest.py, package.json postinstall hooks, or shell rc files that exfiltrate from this job.
2. Compare with diff -ru only. No pip install, pytest, npm ci, make, or any other command that executes code from the regenerated tree. A malicious manifest can plant code that runs during install/test/build and exfiltrates secrets or pivots through the runner.

# Pattern: regenerate into sandbox, diff against committed copy, never execute
- name: Check codegen freshness
  run: |
    SANDBOX=$(mktemp -d)
    scripts/codegen.sh --output "$SANDBOX/generated"
    diff -ru generated/ "$SANDBOX/generated/" || {
      echo "Generated output is stale. Run scripts/codegen.sh and commit."
      exit 1
    }

Applies to any "verify the generated artifact matches the source" CI step: OpenAPI codegen, Helm chart generators, Terraform-from-manifest, protobuf compilation, GraphQL schema diff. The threat model is: a PR author plants a malicious manifest that, when regenerated, contains code that runs at test or install time.

Substring-Matching Security Tests Trip on User-Facing Comments

When a CI-policy validator scans workflow or source files for forbidden binary names (pip install, npm ci, pytest, curl http://), it must distinguish prohibited invocations from echo'd error messages, code comments, and synonyms in user-facing strings.

Common failure mode: the validator forbids pip install in workflows, but the workflow's failure-message handler contains the exact phrase as part of an error string:

- name: Codegen check
  run: |
    diff -ru ... || {
      echo "Stale output. Run pip install . then rerun codegen."  # ← matches the forbidden pattern
      exit 1
    }

The validator fires on its own remediation hint. Fixes:

• Phrase user-facing text without naming the binaries literally: "package-install", "test runner", "build entry point".
• Or use a multi-line form that breaks substring boundaries: echo "Run: $(echo pip)" install.
• Or whitelist explicitly: if [[ "$line" =~ pip[[:space:]]+install ]] && [[ "$line" != *"#"* ]] && [[ "$line" != *"echo"* ]].

Same trap applies to any policy enforcement that matches code patterns in source files: pre-commit hooks scanning for secrets, lint rules forbidding API calls, banned-import walkers. Substring matchers must distinguish active invocations from quoted strings, comments, and documentation. A real AST/lexer-based check is harder to write but immune to the trap.

if: always() Steps Don't Survive Early Runner-Process Death

A dispatch/notify step gated with if: always() is meant to run even when an earlier job step fails. But always() only runs the step while the job process is still alive. If the runner itself dies early (act-runner/agent crashes during checkout, pip, or setup before reaching the step), the whole job is torn down and always() steps never execute — so a downstream build dispatch silently never fires.

• Symptom: intermittent — the same workflow fires the downstream build on most pushes but not all, with no error in the failed run's log.
• Root cause is a runner/env flake, not a workflow bug; not reliably reproducible.
• Do not treat always() as a guaranteed "run no matter what" — it is scoped to a living job.
• Workaround / unblock: dispatch the downstream build manually via workflow_dispatch.
• For a hard guarantee, trigger the downstream from an event the runner can't swallow (a separate scheduled/webhook-driven job, or the platform's native workflow-completed event), not from an in-job step.

Don't Trust the Job-Listing API's run_id Filter to Map Jobs to Runs

On some CI backends (observed on Gitea Actions 1.25.x), the job/task listing API filtered by ?run_id=N returns job entries belonging to other runs — you cannot reliably map a job back to its run through that filter. When correlating a job to a specific run programmatically, identify jobs by descending job ID or by timestamp, not by trusting the run_id filter.

A Bare Language Base Image Breaks Node-Based CI Actions

Setting container: image: python:3-slim (or any minimal single-language base) as the job container removes tooling that CI actions assume is present. Node-based actions like actions/checkout@v4 fail with "executable file not found" because there's no Node.js runtime, and steps using curl (e.g. API dispatch calls) break because curl isn't installed either.

Fix: don't override the job container just to get a language runtime — the standard ubuntu-latest runner already ships Node.js, curl, and python3. Install extra language deps as a step (pip install ...) rather than swapping the whole container image.

Create CI Secrets Before Pushing Code That Uses Them

A CI secret referenced by a workflow (DISPATCH_TOKEN, ARGOCD_TOKEN, a deploy PAT) must exist on the repo/org before you push the code that depends on it. The first run after the push otherwise executes against a missing secret — and many CI systems treat a missing secret as an empty string rather than an error, so the step fails silently (empty auth header, workflow_dispatch that no-ops). Create the secret via the platform API first, then push. Verify the secret exists rather than assuming the push "must have" picked it up.

Verify Which Branch the Deploy Controller Actually Tracks

When porting a CI workflow from a template, confirm which branch the GitOps controller (ArgoCD/Flux) tracks before wiring the image-tag write-back. Templates assume main, but real repos often track staging or a release branch. If the build writes the new image tag to deployment.yaml on main while the app tracks staging, the deploy succeeds, CI goes green, and the pod stays on the old image — a silent no-op. Read/write the tag on the tracked branch (?ref=staging, "branch":"staging").

Semver-Style Release Tools Skip Non-main Release Branches

Tools that gate release-version emission on "the main branch" (semver-ci and similar) output an empty version when run on another release branch (e.g., staging), producing an invalid empty image tag. When you deliberately cut releases from a non-main branch, tell the tool that branch is a release branch (e.g., --main-branch staging), or it silently skips and downstream tagging breaks.