best-practices/ci-container-builds.md

# CI Container Build Best Practices

Practices for fast, reliable container image builds in CI — particularly Gitea Actions with Docker-in-Docker (DinD) runners, but applicable to any ephemeral CI runner environment.

## Enable Registry Cache with Inline Metadata

DinD runners have ephemeral Docker daemons — the local layer cache is lost between builds. Use Docker's inline cache to embed cache metadata in pushed images, then read it back on subsequent builds:

```yaml
- uses: docker/build-push-action@v5
  with:
    cache-from: type=registry,ref=registry.example.com/org/image:latest
    cache-to: type=inline
    tags: registry.example.com/org/image:latest
```

**Key details:**
- `cache-to: type=inline` embeds cache metadata in the image being pushed. No separate cache image needed.
- `cache-from` must reference the **actual image tag** (e.g., `:latest`), not a separate `:buildcache` tag. With inline caching, the cache metadata lives inside the image itself.
- `type=inline` only caches final-stage layers, not intermediate build stages. For single-stage Dockerfiles (most service images), this is equivalent to full caching.
- A missing `cache-from` reference (e.g., first build before any image exists) is a silent cache miss, not an error.
- This approach works with any OCI-compliant registry (Gitea Packages, GHCR, Docker Hub, etc.).

**Why not `type=registry` for cache-to?** It requires the `docker-container` buildx driver, which can stall in DinD (Docker-in-Docker-in-Docker). Inline is simpler and works with the `docker` driver.

## Use the Docker Buildx Driver in DinD

The default buildx driver (`docker-container`) spawns a BuildKit container inside the DinD daemon — Docker-in-Docker-in-Docker. This can silently stall or hang.

```yaml
- uses: docker/setup-buildx-action@v3
  with:
    driver: docker
```

The `docker` driver uses the DinD daemon's built-in BuildKit support directly. It's less feature-rich (no multi-platform builds, no `cache-to: type=registry`) but reliable in DinD environments.

**When to use `docker-container` instead:** If you need multi-platform builds (`platforms: linux/amd64,linux/arm64`) or `cache-to: type=registry`. Test thoroughly in your DinD setup first.

## Separate Dependency Layers from Application Code

Dependency installation (pip, npm, apt) is the most expensive CI step. Structure Dockerfiles so dependency layers cache independently from source code changes.

**For projects with `pyproject.toml` (setuptools):**

```dockerfile
COPY pyproject.toml .
# Create stub so setuptools can resolve deps without real source
RUN mkdir -p src/my_package && touch src/my_package/__init__.py
RUN pip install .

# Copy real source — only this layer busts on code changes
COPY src/ src/
RUN pip install . --no-deps
```

The stub package trick lets pip resolve and install all dependencies from `pyproject.toml` alone. The final `--no-deps` reinstall just links the real source without re-downloading anything.

**For projects with explicit dependency lists:**

```dockerfile
# Dependencies layer — only busts when this list changes
RUN pip install fastapi sqlalchemy httpx pydantic

WORKDIR /app
COPY src/ ./src/
```

**For projects where `pip install .` works without source** (e.g., some hatchling/flit backends that only need `pyproject.toml` for dependency resolution):

```dockerfile
COPY pyproject.toml .
RUN pip install ".[postgres]"

COPY controlplane/ /app/controlplane/
```

Test whether your build backend needs source present. setuptools does; hatchling may not.

## Don't Use `--no-cache-dir` with pip

`pip install --no-cache-dir` forces pip to re-download and rebuild wheels from source on every run. With Docker layer caching, this is unnecessary — if inputs haven't changed, the entire `RUN` layer is skipped.

When a layer cache miss does occur (e.g., a new dependency added), pip's local cache provides faster wheel reuse. The trade-off is ~300-500MB larger image layers, but build time savings of 2-4x on cache misses.

```dockerfile
# Before — slow on cache miss
RUN pip install --no-cache-dir .

# After — faster rebuilds
RUN pip install .
```

## Pre-Build Base Images on a Schedule

Base images with heavy, infrequently-changing layers (OS packages, language runtimes, CLI tools) should be built on a schedule rather than on every push:

```yaml
on:
  schedule:
    - cron: '0 15 * * *'  # Daily at 3am NZST
  push:
    branches: [main]
    paths:
      - "images/base/**"   # Only rebuild on actual base image changes
```

Downstream images reference the base via registry tag (`FROM registry/org/base:latest`). Application pushes skip the expensive base layers entirely.

**Tag strategy for daily builds:**
- `:base-daily` — rolling tag for the latest scheduled build
- `:latest` — always the most recent build (scheduled or push-triggered)
- `:sha-<short>` — commit-pinned for reproducibility

## Measured Impact

Real-world results from a Gitea Actions setup with 8 DinD runner replicas (1-2 CPU, 2-4GB RAM each):

| Image | Before (avg) | After (cached) | Improvement |
|---|---|---|---|
| controlplane (pip deps) | 3.7m | 2.2m | 41% faster |
| dispatcher (pip + SOPS/age) | 4.2m | 1.7m | 60% faster |
| agent-base (apt + pip + binaries) | 5.8m | 3.9m | 33% faster |
| agent-claude (npm install) | 3.8m | 4.0m | ~same |

**Why agent-claude didn't improve:** `npm install -g @anthropic-ai/claude-code` is a single monolithic layer. Any change to the base image (which it depends on) busts this layer. npm doesn't benefit from inline cache the way pip does because there's no equivalent layer separation.

**Total wall-clock for a 4-image push:** ~10m → ~8m (3 parallel + 1 sequential).

First build after enabling caching is slower (cache-seeding). Subsequent builds see the improvement.

## Common Pitfalls

### Cache reference must match the actual tag

With `cache-to: type=inline`, the cache is embedded in the pushed image. If you push `:latest` and `:sha-abc`, the cache lives in both. But `cache-from: type=registry,ref=image:buildcache` fails silently because no `:buildcache` tag exists. Always reference the tag you actually push:

```yaml
cache-from: type=registry,ref=my-image:latest   # correct
cache-from: type=registry,ref=my-image:buildcache  # wrong — tag doesn't exist
```

### pip install . needs source with setuptools

`pip install .` with setuptools requires the package directory to exist. Copying only `pyproject.toml` and running `pip install .` will fail. Use the stub package pattern (empty `__init__.py`) or switch to a backend that resolves deps from metadata alone.

### Ephemeral DinD means no persistent local cache

DinD sidecars use ephemeral storage. Docker's local layer cache is lost when the runner pod restarts. Registry cache is the only reliable persistence mechanism. Don't rely on local `type=local` cache paths — they'll be empty on every build.

### Path filters and workflow-only changes

CI workflows with path filters (e.g., `paths: ["src/**", "Dockerfile"]`) won't trigger when only the workflow file itself changes. This means cache configuration changes require a matching source change to trigger a build. Push a trivial change to a matched path to test.

### CI Path Filters Must Include All COPY'd Directories and Runtime-Mounted Paths

When a Dockerfile COPYs from a directory (e.g., `harnesses/`, `models/`), that directory must be in the CI workflow's `paths:` trigger filter. Otherwise, changes to those directories won't trigger image rebuilds, leaving deployed images stale. Always cross-check CI path triggers against Dockerfile COPY sources.

This also applies to directories that are **mounted at runtime** (not COPY'd) but whose contents affect the container's behaviour — e.g., a `config/` directory bind-mounted into a container via Compose or K8s volume mount. A change to mounted config doesn't change the image, but it may require a rolling restart or cache invalidation step that the CI workflow should trigger. Include these paths in the workflow trigger and add a separate step (e.g., `kubectl rollout restart`) rather than assuming a build is required.

### CI Image Tagging Strategy: Short SHA + Full SHA + Latest

Tag container images with three tags: `sha-<7char>` (human-readable in kubectl output), `<full-sha>` (exact traceability), and `latest` (local dev convenience). The `sha-` prefix distinguishes commit tags from version tags. Pin deploy manifests to commit SHAs via Kustomize `images:` blocks — `git blame` on the kustomization shows exactly when each version was deployed.

### Multi-Registry / Tier-Separated Image Publishing

When separating image tiers by registry (e.g., `org-nonprod/app` on push-to-main, `org-prod/app` on `v*` tag):

- Give CI a **service account that is a member of both registry orgs**, and store one credential per registry (do not share a single token across tiers).
- Keep tag formats **distinct per tier** (e.g., `sha-<8>` for non-prod, `prod-sha-<8>` for prod) so downstream systems — Octopus channels, Kustomize overlays, audit tooling — can reason about provenance from the tag alone.
- Gate the prod publish workflow on `v*` tags or an explicit release event, never on main-branch pushes.

## Codegen-Freshness Scripts Must Be Observation-Only and Sandboxed

CI scripts that regenerate generated code from manifests and compare it to committed output (e.g., "the OpenAPI client matches the spec", "the generated K8s manifests match the Helm chart") must obey two rules:

1. **Regenerate into a per-job temp directory** — use `${RUNNER_TEMP}` or `mktemp -d` inside the job's workspace. **Never `/tmp`** on shared multi-tenant runners — other jobs can plant `conftest.py`, `package.json` postinstall hooks, or shell rc files that exfiltrate from this job.
2. **Compare with `diff -ru` only.** No `pip install`, `pytest`, `npm ci`, `make`, or any other command that executes code from the regenerated tree. A malicious manifest can plant code that runs during install/test/build and exfiltrates secrets or pivots through the runner.

```yaml
# Pattern: regenerate into sandbox, diff against committed copy, never execute
- name: Check codegen freshness
  run: |
    SANDBOX=$(mktemp -d)
    scripts/codegen.sh --output "$SANDBOX/generated"
    diff -ru generated/ "$SANDBOX/generated/" || {
      echo "Generated output is stale. Run scripts/codegen.sh and commit."
      exit 1
    }
```

Applies to any "verify the generated artifact matches the source" CI step: OpenAPI codegen, Helm chart generators, Terraform-from-manifest, protobuf compilation, GraphQL schema diff. The threat model is: a PR author plants a malicious manifest that, when regenerated, contains code that runs at test or install time.

## Substring-Matching Security Tests Trip on User-Facing Comments

When a CI-policy validator scans workflow or source files for forbidden binary names (`pip install`, `npm ci`, `pytest`, `curl http://`), it must distinguish prohibited **invocations** from echo'd error messages, code comments, and synonyms in user-facing strings.

**Common failure mode:** the validator forbids `pip install` in workflows, but the workflow's failure-message handler contains the exact phrase as part of an error string:

```yaml
- name: Codegen check
  run: |
    diff -ru ... || {
      echo "Stale output. Run pip install . then rerun codegen."  # ← matches the forbidden pattern
      exit 1
    }
```

The validator fires on its own remediation hint. **Fixes:**
- Phrase user-facing text without naming the binaries literally: "package-install", "test runner", "build entry point".
- Or use a multi-line form that breaks substring boundaries: `echo "Run: $(echo pip)" install`.
- Or whitelist explicitly: `if [[ "$line" =~ pip[[:space:]]+install ]] && [[ "$line" != *"#"* ]] && [[ "$line" != *"echo"* ]]`.

Same trap applies to any policy enforcement that matches code patterns in source files: pre-commit hooks scanning for secrets, lint rules forbidding API calls, banned-import walkers. Substring matchers must distinguish active invocations from quoted strings, comments, and documentation. A real AST/lexer-based check is harder to write but immune to the trap.

## `if: always()` Steps Don't Survive Early Runner-Process Death

A dispatch/notify step gated with `if: always()` is meant to run even when an earlier job step fails. But `always()` only runs the step while the **job process is still alive**. If the runner itself dies early (act-runner/agent crashes during checkout, pip, or setup before reaching the step), the whole job is torn down and `always()` steps never execute — so a downstream build dispatch silently never fires.

- Symptom: intermittent — the same workflow fires the downstream build on most pushes but not all, with no error in the failed run's log.
- Root cause is a runner/env flake, not a workflow bug; not reliably reproducible.
- Do not treat `always()` as a guaranteed "run no matter what" — it is scoped to a living job.
- Workaround / unblock: dispatch the downstream build manually via `workflow_dispatch`.
- For a hard guarantee, trigger the downstream from an event the runner can't swallow (a separate scheduled/webhook-driven job, or the platform's native workflow-completed event), not from an in-job step.

## Don't Trust the Job-Listing API's `run_id` Filter to Map Jobs to Runs

On some CI backends (observed on Gitea Actions 1.25.x), the job/task listing API filtered by `?run_id=N` returns job entries belonging to *other* runs — you cannot reliably map a job back to its run through that filter. When correlating a job to a specific run programmatically, identify jobs by descending job ID or by timestamp, not by trusting the `run_id` filter.

## A Bare Language Base Image Breaks Node-Based CI Actions

Setting `container: image: python:3-slim` (or any minimal single-language base) as the job container removes tooling that CI actions assume is present. Node-based actions like `actions/checkout@v4` fail with "executable file not found" because there's no Node.js runtime, and steps using `curl` (e.g. API dispatch calls) break because curl isn't installed either.

Fix: don't override the job container just to get a language runtime — the standard `ubuntu-latest` runner already ships Node.js, curl, and python3. Install extra language deps as a step (`pip install ...`) rather than swapping the whole container image.

## Create CI Secrets Before Pushing Code That Uses Them

A CI secret referenced by a workflow (`DISPATCH_TOKEN`, `ARGOCD_TOKEN`, a deploy PAT) must exist on the repo/org **before** you push the code that depends on it. The first run after the push otherwise executes against a missing secret — and many CI systems treat a missing secret as an empty string rather than an error, so the step **fails silently** (empty auth header, `workflow_dispatch` that no-ops). Create the secret via the platform API first, then push. Verify the secret exists rather than assuming the push "must have" picked it up.

## Verify Which Branch the Deploy Controller Actually Tracks

When porting a CI workflow from a template, confirm which branch the GitOps controller (ArgoCD/Flux) tracks before wiring the image-tag write-back. Templates assume `main`, but real repos often track `staging` or a release branch. If the build writes the new image tag to `deployment.yaml` on `main` while the app tracks `staging`, the deploy succeeds, CI goes green, and the pod stays on the old image — a silent no-op. Read/write the tag on the tracked branch (`?ref=staging`, `"branch":"staging"`).

## Semver-Style Release Tools Skip Non-`main` Release Branches

Tools that gate release-version emission on "the main branch" (semver-ci and similar) output an **empty version** when run on another release branch (e.g., `staging`), producing an invalid empty image tag. When you deliberately cut releases from a non-`main` branch, tell the tool that branch is a release branch (e.g., `--main-branch staging`), or it silently skips and downstream tagging breaks.