# CI Container Build Best Practices

Practices for fast, reliable container image builds in CI — particularly Gitea Actions with Docker-in-Docker (DinD) runners, but applicable to any ephemeral CI runner environment.

## Enable Registry Cache with Inline Metadata

DinD runners have ephemeral Docker daemons — the local layer cache is lost between builds. Use Docker's inline cache to embed cache metadata in pushed images, then read it back on subsequent builds:

```yaml
- uses: docker/build-push-action@v5
  with:
    cache-from: type=registry,ref=registry.example.com/org/image:latest
    cache-to: type=inline
    tags: registry.example.com/org/image:latest
```

**Key details:**
- `cache-to: type=inline` embeds cache metadata in the image being pushed. No separate cache image needed.
- `cache-from` must reference the **actual image tag** (e.g., `:latest`), not a separate `:buildcache` tag. With inline caching, the cache metadata lives inside the image itself.
- `type=inline` only caches final-stage layers, not intermediate build stages. For single-stage Dockerfiles (most service images), this is equivalent to full caching.
- A missing `cache-from` reference (e.g., first build before any image exists) is a silent cache miss, not an error.
- This approach works with any OCI-compliant registry (Gitea Packages, GHCR, Docker Hub, etc.).

**Why not `type=registry` for cache-to?** It requires the `docker-container` buildx driver, which can stall in DinD (Docker-in-Docker-in-Docker). Inline is simpler and works with the `docker` driver.

## Use the Docker Buildx Driver in DinD

The default buildx driver (`docker-container`) spawns a BuildKit container inside the DinD daemon — Docker-in-Docker-in-Docker. This can silently stall or hang.

```yaml
- uses: docker/setup-buildx-action@v3
  with:
    driver: docker
```

The `docker` driver uses the DinD daemon's built-in BuildKit support directly. It's less feature-rich (no multi-platform builds, no `cache-to: type=registry`) but reliable in DinD environments.

**When to use `docker-container` instead:** If you need multi-platform builds (`platforms: linux/amd64,linux/arm64`) or `cache-to: type=registry`. Test thoroughly in your DinD setup first.

## Separate Dependency Layers from Application Code

Dependency installation (pip, npm, apt) is the most expensive CI step. Structure Dockerfiles so dependency layers cache independently from source code changes.

**For projects with `pyproject.toml` (setuptools):**

```dockerfile
COPY pyproject.toml .
# Create stub so setuptools can resolve deps without real source
RUN mkdir -p src/my_package && touch src/my_package/__init__.py
RUN pip install .

# Copy real source — only this layer busts on code changes
COPY src/ src/
RUN pip install . --no-deps
```

The stub package trick lets pip resolve and install all dependencies from `pyproject.toml` alone. The final `--no-deps` reinstall just links the real source without re-downloading anything.

**For projects with explicit dependency lists:**

```dockerfile
# Dependencies layer — only busts when this list changes
RUN pip install fastapi sqlalchemy httpx pydantic

WORKDIR /app
COPY src/ ./src/
```

**For projects where `pip install .` works without source** (e.g., some hatchling/flit backends that only need `pyproject.toml` for dependency resolution):

```dockerfile
COPY pyproject.toml .
RUN pip install ".[postgres]"

COPY controlplane/ /app/controlplane/
```

Test whether your build backend needs source present. setuptools does; hatchling may not.

## Don't Use `--no-cache-dir` with pip

`pip install --no-cache-dir` forces pip to re-download and rebuild wheels from source on every run. With Docker layer caching, this is unnecessary — if inputs haven't changed, the entire `RUN` layer is skipped.

When a layer cache miss does occur (e.g., a new dependency added), pip's local cache provides faster wheel reuse. The trade-off is ~300-500MB larger image layers, but build time savings of 2-4x on cache misses.

```dockerfile
# Before — slow on cache miss
RUN pip install --no-cache-dir .

# After — faster rebuilds
RUN pip install .
```

## Pre-Build Base Images on a Schedule

Base images with heavy, infrequently-changing layers (OS packages, language runtimes, CLI tools) should be built on a schedule rather than on every push:

```yaml
on:
  schedule:
    - cron: '0 15 * * *'  # Daily at 3am NZST
  push:
    branches: [main]
    paths:
      - "images/base/**"   # Only rebuild on actual base image changes
```

Downstream images reference the base via registry tag (`FROM registry/org/base:latest`). Application pushes skip the expensive base layers entirely.

**Tag strategy for daily builds:**
- `:base-daily` — rolling tag for the latest scheduled build
- `:latest` — always the most recent build (scheduled or push-triggered)
- `:sha-<short>` — commit-pinned for reproducibility

## Measured Impact

Real-world results from a Gitea Actions setup with 8 DinD runner replicas (1-2 CPU, 2-4GB RAM each):

| Image | Before (avg) | After (cached) | Improvement |
|---|---|---|---|
| controlplane (pip deps) | 3.7m | 2.2m | 41% faster |
| dispatcher (pip + SOPS/age) | 4.2m | 1.7m | 60% faster |
| agent-base (apt + pip + binaries) | 5.8m | 3.9m | 33% faster |
| agent-claude (npm install) | 3.8m | 4.0m | ~same |

**Why agent-claude didn't improve:** `npm install -g @anthropic-ai/claude-code` is a single monolithic layer. Any change to the base image (which it depends on) busts this layer. npm doesn't benefit from inline cache the way pip does because there's no equivalent layer separation.

**Total wall-clock for a 4-image push:** ~10m → ~8m (3 parallel + 1 sequential).

First build after enabling caching is slower (cache-seeding). Subsequent builds see the improvement.

## Common Pitfalls

### Cache reference must match the actual tag

With `cache-to: type=inline`, the cache is embedded in the pushed image. If you push `:latest` and `:sha-abc`, the cache lives in both. But `cache-from: type=registry,ref=image:buildcache` fails silently because no `:buildcache` tag exists. Always reference the tag you actually push:

```yaml
cache-from: type=registry,ref=my-image:latest   # correct
cache-from: type=registry,ref=my-image:buildcache  # wrong — tag doesn't exist
```

### pip install . needs source with setuptools

`pip install .` with setuptools requires the package directory to exist. Copying only `pyproject.toml` and running `pip install .` will fail. Use the stub package pattern (empty `__init__.py`) or switch to a backend that resolves deps from metadata alone.

### Ephemeral DinD means no persistent local cache

DinD sidecars use ephemeral storage. Docker's local layer cache is lost when the runner pod restarts. Registry cache is the only reliable persistence mechanism. Don't rely on local `type=local` cache paths — they'll be empty on every build.

### Path filters and workflow-only changes

CI workflows with path filters (e.g., `paths: ["src/**", "Dockerfile"]`) won't trigger when only the workflow file itself changes. This means cache configuration changes require a matching source change to trigger a build. Push a trivial change to a matched path to test.