distill: 48 cross-project best-practices from 2026-07 reflection sweep

Promotions from reflecting 21 projects' session logs (incl. agent-runtimes
122-log drain). Adds coverage across networking (eBPF VIP/VPN SNAT/VLAN
bridge/forward-auth preflight/ingress TLS), kubernetes (CSI hotplug/PodSecurity
debug/self-managed GitOps/runtime annotations), CI (dispatch tokens/runner
death/base image), git (CI-rebase/shallow reset/PR governance), python (async
session pool/httpx redirects/logging), TDD (AsyncMock/xfail lifecycle),
api-integration (SDK parse/token-scope 404/schema probing), plus docker,
scripting, debugging, security-architecture, secrets, react, octopus.

State: .distill-state.json refreshed with current HEADs + 5 newly-tracked projects.

ci-container-builds.md

@@ -169,3 +169,78 @@ When separating image tiers by registry (e.g., `org-nonprod/app` on push-to-main
 - Give CI a **service account that is a member of both registry orgs**, and store one credential per registry (do not share a single token across tiers).
 - Keep tag formats **distinct per tier** (e.g., `sha-<8>` for non-prod, `prod-sha-<8>` for prod) so downstream systems — Octopus channels, Kustomize overlays, audit tooling — can reason about provenance from the tag alone.
 - Gate the prod publish workflow on `v*` tags or an explicit release event, never on main-branch pushes.
+
+## Codegen-Freshness Scripts Must Be Observation-Only and Sandboxed
+
+CI scripts that regenerate generated code from manifests and compare it to committed output (e.g., "the OpenAPI client matches the spec", "the generated K8s manifests match the Helm chart") must obey two rules:
+
+1. **Regenerate into a per-job temp directory** — use `${RUNNER_TEMP}` or `mktemp -d` inside the job's workspace. **Never `/tmp`** on shared multi-tenant runners — other jobs can plant `conftest.py`, `package.json` postinstall hooks, or shell rc files that exfiltrate from this job.
+2. **Compare with `diff -ru` only.** No `pip install`, `pytest`, `npm ci`, `make`, or any other command that executes code from the regenerated tree. A malicious manifest can plant code that runs during install/test/build and exfiltrates secrets or pivots through the runner.
+
+```yaml
+# Pattern: regenerate into sandbox, diff against committed copy, never execute
+- name: Check codegen freshness
+  run: |
+    SANDBOX=$(mktemp -d)
+    scripts/codegen.sh --output "$SANDBOX/generated"
+    diff -ru generated/ "$SANDBOX/generated/" || {
+      echo "Generated output is stale. Run scripts/codegen.sh and commit."
+      exit 1
+    }
+```
+
+Applies to any "verify the generated artifact matches the source" CI step: OpenAPI codegen, Helm chart generators, Terraform-from-manifest, protobuf compilation, GraphQL schema diff. The threat model is: a PR author plants a malicious manifest that, when regenerated, contains code that runs at test or install time.
+
+## Substring-Matching Security Tests Trip on User-Facing Comments
+
+When a CI-policy validator scans workflow or source files for forbidden binary names (`pip install`, `npm ci`, `pytest`, `curl http://`), it must distinguish prohibited **invocations** from echo'd error messages, code comments, and synonyms in user-facing strings.
+
+**Common failure mode:** the validator forbids `pip install` in workflows, but the workflow's failure-message handler contains the exact phrase as part of an error string:
+
+```yaml
+- name: Codegen check
+  run: |
+    diff -ru ... || {
+      echo "Stale output. Run pip install . then rerun codegen."  # ← matches the forbidden pattern
+      exit 1
+    }
+```
+
+The validator fires on its own remediation hint. **Fixes:**
+- Phrase user-facing text without naming the binaries literally: "package-install", "test runner", "build entry point".
+- Or use a multi-line form that breaks substring boundaries: `echo "Run: $(echo pip)" install`.
+- Or whitelist explicitly: `if [[ "$line" =~ pip[[:space:]]+install ]] && [[ "$line" != *"#"* ]] && [[ "$line" != *"echo"* ]]`.
+
+Same trap applies to any policy enforcement that matches code patterns in source files: pre-commit hooks scanning for secrets, lint rules forbidding API calls, banned-import walkers. Substring matchers must distinguish active invocations from quoted strings, comments, and documentation. A real AST/lexer-based check is harder to write but immune to the trap.
+
+## `if: always()` Steps Don't Survive Early Runner-Process Death
+
+A dispatch/notify step gated with `if: always()` is meant to run even when an earlier job step fails. But `always()` only runs the step while the **job process is still alive**. If the runner itself dies early (act-runner/agent crashes during checkout, pip, or setup before reaching the step), the whole job is torn down and `always()` steps never execute — so a downstream build dispatch silently never fires.
+
+- Symptom: intermittent — the same workflow fires the downstream build on most pushes but not all, with no error in the failed run's log.
+- Root cause is a runner/env flake, not a workflow bug; not reliably reproducible.
+- Do not treat `always()` as a guaranteed "run no matter what" — it is scoped to a living job.
+- Workaround / unblock: dispatch the downstream build manually via `workflow_dispatch`.
+- For a hard guarantee, trigger the downstream from an event the runner can't swallow (a separate scheduled/webhook-driven job, or the platform's native workflow-completed event), not from an in-job step.
+
+## Don't Trust the Job-Listing API's `run_id` Filter to Map Jobs to Runs
+
+On some CI backends (observed on Gitea Actions 1.25.x), the job/task listing API filtered by `?run_id=N` returns job entries belonging to *other* runs — you cannot reliably map a job back to its run through that filter. When correlating a job to a specific run programmatically, identify jobs by descending job ID or by timestamp, not by trusting the `run_id` filter.
+
+## A Bare Language Base Image Breaks Node-Based CI Actions
+
+Setting `container: image: python:3-slim` (or any minimal single-language base) as the job container removes tooling that CI actions assume is present. Node-based actions like `actions/checkout@v4` fail with "executable file not found" because there's no Node.js runtime, and steps using `curl` (e.g. API dispatch calls) break because curl isn't installed either.
+
+Fix: don't override the job container just to get a language runtime — the standard `ubuntu-latest` runner already ships Node.js, curl, and python3. Install extra language deps as a step (`pip install ...`) rather than swapping the whole container image.
+
+## Create CI Secrets Before Pushing Code That Uses Them
+
+A CI secret referenced by a workflow (`DISPATCH_TOKEN`, `ARGOCD_TOKEN`, a deploy PAT) must exist on the repo/org **before** you push the code that depends on it. The first run after the push otherwise executes against a missing secret — and many CI systems treat a missing secret as an empty string rather than an error, so the step **fails silently** (empty auth header, `workflow_dispatch` that no-ops). Create the secret via the platform API first, then push. Verify the secret exists rather than assuming the push "must have" picked it up.
+
+## Verify Which Branch the Deploy Controller Actually Tracks
+
+When porting a CI workflow from a template, confirm which branch the GitOps controller (ArgoCD/Flux) tracks before wiring the image-tag write-back. Templates assume `main`, but real repos often track `staging` or a release branch. If the build writes the new image tag to `deployment.yaml` on `main` while the app tracks `staging`, the deploy succeeds, CI goes green, and the pod stays on the old image — a silent no-op. Read/write the tag on the tracked branch (`?ref=staging`, `"branch":"staging"`).
+
+## Semver-Style Release Tools Skip Non-`main` Release Branches
+
+Tools that gate release-version emission on "the main branch" (semver-ci and similar) output an **empty version** when run on another release branch (e.g., `staging`), producing an invalid empty image tag. When you deliberately cut releases from a non-`main` branch, tell the tool that branch is a release branch (e.g., `--main-branch staging`), or it silently skips and downstream tagging breaks.