distill: 48 cross-project best-practices from 2026-07 reflection sweep

Promotions from reflecting 21 projects' session logs (incl. agent-runtimes
122-log drain). Adds coverage across networking (eBPF VIP/VPN SNAT/VLAN
bridge/forward-auth preflight/ingress TLS), kubernetes (CSI hotplug/PodSecurity
debug/self-managed GitOps/runtime annotations), CI (dispatch tokens/runner
death/base image), git (CI-rebase/shallow reset/PR governance), python (async
session pool/httpx redirects/logging), TDD (AsyncMock/xfail lifecycle),
api-integration (SDK parse/token-scope 404/schema probing), plus docker,
scripting, debugging, security-architecture, secrets, react, octopus.

State: .distill-state.json refreshed with current HEADs + 5 newly-tracked projects.

kubernetes.md

@@ -52,6 +52,10 @@ Manual bootstrap secrets (encryption keys, OIDC client secrets) must be document
 
 - **Never use imperative operations on GitOps-managed resources.** `kubectl rollout restart` adds annotations that conflict with the GitOps controller's desired state, causing permanent OutOfSync. Use declarative paths instead — update a configmap hash annotation in Git, or change a pod template label.
 - **ArgoCD reconciliation has latency.** New Application manifests don't appear immediately due to polling intervals. Use manual refresh annotations when automation needs immediate reconciliation.
+- **Self-managed GitOps controllers revert their own live config.** When the GitOps controller manages itself via its own Helm chart with `selfHeal: true` (e.g., ArgoCD reconciling `argocd-cm`/`argocd-rbac-cm`), direct `kubectl patch`/`apply` on its ConfigMaps is reverted within seconds. Change the controller's Helm `values.yaml` in the deploy repo (accounts, RBAC policy CSV, server settings) — never the live ConfigMaps. Applies to any account/RBAC/config change on a self-managed controller.
+- **A newly-created API account/token 403s until the config sync completes.** If you generate an API token for a GitOps-managed service account before the account exists in the live config (still mid-Helm-sync), the token returns 403. Wait for self-sync to complete, confirm the account is present, then generate the token.
+- **Grant the narrowest role for the job.** A service that only reads controller/app status (e.g., a preview-readiness check) needs a read-only role, not a sync/deploy role. Scope the GitOps API account to exactly what it does.
+- **Runtime-only "poke" annotations cause persistent OutOfSync.** Annotations added imperatively to trigger controller behaviour — e.g., a `force-sync`/`reconcile` annotation on an ExternalSecret to make External Secrets Operator refresh — are not present in Git, so the GitOps controller reports the resource `OutOfSync` indefinitely. Remove the annotation once it has done its job: `kubectl annotate <kind> <name> -n <ns> <annotation-key>-`. Applies to any "poke the controller" annotation not stored in the source manifest.
 
 ## PodSecurity Alignment
 
@@ -75,6 +79,25 @@ The `namespaceSelector: kube-system` rule is also ineffective for kube-apiserver
 
 Using the wrong entity (or the wrong policy kind) results in silent policy drops. Symptom for apiserver-bound traffic: HTTPS calls hang until the client times out (typically 30s for Go HTTP defaults), then the upstream returns a generic error like `permission denied`. Test with `cilium monitor --type drop` to confirm the drop is at L3, or — if you can `exec` into the pod — try `wget --timeout=5 https://kubernetes.default.svc/healthz` and see if it `Terminated`s.
 
+**Worked example (recurring incident pattern):** A secrets backend pod calls `auth/kubernetes/login` against the apiserver and hangs for exactly 30 seconds before returning a generic `permission denied`. The first hypothesis is RBAC or token misconfiguration — neither is the cause. Standard `NetworkPolicy ipBlock` rules listing the control-plane subnet don't match because cluster nodes carry the `kube-apiserver` / `remote-node` Cilium identity, and `ipBlock` only matches IPs without an identity. Fix is `CiliumNetworkPolicy` with `toEntities: [kube-apiserver]` on TCP 6443. Same pattern affects every workload that hits `https://kubernetes.default.svc` — External Secrets Operator, custom controllers, anything doing TokenReview / SubjectAccessReview.
+
+## Every New K8s API Surface Needs Explicit RBAC in the Deploy Repo
+
+When a service starts touching a new K8s API resource (CRDs, custom controllers, ExternalSecrets, Jobs, Leases), add a namespace-scoped `Role` + `RoleBinding` to the deploy repo in the **same commit** as the code that touches the API.
+
+**Why this matters:**
+- Manual `kubectl apply` during development masks the gap because the local kubectl admin context bypasses RBAC. Production then 403s silently and stalls reconcile loops.
+- The GitOps reconciliation tries to apply the controller config but fails to read the underlying API; the symptom is a "permission denied" log line lost among thousands of other logs.
+- The RBAC commit lands AFTER the feature commit, leaving an interval where the deployed code is broken in any environment that doesn't have the dev's admin context.
+
+**Pattern:** every new API call requires either:
+1. A namespace-scoped `Role` with the specific verbs (`get`, `list`, `watch`, `create`, `update`, `patch`, `delete` — only the ones actually used) on the specific resource
+2. A `RoleBinding` to the service's `ServiceAccount`
+
+Bundle both into the same deploy-repo commit as the code change. A reviewer should be able to grep for any new K8s API call in the code diff and find the matching `Role` verbs in the manifest diff.
+
+**Common miss:** CRD custom resources need explicit `apiGroups` entries (e.g., `apiGroups: ["external-secrets.io"]`), not just `resources`. Forgetting the API group looks like a working Role definition but matches nothing.
+
 ## Kustomize Overlay `images:` Blocks Silently Override Base Tags
 
 Kustomize `images:` blocks in an overlay apply to the entire rendered manifest, including any images defined in `base/`. If the base defines `image: my-app:v1.0.0` and the overlay has an `images:` block targeting `my-app`, the overlay's `newTag` silently wins — even if you intended the base tag to remain. When deploying a new image version via Kustomize, always update the `images:` block in the overlay, not just the base manifest. If the overlay doesn't have an `images:` block, add one rather than editing the base tag directly.
@@ -102,6 +125,10 @@ ArgoCD with ServerSideApply sometimes fails to detect changes to ConfigMap `data
 
 CSI hot-plug of storage devices can fail silently — the VolumeAttachment object says `attached: true` but the device never appeared on the node. Pods get stuck in `ContainerCreating` with "device not found." Fix: delete the stale VolumeAttachment (`kubectl delete volumeattachment <name>`). The CSI driver recreates it and retries the attach.
 
+**QMP-timeout leading indicator for hypervisor CSI hotplug failures.** When a hypervisor-backed CSI driver (Proxmox, vSphere) returns `ControllerPublishVolume` "published" but the block device never appears in the guest (`/dev/disk/by-id/wwn-0x...` absent) and the pod stays in `ContainerCreating`, check the hypervisor logs for a QMP command timeout during the preceding unpublish (e.g. `qmp command 'query-pci' failed - got timeout`). This points to an unstable backing VM/host, not a Kubernetes bug. Short-term workaround: add `nodeAffinity` with `NotIn: <bad-node>` on the deployment to steer the workload off the flaky VM while the hypervisor-side QMP stability is investigated.
+
+**Targeted SCSI scan instead of full-bus rescan when hotplugged disks don't appear.** After a CSI hotplug, forcing a device rescan with a full-bus wildcard (`echo "- - -" > /sys/class/scsi_host/host*/scan`) hangs on some controllers. Use a targeted single-LUN scan instead: `echo "0 0 <lun>" > /sys/class/scsi_host/host<X>/scan` to probe a specific LUN without blocking the whole bus.
+
 ## Delete and Recreate ArgoCD Apps on Source Type Changes
 
 When changing an ArgoCD Application's source type (e.g., multi-source Helm to single-source Kustomize), the repo-server may serve cached manifests from the old configuration, and old Helm hook resources become ghost entries that block deletion via finalizers. Delete the Application entirely and let the root app recreate it rather than patching source types in-place.
@@ -181,3 +208,15 @@ Pair periodic reconciliation with an authenticated POST `/reconcile` endpoint so
 ## kubernetes-py CustomObjectsApi: SSA Requires a Dedicated ApiClient
 
 `CustomObjectsApi.patch_namespaced_custom_object(force=True)` fails with HTTP 422 on kubernetes-py v35 (`PatchOptions.meta.k8s.io is invalid: force: Forbidden: may not be specified for non-apply patch`). The default Content-Type is `application/merge-patch+json`; `force` is only valid on real server-side applies (`application/apply-patch+yaml`). The `_content_type` kwarg that older docs reference is not exposed in v35. Workaround: build a dedicated `ApiClient` and `set_default_header("Content-Type", "application/apply-patch+yaml")` on it; pass that client to a separate `CustomObjectsApi` used only for SSA patches. Reads/deletes use the default client (no body, default Content-Type harmless). The bug is silent in tests because mocks accept any kwargs — only a real apiserver round-trip surfaces it.
+
+## `kubectl debug node/<node>` Fails Under Enforced PodSecurity
+
+In namespaces/clusters enforcing PodSecurity `baseline` or `restricted`, `kubectl debug node/...` is rejected because its debug pod uses `hostPID` and `hostPath` (a baseline violation). Workaround: manually create a debug pod in a `privileged`-labelled namespace (e.g. the CSI driver's namespace) with `nodeSelector: kubernetes.io/hostname: <node>` and `securityContext.privileged: true`, rather than relying on `kubectl debug node`.
+
+## `kubectl apply` of a New Image Tag May Not Roll Pods
+
+Re-applying a Deployment with a bumped image tag does not reliably trigger a new rollout — the in-cluster image reference can stay cached, and `kubectl apply --force` does not fix it. When bumping an image version imperatively, `kubectl delete deployment <name>` before re-applying to guarantee a fresh pull. (In GitOps flows, prefer a digest pin or a template-hash annotation change; this delete-then-apply pattern is for imperative/dev workflows only.)
+
+## A Correct CiliumNetworkPolicy Egress Rule Won't Help If Ingress Is Gated by a Plain NetworkPolicy
+
+When a client pod times out reaching a service despite a correct `CiliumNetworkPolicy` egress rule on the client side, check the target's **ingress** policy — it may be a standard Kubernetes `NetworkPolicy` (not Cilium) that whitelists only specific source namespaces. Cilium and plain NetworkPolicy coexist and are additive; both directions must permit the flow. The ingress policy is often owned by the deploy repo, separate from the application and cluster-bootstrap repos, so grep there first. When debugging cross-namespace connectivity, enumerate both the client's egress rules and every ingress policy selecting the target pod.