distill: 48 cross-project best-practices from 2026-07 reflection sweep

Promotions from reflecting 21 projects' session logs (incl. agent-runtimes
122-log drain). Adds coverage across networking (eBPF VIP/VPN SNAT/VLAN
bridge/forward-auth preflight/ingress TLS), kubernetes (CSI hotplug/PodSecurity
debug/self-managed GitOps/runtime annotations), CI (dispatch tokens/runner
death/base image), git (CI-rebase/shallow reset/PR governance), python (async
session pool/httpx redirects/logging), TDD (AsyncMock/xfail lifecycle),
api-integration (SDK parse/token-scope 404/schema probing), plus docker,
scripting, debugging, security-architecture, secrets, react, octopus.

State: .distill-state.json refreshed with current HEADs + 5 newly-tracked projects.

networking.md

@@ -45,3 +45,43 @@ This breaks on-demand and "scale-to-zero" backends: the proxy has no route to th
 ## Cilium DNAT Resolves LB VIP Before NetworkPolicy Evaluation
 
 Cilium performs DNAT on LoadBalancer VIP traffic before evaluating NetworkPolicy. Traffic to a VIP is rewritten to a backend pod IP before the policy check. For egress to LoadBalancer services in CiliumNetworkPolicy, use `toEndpoints` targeting the backend pods (by namespace/label), not `toCIDR` targeting the VIP.
+
+## Don't Use ICMP/ping to Test eBPF LoadBalancer VIPs
+
+Cilium (and other eBPF LB datapaths) only program TCP/UDP for a LoadBalancer VIP; ICMP is not handled by the eBPF LB and falls through to the kernel, which can generate redirect loops. Symptom: `ping <lb-vip>` returns "TTL exceeded" or "connection refused" even from a host on the same network, tempting a false "the load balancer is broken" conclusion — while TCP/HTTPS to the same VIP works fine.
+
+**Rule:** test service-VIP reachability with `curl`/`nc` (TCP), never `ping`. A failed ping to an eBPF LB VIP is meaningless.
+
+## VPN Subnet/Exit Routing Needs Return-Path SNAT
+
+For any mesh/overlay VPN (Tailscale/Headscale, WireGuard subnet routers) that advertises LAN subnets or acts as an exit node, outbound reachability proves nothing on its own. LAN hosts have no route back to the VPN's client address range (e.g. Tailscale's `100.64.0.0/10` CGNAT range), so return traffic is silently dropped — the symptom is partial connectivity where some destinations answer and others don't.
+
+**Fix:** MASQUERADE/SNAT traffic sourced from the VPN client range out each physical LAN interface on the router node, and persist it (`netfilter-persistent` or equivalent). **When testing any VPN or subnet-routing change, verify both directions** — confirm the SNAT rule exists before concluding a route works. Outbound success alone is a false positive.
+
+## VLAN-Aware Bridge vs Raw Sub-Interface Coexistence (Proxmox/Linux bridges)
+
+On a host that runs BOTH a VLAN-aware bridge (`vmbr0` on a bond/uplink) AND raw `vlanN@bondX` sub-interfaces on the same uplink, the kernel delivers tagged frames for VLAN N to the raw sub-interface, NOT to the vlan-aware bridge's VLAN handling. A VM NIC attached as `vmbr0,tag=N` then receives nothing. Symptom: guest has the right IP/routes and the switch port is healthy, but there is zero L2 connectivity (no DHCP lease → APIPA `169.254.x`, gateway un-ARPable) — the guest config looks perfect while frames go nowhere.
+
+**Rules:**
+- A genuinely-tagged VLAN that also has a raw `vlanN` sub-interface must attach to its **dedicated named bridge** (whose port is `vlanN`), never `vmbr0,tag=N`.
+- A VM on the trunk's **native/untagged** VLAN must use untagged `vmbr0` (no `tag=`); a `tag=` NIC emits tagged frames that don't match the native VLAN and are dropped both directions.
+- For a multi-VLAN VM, one NIC per VLAN on per-VLAN bridges is more reliable than Proxmox `trunks=` (which silently fails to route on some `vmbr0` configs).
+
+**Diagnostic shortcut:** a guest with correct IP/routes but unreachable from a same-subnet host is an L2/VLAN problem, not guest config. Check which bridge the tap landed on (`ls /sys/class/net/<Bridge>/brif/`, `bridge vlan show dev <tap>`) and whether the VLAN is native(untagged) vs tagged on the uplink.
+
+## Forward-Auth Proxy Redirects Break CORS Preflight (302 on OPTIONS → ERR_INVALID_REDIRECT)
+
+A forward-auth layer (Authelia, oauth2-proxy, etc.) sitting in front of a backend intercepts *unauthenticated* requests and 302-redirects them to its login portal — including CORS OPTIONS preflight requests. The browser reports `ERR_INVALID_REDIRECT` (not `ERR_FAILED`), pointing at a 302 whose target is the auth portal.
+
+**This is an auth-bypass gap, not a CORS bug.** The backend's CORS middleware handles OPTIONS correctly once the auth layer stops intercepting. Do not touch the CORS config.
+
+- **Signature:** `ERR_INVALID_REDIRECT` + a 302 whose `Location` is the auth portal ⇒ bypass gap. A true CORS bug shows `ERR_FAILED` / missing `Access-Control-*` headers with no redirect.
+- **Fix:** add the endpoint path to the auth layer's bypass/allow list, then reload the auth layer.
+- **Bypass rules are path-scoped.** Every new backend endpoint needs its own bypass entry — one that works does not cover a sibling. Enumerate all public paths and confirm each is bypassed, including root-level paths a client library may call outside the expected prefix (e.g. `/schemas`, `/health`).
+
+## `tlsv1 alert internal error` = DNS Resolves to the Ingress, but No Route Matches the Host
+
+A domain resolves to the ingress IP, but `https://<domain>` aborts during the TLS handshake with `tlsv1 alert internal error`. Root cause: the ingress controller (Traefik, and equivalents) has no route matching `Host(<domain>)` — only related hosts exist (a different subdomain, a sibling domain). With no matching route there is no certificate to present, so the handshake aborts before any HTTP layer. This is the expected "nothing is served here" behaviour, not a cert or reverse-proxy misconfiguration.
+
+- Diagnostic: if `curl -v https://<domain>` fails at TLS but a known-good sibling host on the same ingress succeeds, suspect a missing route, not a broken cert.
+- Fix (if the site should be served): add an ingress route/IngressRoute matching that exact Host. Otherwise it is working as intended.