distill: 48 cross-project best-practices from 2026-07 reflection sweep

Promotions from reflecting 21 projects' session logs (incl. agent-runtimes
122-log drain). Adds coverage across networking (eBPF VIP/VPN SNAT/VLAN
bridge/forward-auth preflight/ingress TLS), kubernetes (CSI hotplug/PodSecurity
debug/self-managed GitOps/runtime annotations), CI (dispatch tokens/runner
death/base image), git (CI-rebase/shallow reset/PR governance), python (async
session pool/httpx redirects/logging), TDD (AsyncMock/xfail lifecycle),
api-integration (SDK parse/token-scope 404/schema probing), plus docker,
scripting, debugging, security-architecture, secrets, react, octopus.

State: .distill-state.json refreshed with current HEADs + 5 newly-tracked projects.

security-architecture.md

@@ -78,6 +78,17 @@ Some scenarios genuinely require client-side credentials (e.g., direct S3 upload
 
 **Right:** CMS sits behind Authelia (MFA). A proxy service intercepts the OAuth flow, issues a proxy session token (containing only the user's identity), and forwards all API calls to Gitea using a per-user server-side Gitea token. The Gitea token never leaves the server.
 
+### Browser-Side OAuth Popup: COOP Severs `window.opener`
+
+When a client-side app (CMS admin UI, SPA) runs an OAuth popup flow and the callback redirects through a different origin (e.g., an auth proxy on another subdomain), the Cross-Origin-Opener-Policy (COOP) mismatch severs `window.opener`. The popup can no longer `postMessage` its result back to the opener, and the parent's popup-detection logic (commonly `window.opener?.origin === location.origin && window.name === 'auth'`) silently fails — producing zero token exchanges and, if the callback falls through to re-init the login flow, an infinite popup loop.
+
+Mitigations:
+- Set `COOP: same-origin-allow-popups` on the page that opens the popup; keep COOP off the cross-origin callback/proxy page.
+- Handle the OAuth callback explicitly: when `?code=&state=` are present, always `return` after processing — never fall through to auto-init/auto-click, which is what amplifies a broken flow into an infinite loop.
+- When `window.opener` is severed, deliver the auth result via a `BroadcastChannel` fallback. Re-dispatch it as a synthetic `MessageEvent` so the framework's existing handler fires without patching third-party (often minified) source.
+
+This is the browser-mechanics complement to the server-boundary rule above: even with a correct backend proxy, the popup handshake itself breaks on COOP. Check security headers (COOP/COEP) early when a login popup silently never completes — before investigating sessions, cookies, or backend state.
+
 ## Multi-Tenant Isolation Guard on Outbound Writes
 
 When a service writes into per-tenant destinations — customer Slack channels, per-customer boards, per-org webhooks, tenant-scoped buckets — route every outbound write through a single **IsolationGuard** module.
@@ -111,3 +122,37 @@ CallSite ──→ IsolationGuard.send(tenant_id, destination, payload)
 ```
 
 No call site should import the underlying transport directly. Lint or grep for direct imports as a CI check.
+
+## File-Only Secret Delivery for Long-Lived Processes
+
+For any container or process running longer than a few minutes, deliver secrets via tmpfs-backed file mounts (mode 0400, owned by root or the secrets operator), never via environment variables. Env vars leak into:
+
+- `ps -auxe` and `/proc/<pid>/environ` for any process in the PID namespace
+- Log aggregators (anything that captures the process tree)
+- Accidentally-echoed error output (`echo "config: $DATABASE_URL"` leaks the secret to stdout)
+- Child processes that inherit the environment by default
+
+Env-var delivery is acceptable only for short-lived (<60s) ephemeral tasks where the exposure window is bounded and no other workloads share the PID namespace.
+
+**Defense-in-depth pattern (External Secrets Operator + init):**
+
+1. ESO mounts the Secret at `/var/run/secrets/...` as mode 0400, owned by root (controller's service account)
+2. The pod's init container stages a 0600 copy to a path owned by the application's UID
+3. The application reads from disk at invocation time, not at startup — so rotation lands without restart
+
+**Two-layer scoping:** the ESO mount is the source-of-truth; the init copy is the application-readable view. A compromised application can read its copy but cannot read the ESO source. A compromised init container does not survive past startup.
+
+## Server Never Holds the Private Key — CSR-Based Flow for Server-Issued Identity
+
+When a server issues identity material to a client (mTLS cert, agent signing key, per-request token), design the protocol so the server signs a CSR rather than generating the keypair.
+
+**Rationale:** Most managed-runtime languages (CPython, Java, Go via reflection) cannot reliably zeroise private bytes. CPython strings are immutable; there is no `memset` equivalent; GC residue, copy-on-write, and string interning all leave secret material in addressable memory after "deletion." Once the server has held the private key in process memory, you cannot prove it is gone.
+
+**Flow:**
+
+1. Client generates the keypair locally via the standard `cryptography` primitives
+2. Client builds a CSR including its identity claims (SAN URI, subject CN)
+3. Server receives the CSR over an authenticated transport, validates the identity claims, signs the CSR, returns the certificate
+4. The server never sees the private key
+
+Applies to any mTLS issuance, ACL/agent-signing keys, per-request token issuance, and any "server hands the client an identity" protocol. The pattern survives memory-dump forensic analysis of the server.