distill: 48 cross-project best-practices from 2026-07 reflection sweep

Promotions from reflecting 21 projects' session logs (incl. agent-runtimes
122-log drain). Adds coverage across networking (eBPF VIP/VPN SNAT/VLAN
bridge/forward-auth preflight/ingress TLS), kubernetes (CSI hotplug/PodSecurity
debug/self-managed GitOps/runtime annotations), CI (dispatch tokens/runner
death/base image), git (CI-rebase/shallow reset/PR governance), python (async
session pool/httpx redirects/logging), TDD (AsyncMock/xfail lifecycle),
api-integration (SDK parse/token-scope 404/schema probing), plus docker,
scripting, debugging, security-architecture, secrets, react, octopus.

State: .distill-state.json refreshed with current HEADs + 5 newly-tracked projects.
This commit is contained in:
Paul O'Reilly
2026-07-02 15:57:42 +12:00
parent 5e67cbcfbb
commit 7e348f5ee3
16 changed files with 577 additions and 57 deletions

View File

@@ -164,3 +164,36 @@ For services that maintain in-memory state that should survive restarts (task qu
3. **Clean-slate fallback** — if the state file is missing, corrupt, or incompatible, start with empty state and log a warning rather than crashing
This pattern handles pod restarts, rolling deploys, and upgrade scenarios without requiring an external database. The clean-slate fallback is critical — a startup crash due to a corrupt state file is worse than losing the state.
## Phase-Done Gate: Wired, Integrated, AND Exercised End-to-End
A subsystem is not "complete" because its unit tests pass. A phase ships when ALL THREE hold:
1. **Unit tests pass** — the new code is exercised in isolation
2. **Wired into the application lifespan and covered by integration tests** — the new code is reachable from production startup paths (imported by `main.py`, registered in lifespan handlers, included in the DI graph) AND integration tests confirm it
3. **Exercised in one full end-to-end production-equivalent run** — the side effect is observed: resource materialised, event delivered, persistence verified, callback fired
The dominant failure mode is two-of-three: code exists and passes unit tests but was never imported from `main.py`; or wired up but never exercised live. Both silently look "done." Walking through these three checkpoints before stamping a phase complete catches the gap before it becomes a production incident. Generalises to any service with lifespan startup, scheduled drainers, background workers, or out-of-band reconciliation loops.
## Verify Agent-Chosen Version Pins Against the Live Registry
When an AI agent or research tool produces version pins (`react@18.2.0`, `httpx==0.28.0`, `<image>:1.4.2`), validate every pin against the live registry before committing — `npm view <pkg> version`, `pip index versions <pkg>`, `docker manifest inspect <image>:<tag>`, `helm search repo <chart>`. Agents (especially those without web access, including all container-bound agents) fabricate plausible-looking version numbers from training data with measured ~30% inaccuracy.
A 5-second registry check prevents a multi-cycle push-debug-fix loop. Same rule applies to release dates, "actively maintained" claims, and any other version-shaped assertion an agent produces.
Generalises the existing "Verify Container Image Tags Before Writing References" rule to cover npm/PyPI/Helm-chart pins.
## Restart Services That Read Config Only at Startup After a Config Change
Many services (Authelia, and any app that loads config at boot) read their config file exactly once at startup. Editing the mounted config — ConfigMap, file, env source — has **no effect until the process restarts**. A config change that "did nothing" almost always means the process was never restarted.
- **In GitOps/K8s:** a ConfigMap edit does not restart the consuming pod. Force it with a `kubectl.kubernetes.io/restartedAt` pod-template annotation (syncs through GitOps) or `kubectl rollout restart deployment/<name>`. The annotation is preferred when the change must land purely through GitOps.
- Before diagnosing a "config change had no effect" symptom, confirm the process actually restarted. Check pod age / start time, not just that the ConfigMap contains the new value.
## Long-Lived Compose/Service Stacks Must Survive Reboots
A Docker Compose stack (or any long-lived service) started manually does not come back after a power cut or reboot. Any long-lived stack on a machine that can lose power must be wired into the boot path — a systemd unit or an Ansible/config-management-managed equivalent — as part of bringing it up, not deferred to "later." Verify it restarts unattended (e.g. reboot the host, or `systemctl restart`) rather than assuming the unit works.
## Pause and Update the Plan When Verification Changes a Mechanism
When an early verification step (schema check, API probe, capability survey) reveals the real system cannot express a planned mechanism — e.g. native recurrence can't do the cadence you designed for, or a linking primitive doesn't exist — stop before building the affected milestone and revise the plan's mechanism table first. Building against a disproven assumption and reworking it later is far more expensive than a mid-plan pause. This is the corrective action that pairs with front-loaded capability verification.