# Validation & Deployment ## Validate Locally, Deploy Once The single biggest time sink across projects is "deploy first, validate later." Real-world stats from a 9-milestone infrastructure project showed 50-60% of commits were fixes that could have been caught locally. **Always validate before pushing:** - `helm template` for Helm chart values - `kustomize build` (or `kubectl kustomize`) for Kustomize apps - `kubectl apply --dry-run=server` for K8s naming/schema issues - `docker run validate-configuration` for apps that support it (Authelia, Homepage, etc.) - `docker inspect` for unfamiliar container images before writing init containers - Lint/typecheck/test for application code **Batch fixes locally, push once.** Each push-sync-crash-fix cycle wastes minutes and clutters Git history. ## Test the Full Chain Immediately After wiring up any new service or endpoint, test end-to-end from the user's perspective right away. Don't assume intermediate steps working means the whole chain works. - `curl --resolve domain:443: https://domain` to test bypassing DNS/proxy layers - Test from the actual consumer (not same-namespace test pods for network policies) - Test DNS resolution after deploying FQDN-based policies ## Pre-Flight Checks Before starting a deploy or automation phase: - Verify SSH keys are loaded (`ssh -T git@`) - Confirm environment variables and credentials are available - Check that the target environment is in the expected state - Verify DNS records resolve as expected ## Scripts That Change Config Must Self-Verify After updating and restarting a service, the script should test that the change actually took effect (e.g., curl an API endpoint, check a config value). A "success" message without verification hides failures. ## Check Container Image Runtime Requirements First Before writing deployment manifests (StatefulSets, Deployments, init containers), check the image's runtime expectations: UID it runs as, writable directories it needs, filesystem layout. Use `docker inspect` or image documentation. Modern images often run as non-root with specific writable directory requirements that aren't obvious from docs alone. Discovering these at deploy time wastes an entire push-crash-fix cycle per missed requirement. ## Verify Counts and Summaries Mechanically After editing specification or documentation files that include summary counts (e.g., "14 requirements"), verify them with `grep` or `wc` rather than counting manually. Manual counting of dozens of items is error-prone and produces incorrect summaries that erode trust in the documentation. ## Test Pre-Commit Hooks Manually After Adding Dependencies Run `bash .githooks/pre-commit` (or your hook path) manually after adding new dependencies or changing test imports. Hidden virtual environments (`.venv/`) that the hook discovers before system Python can cause `ModuleNotFoundError` at commit time even though tests pass from the terminal. Discovering hook failures during a real commit wastes debugging effort on environment issues rather than code issues. After adding a dependency, check all Python environments: `find . -name "activate" -o -name "pytest"` to discover venvs, and install into each. ## Order Multi-Step Migrations Carefully When performing multi-step changes on remote systems (port changes, firewall rules, service migrations), plan explicit ordering to avoid lockout: 1. Open the new path first (new port, new firewall rule) 2. Migrate the service to use the new path 3. Add redirects or backward-compatibility rules 4. Remove the old path Doing all steps at once risks losing access if any step fails. Plan the ordering upfront, not mid-deploy. ## Smoke-Test Service Images Locally Before CI Before pushing Dockerfile or service configuration changes, run `docker compose up` locally with a real database and real service images. Unit tests cannot catch deployment-category bugs: Dockerfile CMD syntax, import path errors, env var prefix mismatches, URL encoding issues, factory patterns, startup ordering. A single local `docker compose up` catches these in seconds vs. the 3+ minute CI cycle per fix. **This applies to ANY iteration on K8s-deployed features, not just initial setup.** Budget 2-3 fixup deploy cycles (~7-10 min each) for any feature first deployed to K8s. Infrastructure gaps between dev and prod always surface issues that unit tests cannot catch: missing COPY directives in Dockerfiles, missing RBAC permissions, wrong file permissions, read-only filesystem constraints. Accepting this cost upfront and smoke-testing locally before each push minimises the number of wasted cycles. ## Stream Secrets from Source Files, Never from Context When piping secrets into commands (base64 encoding, kubectl create secret, etc.), always stream from the source file in the same pipeline: `cat /path/to/secret | base64`. Never reconstruct a secret value from conversation context or memory — single-character typos in tokens cause authentication failures that are extremely difficult to diagnose. Save generated secrets to `local_secrets/` immediately upon creation, then reference that file for all subsequent uses. ## Verify Container Image Tags Before Writing References Always verify that a container image tag exists before writing it into manifests, scripts, CI configs, or templates. Use `docker manifest inspect :` or query the registry API directly. AI agents and cached knowledge frequently produce outdated or incorrect tag formats (e.g., `ci-0.159.0` instead of `0.159.0`, `v0.6.0` when only `v0.5.0` exists). A 5-second manifest inspect catches it immediately vs. a full push-sync-crash-fix cycle. ## Front-Load Decision Questions with Recommended Defaults When starting a milestone with multiple architectural choices, batch all decision questions into a single set with recommended defaults for each. This gets answers in one round and prevents mid-implementation direction changes. Example: "Domain? (recommend X) | Manifest location? (recommend Y) | Database approach? (recommend Z)" — all answered at once, zero backtracking. ## Explore the Target Environment Before Planning For infrastructure-heavy work, research the target environment's actual state before making design decisions. This means checking: what ingress controller is in use, how DNS resolves, what TLS strategy exists, what storage backends are available, what auth middleware is configured. Discovering these facts during planning (not implementation) prevents architectural surprises. In agent-orchestrated workflows, dedicated exploration agents that survey the target environment pay for themselves by eliminating implementation detours. ## Two-Commit Pattern for In-Cluster Database Migrations When migrating a K8s service to a new database backend: commit 1 = additive (deploy new database alongside existing setup), commit 2 = config switch (point the app at the new database). This avoids fighting GitOps controllers with `selfHeal: true`, which immediately revert manual scale-down operations. The pattern also provides a rollback path — if the config switch fails, revert commit 2. ## Rehearse Database Migrations on a Disposable Instance Before running any production database migration, rehearse the full path on a disposable instance (Docker container, test namespace). Migration tooling has undocumented quirks: missing commands, flag-dependent output formats, permission side effects. A 5-minute rehearsal catches these, vs. 30+ minutes debugging live. ## Read the App's Entrypoint Script Before Configuring Env Vars For containerized apps with custom env var conventions, read the entrypoint script once before writing any configuration. Many apps use prefix-based conventions that aren't fully documented. Discovering these through trial and error costs a push-restart-debug cycle per mistake. Five minutes reading the entrypoint saves 30-60 minutes of iterative fixing. ## New Platform Service Deployment Checklist When adding any new service to a platform/cluster, use a standard checklist: (1) application manifests, (2) secrets management, (3) GitOps application definition, (4) auth/SSO integration, (5) dashboard/UI registration, (6) ingress/routing rules, (7) reverse proxy config, (8) DNS records, (9) deploy automation, (10) full-chain test. A written checklist prevents the "forgot to add the DNS record" class of errors. ## Evaluate Content Home Before Building Before creating a new system, document type, or knowledge artifact, discuss where it belongs conceptually. Different content types have different lifecycles: accumulated learnings (MEMORY.md) vs authoritative maintained maps (SPEC, CLAUDE.md) vs behavioural contracts (spec files) vs current focus (CONTEXT.md). Picking the wrong home means future maintenance friction. ## Add SSH-Authenticating User as Collaborator When Creating Repos via API When creating Git repos via API token (which authenticates as one user) but pushing via SSH (which authenticates as a different user based on SSH key config), always add the SSH user as a collaborator with write access before the first push. A 403 on push after a successful API create is the symptom. ## Use kubectl exec to Verify Deployed Container Contents When confirming whether a fix is deployed, `kubectl exec deploy/ -- ls ` or `kubectl exec deploy/ -- cat ` is faster and more reliable than correlating CI build timestamps with commit times or checking registry tags. ## Think Through K8s Constraints Before Coding Docker-First Solutions Before implementing a feature that works in Docker, enumerate the K8s differences: read-only Secret volumes, root-owned files, no host-path mounts, separate pod filesystem, env var size limits. Design for both backends upfront. Planning for both environments from the start eliminates costly iteration cycles.