best-practices/validation.md

# Validation & Deployment

## Validate Locally, Deploy Once

The single biggest time sink across projects is "deploy first, validate later." Real-world stats from a 9-milestone infrastructure project showed 50-60% of commits were fixes that could have been caught locally.

**Always validate before pushing:**
- `helm template` for Helm chart values
- `kustomize build` (or `kubectl kustomize`) for Kustomize apps
- `kubectl apply --dry-run=server` for K8s naming/schema issues
- `docker run <app> validate-configuration` for apps that support it (Authelia, Homepage, etc.)
- `docker inspect` for unfamiliar container images before writing init containers
- Lint/typecheck/test for application code

**Batch fixes locally, push once.** Each push-sync-crash-fix cycle wastes minutes and clutters Git history.

## Test the Full Chain Immediately

After wiring up any new service or endpoint, test end-to-end from the user's perspective right away. Don't assume intermediate steps working means the whole chain works.

- `curl --resolve domain:443:<ip> https://domain` to test bypassing DNS/proxy layers
- Test from the actual consumer (not same-namespace test pods for network policies)
- Test DNS resolution after deploying FQDN-based policies

## Pre-Flight Checks

Before starting a deploy or automation phase:
- Verify SSH keys are loaded (`ssh -T git@<host>`)
- Confirm environment variables and credentials are available
- Check that the target environment is in the expected state
- Verify DNS records resolve as expected

## Scripts That Change Config Must Self-Verify

After updating and restarting a service, the script should test that the change actually took effect (e.g., curl an API endpoint, check a config value). A "success" message without verification hides failures.

## Check Container Image Runtime Requirements First

Before writing deployment manifests (StatefulSets, Deployments, init containers), check the image's runtime expectations: UID it runs as, writable directories it needs, filesystem layout. Use `docker inspect` or image documentation.

Modern images often run as non-root with specific writable directory requirements that aren't obvious from docs alone. Discovering these at deploy time wastes an entire push-crash-fix cycle per missed requirement.

## Verify Counts and Summaries Mechanically

After editing specification or documentation files that include summary counts (e.g., "14 requirements"), verify them with `grep` or `wc` rather than counting manually. Manual counting of dozens of items is error-prone and produces incorrect summaries that erode trust in the documentation.

## Test Pre-Commit Hooks Manually After Adding Dependencies

Run `bash .githooks/pre-commit` (or your hook path) manually after adding new dependencies or changing test imports. Hidden virtual environments (`.venv/`) that the hook discovers before system Python can cause `ModuleNotFoundError` at commit time even though tests pass from the terminal. Discovering hook failures during a real commit wastes debugging effort on environment issues rather than code issues. After adding a dependency, check all Python environments: `find . -name "activate" -o -name "pytest"` to discover venvs, and install into each.

## Order Multi-Step Migrations Carefully

When performing multi-step changes on remote systems (port changes, firewall rules, service migrations), plan explicit ordering to avoid lockout:

1. Open the new path first (new port, new firewall rule)
2. Migrate the service to use the new path
3. Add redirects or backward-compatibility rules
4. Remove the old path

Doing all steps at once risks losing access if any step fails. Plan the ordering upfront, not mid-deploy.

## Smoke-Test Service Images Locally Before CI

Before pushing Dockerfile or service configuration changes, run `docker compose up` locally with a real database and real service images. Unit tests cannot catch deployment-category bugs: Dockerfile CMD syntax, import path errors, env var prefix mismatches, URL encoding issues, factory patterns, startup ordering. A single local `docker compose up` catches these in seconds vs. the 3+ minute CI cycle per fix.

## Stream Secrets from Source Files, Never from Context

When piping secrets into commands (base64 encoding, kubectl create secret, etc.), always stream from the source file in the same pipeline: `cat /path/to/secret | base64`. Never reconstruct a secret value from conversation context or memory — single-character typos in tokens cause authentication failures that are extremely difficult to diagnose. Save generated secrets to `local_secrets/` immediately upon creation, then reference that file for all subsequent uses.

## Verify Container Image Tags Before Writing References

Always verify that a container image tag exists before writing it into manifests, scripts, CI configs, or templates. Use `docker manifest inspect <image>:<tag>` or query the registry API directly. AI agents and cached knowledge frequently produce outdated or incorrect tag formats (e.g., `ci-0.159.0` instead of `0.159.0`, `v0.6.0` when only `v0.5.0` exists). A 5-second manifest inspect catches it immediately vs. a full push-sync-crash-fix cycle.

## Front-Load Decision Questions with Recommended Defaults

When starting a milestone with multiple architectural choices, batch all decision questions into a single set with recommended defaults for each. This gets answers in one round and prevents mid-implementation direction changes. Example: "Domain? (recommend X) | Manifest location? (recommend Y) | Database approach? (recommend Z)" — all answered at once, zero backtracking.

## Explore the Target Environment Before Planning

For infrastructure-heavy work, research the target environment's actual state before making design decisions. This means checking: what ingress controller is in use, how DNS resolves, what TLS strategy exists, what storage backends are available, what auth middleware is configured. Discovering these facts during planning (not implementation) prevents architectural surprises. In agent-orchestrated workflows, dedicated exploration agents that survey the target environment pay for themselves by eliminating implementation detours.