# Validation & Deployment

## Validate Locally, Deploy Once

The single biggest time sink across projects is "deploy first, validate later." Real-world stats from a 9-milestone infrastructure project showed 50-60% of commits were fixes that could have been caught locally.

**Always validate before pushing:**
- `helm template` for Helm chart values
- `kustomize build` (or `kubectl kustomize`) for Kustomize apps
- `kubectl apply --dry-run=server` for K8s naming/schema issues
- `docker run <app> validate-configuration` for apps that support it (Authelia, Homepage, etc.)
- `docker inspect` for unfamiliar container images before writing init containers
- Lint/typecheck/test for application code

**Batch fixes locally, push once.** Each push-sync-crash-fix cycle wastes minutes and clutters Git history.

## Test the Full Chain Immediately

After wiring up any new service or endpoint, test end-to-end from the user's perspective right away. Don't assume intermediate steps working means the whole chain works.

- `curl --resolve domain:443:<ip> https://domain` to test bypassing DNS/proxy layers
- Test from the actual consumer (not same-namespace test pods for network policies)
- Test DNS resolution after deploying FQDN-based policies

## Pre-Flight Checks

Before starting a deploy or automation phase:
- Verify SSH keys are loaded (`ssh -T git@<host>`)
- Confirm environment variables and credentials are available
- Check that the target environment is in the expected state
- Verify DNS records resolve as expected

## Scripts That Change Config Must Self-Verify

After updating and restarting a service, the script should test that the change actually took effect (e.g., curl an API endpoint, check a config value). A "success" message without verification hides failures.

## Check Container Image Runtime Requirements First

Before writing deployment manifests (StatefulSets, Deployments, init containers), check the image's runtime expectations: UID it runs as, writable directories it needs, filesystem layout. Use `docker inspect` or image documentation.

Modern images often run as non-root with specific writable directory requirements that aren't obvious from docs alone. Discovering these at deploy time wastes an entire push-crash-fix cycle per missed requirement.

## Verify Counts and Summaries Mechanically

After editing specification or documentation files that include summary counts (e.g., "14 requirements"), verify them with `grep` or `wc` rather than counting manually. Manual counting of dozens of items is error-prone and produces incorrect summaries that erode trust in the documentation.

## Test Pre-Commit Hooks Manually After Adding Dependencies

Run `bash .githooks/pre-commit` (or your hook path) manually after adding new dependencies or changing test imports. Hidden virtual environments (`.venv/`) that the hook discovers before system Python can cause `ModuleNotFoundError` at commit time even though tests pass from the terminal. Discovering hook failures during a real commit wastes debugging effort on environment issues rather than code issues. After adding a dependency, check all Python environments: `find . -name "activate" -o -name "pytest"` to discover venvs, and install into each.

## Order Multi-Step Migrations Carefully

When performing multi-step changes on remote systems (port changes, firewall rules, service migrations), plan explicit ordering to avoid lockout:

1. Open the new path first (new port, new firewall rule)
2. Migrate the service to use the new path
3. Add redirects or backward-compatibility rules
4. Remove the old path

Doing all steps at once risks losing access if any step fails. Plan the ordering upfront, not mid-deploy.