claude-foundations/best-practices/validation.md

Validation & Deployment

Validate Locally, Deploy Once

The single biggest time sink across projects is "deploy first, validate later." Real-world stats from a 9-milestone infrastructure project showed 50-60% of commits were fixes that could have been caught locally.

Always validate before pushing:

• helm template for Helm chart values
• kustomize build (or kubectl kustomize) for Kustomize apps
• kubectl apply --dry-run=server for K8s naming/schema issues
• docker run <app> validate-configuration for apps that support it (Authelia, Homepage, etc.)
• docker inspect for unfamiliar container images before writing init containers
• Lint/typecheck/test for application code

Batch fixes locally, push once. Each push-sync-crash-fix cycle wastes minutes and clutters Git history.

Test the Full Chain Immediately

After wiring up any new service or endpoint, test end-to-end from the user's perspective right away. Don't assume intermediate steps working means the whole chain works.

• curl --resolve domain:443:<ip> https://domain to test bypassing DNS/proxy layers
• Test from the actual consumer (not same-namespace test pods for network policies)
• Test DNS resolution after deploying FQDN-based policies

Pre-Flight Checks

Before starting a deploy or automation phase:

• Verify SSH keys are loaded (ssh -T git@<host>)
• Confirm environment variables and credentials are available
• Check that the target environment is in the expected state
• Verify DNS records resolve as expected

Scripts That Change Config Must Self-Verify

After updating and restarting a service, the script should test that the change actually took effect (e.g., curl an API endpoint, check a config value). A "success" message without verification hides failures.

Check Container Image Runtime Requirements First

Before writing deployment manifests (StatefulSets, Deployments, init containers), check the image's runtime expectations: UID it runs as, writable directories it needs, filesystem layout. Use docker inspect or image documentation.

Modern images often run as non-root with specific writable directory requirements that aren't obvious from docs alone. Discovering these at deploy time wastes an entire push-crash-fix cycle per missed requirement.

Verify Counts and Summaries Mechanically

After editing specification or documentation files that include summary counts (e.g., "14 requirements"), verify them with grep or wc rather than counting manually. Manual counting of dozens of items is error-prone and produces incorrect summaries that erode trust in the documentation.

Test Pre-Commit Hooks Manually After Adding Dependencies

Run bash .githooks/pre-commit (or your hook path) manually after adding new dependencies or changing test imports. Hidden virtual environments (.venv/) that the hook discovers before system Python can cause ModuleNotFoundError at commit time even though tests pass from the terminal. Discovering hook failures during a real commit wastes debugging effort on environment issues rather than code issues. After adding a dependency, check all Python environments: find . -name "activate" -o -name "pytest" to discover venvs, and install into each.

Order Multi-Step Migrations Carefully

When performing multi-step changes on remote systems (port changes, firewall rules, service migrations), plan explicit ordering to avoid lockout:

1. Open the new path first (new port, new firewall rule)
2. Migrate the service to use the new path
3. Add redirects or backward-compatibility rules
4. Remove the old path

Doing all steps at once risks losing access if any step fails. Plan the ordering upfront, not mid-deploy.