best-practices/validation.md

Validation & Deployment

Validate Locally, Deploy Once

The single biggest time sink across projects is "deploy first, validate later." Real-world stats from a 9-milestone infrastructure project showed 50-60% of commits were fixes that could have been caught locally.

Always validate before pushing:

• helm template for Helm chart values
• kustomize build (or kubectl kustomize) for Kustomize apps
• kubectl apply --dry-run=server for K8s naming/schema issues
• docker run <app> validate-configuration for apps that support it (Authelia, Homepage, etc.)
• docker inspect for unfamiliar container images before writing init containers
• Lint/typecheck/test for application code

Batch fixes locally, push once. Each push-sync-crash-fix cycle wastes minutes and clutters Git history.

Test the Full Chain Immediately

After wiring up any new service or endpoint, test end-to-end from the user's perspective right away. Don't assume intermediate steps working means the whole chain works.

• curl --resolve domain:443:<ip> https://domain to test bypassing DNS/proxy layers
• Test from the actual consumer (not same-namespace test pods for network policies)
• Test DNS resolution after deploying FQDN-based policies

Healthy != reachable. A container showing healthy (pg_isready, HTTP 200, internal probe) can still have stale or broken external listeners — e.g., a TCP port bound to a namespace that was recreated, or an auth-backed endpoint accepting default credentials. Always probe the actual path users traverse: nc -zw2 <ip> <port> for TCP, curl the public URL, test auth APIs with known-bad and known-good creds. Probing the real path saves diagnosis time when the container's internal health signal diverges from external reachability.

Pre-Flight Checks

Before starting a deploy or automation phase:

• Verify SSH keys are loaded (ssh -T git@<host>)
• Confirm environment variables and credentials are available
• Check that the target environment is in the expected state
• Verify DNS records resolve as expected

Scripts That Change Config Must Self-Verify

After updating and restarting a service, the script should test that the change actually took effect (e.g., curl an API endpoint, check a config value). A "success" message without verification hides failures.

Check Container Image Runtime Requirements First

Before writing deployment manifests (StatefulSets, Deployments, init containers), check the image's runtime expectations: UID it runs as, writable directories it needs, filesystem layout. Use docker inspect or image documentation.

Modern images often run as non-root with specific writable directory requirements that aren't obvious from docs alone. Discovering these at deploy time wastes an entire push-crash-fix cycle per missed requirement.

Verify Counts and Summaries Mechanically

After editing specification or documentation files that include summary counts (e.g., "14 requirements"), verify them with grep or wc rather than counting manually. Manual counting of dozens of items is error-prone and produces incorrect summaries that erode trust in the documentation.

Test Pre-Commit Hooks Manually After Adding Dependencies

Run bash .githooks/pre-commit (or your hook path) manually after adding new dependencies or changing test imports. Hidden virtual environments (.venv/) that the hook discovers before system Python can cause ModuleNotFoundError at commit time even though tests pass from the terminal. Discovering hook failures during a real commit wastes debugging effort on environment issues rather than code issues. After adding a dependency, check all Python environments: find . -name "activate" -o -name "pytest" to discover venvs, and install into each.

Order Multi-Step Migrations Carefully

When performing multi-step changes on remote systems (port changes, firewall rules, service migrations), plan explicit ordering to avoid lockout:

1. Open the new path first (new port, new firewall rule)
2. Migrate the service to use the new path
3. Add redirects or backward-compatibility rules
4. Remove the old path

Doing all steps at once risks losing access if any step fails. Plan the ordering upfront, not mid-deploy.

Smoke-Test Service Images Locally Before CI

Before pushing Dockerfile or service configuration changes, run docker compose up locally with a real database and real service images. Unit tests cannot catch deployment-category bugs: Dockerfile CMD syntax, import path errors, env var prefix mismatches, URL encoding issues, factory patterns, startup ordering. A single local docker compose up catches these in seconds vs. the 3+ minute CI cycle per fix.

This applies to ANY iteration on K8s-deployed features, not just initial setup. Budget 2-3 fixup deploy cycles (~7-10 min each) for any feature first deployed to K8s. Infrastructure gaps between dev and prod always surface issues that unit tests cannot catch: missing COPY directives in Dockerfiles, missing RBAC permissions, wrong file permissions, read-only filesystem constraints. Accepting this cost upfront and smoke-testing locally before each push minimises the number of wasted cycles.

Stream Secrets from Source Files, Never from Context

When piping secrets into commands (base64 encoding, kubectl create secret, etc.), always stream from the source file in the same pipeline: cat /path/to/secret | base64. Never reconstruct a secret value from conversation context or memory — single-character typos in tokens cause authentication failures that are extremely difficult to diagnose. Save generated secrets to local_secrets/ immediately upon creation, then reference that file for all subsequent uses.

Verify Container Image Tags Before Writing References

Always verify that a container image tag exists before writing it into manifests, scripts, CI configs, or templates. Use docker manifest inspect <image>:<tag> or query the registry API directly. AI agents and cached knowledge frequently produce outdated or incorrect tag formats (e.g., ci-0.159.0 instead of 0.159.0, v0.6.0 when only v0.5.0 exists). A 5-second manifest inspect catches it immediately vs. a full push-sync-crash-fix cycle.

Front-Load Decision Questions with Recommended Defaults

When starting a milestone with multiple architectural choices, batch all decision questions into a single set with recommended defaults for each. This gets answers in one round and prevents mid-implementation direction changes. Example: "Domain? (recommend X) | Manifest location? (recommend Y) | Database approach? (recommend Z)" — all answered at once, zero backtracking.

Explore the Target Environment Before Planning

For infrastructure-heavy work, research the target environment's actual state before making design decisions. This means checking: what ingress controller is in use, how DNS resolves, what TLS strategy exists, what storage backends are available, what auth middleware is configured. Discovering these facts during planning (not implementation) prevents architectural surprises. In agent-orchestrated workflows, dedicated exploration agents that survey the target environment pay for themselves by eliminating implementation detours.

Two-Commit Pattern for In-Cluster Database Migrations

When migrating a K8s service to a new database backend: commit 1 = additive (deploy new database alongside existing setup), commit 2 = config switch (point the app at the new database). This avoids fighting GitOps controllers with selfHeal: true, which immediately revert manual scale-down operations. The pattern also provides a rollback path — if the config switch fails, revert commit 2.

Rehearse Database Migrations on a Disposable Instance

Before running any production database migration, rehearse the full path on a disposable instance (Docker container, test namespace). Migration tooling has undocumented quirks: missing commands, flag-dependent output formats, permission side effects. A 5-minute rehearsal catches these, vs. 30+ minutes debugging live.

Read the App's Entrypoint Script Before Configuring Env Vars

For containerized apps with custom env var conventions, read the entrypoint script once before writing any configuration. Many apps use prefix-based conventions that aren't fully documented. Discovering these through trial and error costs a push-restart-debug cycle per mistake. Five minutes reading the entrypoint saves 30-60 minutes of iterative fixing.

New Platform Service Deployment Checklist

When adding any new service to a platform/cluster, use a standard checklist: (1) application manifests, (2) secrets management, (3) GitOps application definition, (4) auth/SSO integration, (5) dashboard/UI registration, (6) ingress/routing rules, (7) reverse proxy config, (8) DNS records, (9) deploy automation, (10) full-chain test. A written checklist prevents the "forgot to add the DNS record" class of errors.

Evaluate Content Home Before Building

Before creating a new system, document type, or knowledge artifact, discuss where it belongs conceptually. Different content types have different lifecycles: accumulated learnings (MEMORY.md) vs authoritative maintained maps (SPEC, CLAUDE.md) vs behavioural contracts (spec files) vs current focus (CONTEXT.md). Picking the wrong home means future maintenance friction.

Add SSH-Authenticating User as Collaborator When Creating Repos via API

When creating Git repos via API token (which authenticates as one user) but pushing via SSH (which authenticates as a different user based on SSH key config), always add the SSH user as a collaborator with write access before the first push. A 403 on push after a successful API create is the symptom.

Use kubectl exec to Verify Deployed Container Contents

When confirming whether a fix is deployed, kubectl exec deploy/<name> -- ls <path> or kubectl exec deploy/<name> -- cat <path> is faster and more reliable than correlating CI build timestamps with commit times or checking registry tags.

Think Through K8s Constraints Before Coding Docker-First Solutions

Before implementing a feature that works in Docker, enumerate the K8s differences: read-only Secret volumes, root-owned files, no host-path mounts, separate pod filesystem, env var size limits. Design for both backends upfront. Planning for both environments from the start eliminates costly iteration cycles.

Inert-by-Default Strategy for Risky Feature Flags

When shipping a significant behaviour change (new persistence layer, new auth path, changed state machine), ship the code in an inert state first — hidden behind a feature flag that defaults to off. Let the code deploy and stabilise in production without activating the behaviour. Then flip the flag in a separate commit or config change. This decouples "did the deploy succeed?" from "did the feature work?" and gives a clean rollback path (revert the flag) without reverting code.

Pattern: ENABLE_NEW_FEATURE=false ships with the code. A follow-up change flips it to true. If the feature has issues, revert only the flag change.

Categorise Integration Failures by Root Cause Before Fixing

When an integration test run produces multiple failures, resist the urge to fix them one by one. Categorise first:

1. Infrastructure — missing env vars, wrong import paths, network timeouts
2. Interface drift — a shared data model changed and callers weren't updated
3. Test rot — test assumptions diverged from the current implementation
4. Logic bugs — actual code defects

Fixing by category is more efficient than fixing in order: all infrastructure failures have the same root fix, all interface drift failures share a data model change, etc. Categorising upfront also reveals the true scope — "3 logic bugs" is manageable; "14 failures across 4 categories" requires a different approach.

Use additionalProperties: false Everywhere in JSON Schemas

For schema-validated YAML/JSON configs, apply additionalProperties: false to every type: object — except intentionally open maps (explicitly documented). Typos in config keys (maxlength vs max_length) are the #1 config error source; without strict rejection, they silently pass validation and produce incorrect behaviour instead of a clear error. Helm covers this for charts; generalise the pattern to all schema-validated configs.

JSON Schema Draft Choice Affects Tuple Validation Syntax

Draft 2020-12 uses prefixItems for positional tuple validation and items: false to disallow extras. The items-as-array syntax (items: [{type: string}, {type: number}]) is Draft-04/07 only — using it under 2020-12 raises jsonschema.SchemaError. Always declare $schema explicitly in the schema root so readers and validators agree on the dialect.

Quote YAML Reserved-Word Keys

In YAML, unquoted reserved words as keys parse to their typed values: null: true becomes {None: True} in Python, not {"null": True}. The same applies to true, false, yes, no, on, off. Always quote reserved-word keys: "null": true. Add a schema-level regex or loader-level check to catch unquoted reserved-word keys before they silently break consumers — in one project this hit 45 occurrences across 4 manifests.

Coverage Thresholds vs Marker-Filtered Test Runs

pytest -m integration (or any marker-filtered subset) fails --cov-fail-under because most tests are deselected, dropping total coverage below the gate even when the filtered tests pass. Similarly, stub modules with raise NotImplementedError drop coverage — def/import lines still count. Use --no-cov for marker-filtered/smoke CI steps and run full-coverage as a separate step. When scaffolding stubs during milestones, temporarily lower the threshold and restore it after implementation.

Research Target Format and Feature Availability Before Building Artifacts

When integrating with an external platform (Platform Hub, Octopus, any product with its own schema/scope rules), spend the first hour reading the target format documentation (OCL syntax, API schema, scope rules) and verifying feature availability before committing to an approach. Two failure modes this prevents:

1. Wrong-format artifacts — e.g., 12 step templates created via API before discovering the target's process templates can't reference step templates cross-space. All the work was wasted.
2. Unavailable features — ask "what features of this product are actually available to me right now?" before planning. Assuming an unreleased feature (e.g., Project Templates) is available caused a plan to evolve three times mid-implementation.

Safe Persistence Pattern: Persist-on-Change, Rehydrate-at-Startup, Clean-Slate Fallback

For services that maintain in-memory state that should survive restarts (task queues, session state, model scores):

1. Persist on change — write state to disk immediately when it changes, not on a timer
2. Rehydrate at startup — read the persisted state file during service initialisation
3. Clean-slate fallback — if the state file is missing, corrupt, or incompatible, start with empty state and log a warning rather than crashing

This pattern handles pod restarts, rolling deploys, and upgrade scenarios without requiring an external database. The clean-slate fallback is critical — a startup crash due to a corrupt state file is worse than losing the state.