best-practices/debugging.md

Debugging Methodology

Check Before You Act

• Before writing firewall/network rules, check actual routing (ip route get <dest>)
• Before running config management with variables, ensure values are real, not placeholders
• Before assuming a container has a shell, docker inspect it
• Before creating API tokens, research all required scopes upfront — iterating one scope at a time costs a push-debug cycle each

Routing and Networking

• Always run ip route get <dest> on the forwarding host first
• macvlan, Docker bridge, and other virtual interfaces mean the "obvious" physical interface is often wrong
• Test from both in-cluster and external perspectives

Full-Chain Testing

After wiring up any new service:

1. Test direct to backend (bypass all proxies)
2. Test through reverse proxy (bypass DNS)
3. Test end-to-end as a user would

Use curl --resolve to test specific paths without depending on DNS propagation.

Split-Horizon DNS Can Hide Bugs from Local Testing

When /etc/hosts or internal DNS points a public hostname at an internal IP, local curl bypasses the external path (VPS, CDN, cloud LB) and masks bugs that are only visible to external users. Always verify production behaviour through the actual public path:

• curl --resolve domain:443:<public-ip> https://domain/... to force the real external IP
• Or test from an external machine (phone on cellular, a cloud VM, etc.)

Applies to reverse-proxy routing bugs, HTTP/2 SAN mismatches, and TLS configuration that differs between internal and external ingress.

When Something Doesn't Sync/Apply

• Check resource exclusions in the GitOps controller immediately
• Check if the resource type requires special permissions or labels
• Check if ServerSideApply conflicts are preventing field changes
• Don't try workarounds before understanding the root cause

OIDC Integration Checklist

Before starting any OIDC integration, research:

1. What format is the sub claim (UUID? username?)
2. Which claims are in the ID token vs userinfo endpoint
3. How the consumer matches RBAC identities (groups? email? username?)

Log-First Diagnosis

• CrashLoopBackOff: check logs first. Error messages in pod logs usually point directly to the fix. Don't tweak configuration or security contexts blindly — kubectl logs <pod> first.
• Discriminate transient from persistent errors. CSI lock contention, etcd timeouts during first install, and brief connectivity blips are self-healing. Don't spend time debugging errors that resolve on retry. If you see retry/backoff patterns in logs, wait before intervening.
• Trust controller retry logic. CSI controllers, operators, and reconciliation loops have built-in retry. Transient failures during rapid provisioning are expected, not bugs.

Reproduce Before Fixing

When a bug is discovered or reported, do not start by trying to fix it. The first step is always to write a test that reproduces the failure:

1. Write a failing test. Capture the bug as a test case that demonstrates the broken behaviour. This forces you to understand the bug precisely — what input triggers it, what the wrong output is, and what the correct output should be.
2. Fix the bug in isolation. Use a subagent or a separate session to write the fix. The fixing agent gets the failing test as its success criterion — it's done when the test passes. This separation prevents the fixer from unconsciously weakening the test to match a broken implementation.
3. The test stays forever. The reproduction test becomes a permanent regression test. It proves the fix works and prevents the bug from returning.

This workflow has several advantages:

• Forces precise understanding. Writing a test means you know exactly what's broken, not just "it doesn't work."
• Prevents partial fixes. The test defines "done" objectively — the fix either passes or it doesn't.
• Parallelises work. While one agent fixes the bug, you can continue other work.
• Catches regressions. The test remains in the suite, guarding against the same class of failure.

# Step 1: Write the failing test FIRST
def test_regression_issue_427_empty_payload_crashes():
    """Bug #427: Empty payload causes unhandled TypeError in dispatcher.
    Should return a 400 validation error, not crash."""
    response = client.post("/dispatch", json={})
    assert response.status_code == 400  # Currently crashes with 500

# Step 2: Hand to a subagent/session: "Make this test pass without breaking others"

GIT_SSH_COMMAND Only Affects Git-Invoked SSH

GIT_SSH_COMMAND (e.g., ssh -o StrictHostKeyChecking=no) only applies when git invokes SSH internally (clone, push, fetch). Direct ssh calls — such as ssh -T git@host for connectivity testing — ignore it entirely. When working in containers or CI environments where host keys aren't pre-trusted, direct SSH commands need explicit flags: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null.

Pattern Mining Before Authoring

Before building a new service, component, or script, read existing patterns in the codebase first. This matches conventions on the first attempt and avoids rework on naming, structure, and integration points. Applies to K8s manifests, CI pipelines, skill authoring, and script structure.

Grep Your Own Docs

Known issues documented in CLAUDE.md or MEMORY.md but not applied to new scripts/configs waste debugging time. Search your own documentation before writing automation that touches areas with known gotchas.

Read the Spec Before Proposing a Workaround

When a mid-implementation design question arises in a subsystem that already has a written spec, read the spec before proposing a bridge hack. The correct design is often already documented. One session burned hours considering Phase 1 bearer-token bypasses before realising the spec already defined the Phase 2 design (bootstrap tokens + mTLS).

Rule: specs exist to prevent this — grep spec/ or re-read the relevant spec file before inventing a workaround.

Budget Infrastructure-Recovery Time After Disruptions

After any disruption (power cut, network outage, cluster reboot, registry migration), start the next session with an infrastructure health check before planning feature work. Snap confinement quirks, stuck Terminating pods, read-only filesystems, and unreachable Git remotes each consume meaningful time to diagnose. Budget recovery as an explicit first phase rather than discovering it mid-task.

Parallel Research Agents for Broad Topic Coverage

When researching a topic with multiple independent facets, dispatch parallel research agents (e.g., one per sub-topic or source type) rather than sequential queries. Scope each agent narrowly (e.g., jurisdiction, domain filter, doc set) to reduce noise and improve signal. Cheap when facets are independent; poor fit when later queries depend on earlier results.

API Token Scope Errors

When an API endpoint returns a permission/scope error, read the error response body before guessing. Many APIs (Gitea, GitHub, GitLab) explicitly state the required scope in the error message (e.g., required=[write:admin]). This is faster and more reliable than consulting documentation or iterating one scope at a time.

Minimal Container Images Have No Debug Tools

Distroless and single-binary containers (Garage, distroless Go images, etc.) have no shell, curl, wget, or other debug tools. kubectl exec commands will fail.

For HTTP checks: Use kubectl port-forward svc/<name> <local-port>:<svc-port> and run curl locally. For verification scripts: Don't assume exec-based checks will work. Design health checks around port-forward + local tools, or use Kubernetes-native probes.

Use Container Logs to Narrow the Failure Boundary

When a service appears down, check its logs for successful requests from other clients before assuming a total outage. If some clients are connecting successfully (e.g., HTTP/3 but not TCP, or internal but not external), the issue is narrower than "service is down." This distinction dramatically reduces debugging scope.

Structured JSON Logging from Application Entry Points

Web frameworks like uvicorn don't configure application-level loggers — only access logs appear by default. Named loggers have no handler unless logging.basicConfig() is called explicitly. This makes application logs invisible in production (K8s, Docker) with no error — just silence. Always call logging.basicConfig() with a structured format (JSON) early in application startup, before any getLogger() calls.

Verify DB Schema Matches Application Models After Every Deployment

After deploying a new version of an application that uses an ORM or schema migration tool, verify that the live database schema matches what the application expects. Common failure mode: a migration ran in dev/staging but not in production, or a new field was added to a model without a corresponding migration.

Quick check: run the application's schema validation command, or compare alembic current vs alembic head, or run SELECT column_name FROM information_schema.columns WHERE table_name='<table>' and diff against the model definition.

When to check: after every deployment that touches models or migrations — not just on explicit migration commits. An ORM auto-create (e.g., SQLAlchemy create_all) can silently succeed while leaving optional columns missing, causing subtle bugs rather than hard crashes.