best-practices/scripting.md

Scripting Conventions

Structure

• All scripts live in scripts/ and run from the repository root
• Scripts should be idempotent and safe to re-run
• Exit non-zero on failure so && chains work naturally

Verification Scripts

• Automated checks confirming milestone or feature outcomes
• Use colour output (green/red) for pass/fail indicators
• Should be non-destructive and environment-resilient
• Avoid needing sudo — test from the accessible side of a connection instead
• Check for default/insecure credentials and print remediation instructions on failure
• Use curl --resolve to bypass DNS/proxy layers when testing direct connectivity
• Never use set -e in verification scripts. A verify script's job is to run ALL checks and report a summary. set -e exits on the first failure, hiding remaining issues. Use explicit conditional checks and a pass/fail counter instead. Note: ((var++)) under set -e is a classic bash trap — pre-increment of 0 evaluates to falsy, triggering errexit. Use var=$((var + 1)).

Automation Triggers

If you run the same 3+ commands in sequence more than once, it should become a script. Look for:

• Repeated command sequences in conversation history
• Steps requiring careful ordering
• Multi-step manual processes that are error-prone

Error Handling by Tool Purpose

Not all scripts need the same error handling strategy:

• Destructive scripts (deploy, configure, delete) should use set -euo pipefail — fail fast on any error.
• Reporting/read-only scripts (status dashboards, aggregation, monitoring) should start without set -e — complex data collection from multiple sources is hard to debug under errexit. Use explicit conditional checks instead.
• The choice depends on the tool's purpose. A script that writes to production needs strict error handling. A script that reads from 10 sources and aggregates results needs resilience.

Dryrun Mode

Every script that modifies state should support --dryrun / -n:

• Makes the script self-documenting about its side effects
• Enables safe testing and review before execution
• Enables test harnesses that verify output without executing changes
• Dryrun output should show exactly what would happen, not a summary

Shell Gotchas

• ((PASS++)) fails under set -e when PASS=0 — the expression evaluates to 0 (false), triggering errexit. Use PASS=$((PASS + 1)) instead.
• set -e silently terminates complex pipelines and subshells with no output — makes debugging extremely difficult. Also kills command substitutions that capture non-zero exit codes (e.g., result=$(grep "pattern" file) exits if grep finds nothing).
• grep interprets option-like strings (starting with -) as flags — use -- terminator before patterns or input that may start with dashes.
• Always quote variables in conditionals and file paths
• Use trap for cleanup of temp files and credentials
• Use git diff --numstat for binary file detection instead of file. The file command is unreliable (marks shell scripts as "executable"), while git diff --numstat shows - for binary files using git's robust binary detection heuristics.
• Use cat -A to diagnose invisible character issues. Reveals non-printing characters like em dashes, zero-width spaces, and smart quotes that look identical to correct characters but break YAML parsers, config files, and frontmatter. Essential when a file looks correct but tooling rejects it.
• grep returns exit code 1 when no lines match — under set -e, this kills the script even when zero matches is a valid outcome. Append || true to grep commands in pipelines where empty results are expected.
• while read in a pipeline creates a subshell — variables modified inside the loop (counters, accumulators) are lost after the loop ends. Use process substitution (while read line; do ...; done < <(command)) or here-strings to keep the loop in the current shell.
• Order matters in sed/regex transformation pipelines. Process more specific patterns before general ones. For example, if both ![[image.png]] and [[page]] are valid patterns, process the image embed first — otherwise the general wikilink regex matches the inner [[image.png]] and the ! prefix is left orphaned.

JSON Construction in Scripts

Use Python (not shell) for constructing JSON payloads. Multi-line prompts with quotes, backticks, and special characters break shell-based JSON construction (printf/sed/heredocs). Python's json.dump handles escaping correctly every time. For scripts that need to construct and submit JSON payloads, write the construction logic in Python even if the rest of the script is bash.