best-practices/scripting.md

# Scripting Conventions

## Structure

- All scripts live in `scripts/` and run from the repository root
- Scripts should be idempotent and safe to re-run
- Exit non-zero on failure so `&&` chains work naturally

## Verification Scripts

- Automated checks confirming milestone or feature outcomes
- Use colour output (green/red) for pass/fail indicators
- Should be non-destructive and environment-resilient
- Avoid needing sudo — test from the accessible side of a connection instead
- Check for default/insecure credentials and print remediation instructions on failure
- Use `curl --resolve` to bypass DNS/proxy layers when testing direct connectivity
- **Never use `set -e` in verification scripts.** A verify script's job is to run ALL checks and report a summary. `set -e` exits on the first failure, hiding remaining issues. Use explicit conditional checks and a pass/fail counter instead. Note: `((var++))` under `set -e` is a classic bash trap — pre-increment of 0 evaluates to falsy, triggering errexit. Use `var=$((var + 1))`.

## Automation Triggers

If you run the same 3+ commands in sequence more than once, it should become a script. Look for:
- Repeated command sequences in conversation history
- Steps requiring careful ordering
- Multi-step manual processes that are error-prone

## Error Handling by Tool Purpose

Not all scripts need the same error handling strategy:
- **Destructive scripts** (deploy, configure, delete) should use `set -euo pipefail` — fail fast on any error.
- **Reporting/read-only scripts** (status dashboards, aggregation, monitoring) should start without `set -e` — complex data collection from multiple sources is hard to debug under errexit. Use explicit conditional checks instead.
- **The choice depends on the tool's purpose.** A script that writes to production needs strict error handling. A script that reads from 10 sources and aggregates results needs resilience.

## Dryrun Mode

Every script that modifies state should support `--dryrun` / `-n`:
- Makes the script self-documenting about its side effects
- Enables safe testing and review before execution
- Enables test harnesses that verify output without executing changes
- Dryrun output should show exactly what would happen, not a summary

## Shell Gotchas

- `((PASS++))` fails under `set -e` when PASS=0 — the expression evaluates to 0 (false), triggering errexit. Use `PASS=$((PASS + 1))` instead.
- `set -e` silently terminates complex pipelines and subshells with no output — makes debugging extremely difficult. Also kills command substitutions that capture non-zero exit codes (e.g., `result=$(grep "pattern" file)` exits if grep finds nothing).
- `grep` interprets option-like strings (starting with `-`) as flags — use `--` terminator before patterns or input that may start with dashes.
- Always quote variables in conditionals and file paths
- Use `trap` for cleanup of temp files and credentials
- **Use `git diff --numstat` for binary file detection** instead of `file`. The `file` command is unreliable (marks shell scripts as "executable"), while `git diff --numstat` shows `-` for binary files using git's robust binary detection heuristics.
- **Use `cat -A` to diagnose invisible character issues.** Reveals non-printing characters like em dashes, zero-width spaces, and smart quotes that look identical to correct characters but break YAML parsers, config files, and frontmatter. Essential when a file looks correct but tooling rejects it.
- `grep` returns exit code 1 when no lines match — under `set -e`, this kills the script even when zero matches is a valid outcome. Append `|| true` to `grep` commands in pipelines where empty results are expected.
- `while read` in a pipeline creates a subshell — variables modified inside the loop (counters, accumulators) are lost after the loop ends. Use process substitution (`while read line; do ...; done < <(command)`) or here-strings to keep the loop in the current shell.
- **Order matters in sed/regex transformation pipelines.** Process more specific patterns before general ones. For example, if both `![[image.png]]` and `[[page]]` are valid patterns, process the image embed first — otherwise the general wikilink regex matches the inner `[[image.png]]` and the `!` prefix is left orphaned.

## JSON Construction in Scripts

Use Python (not shell) for constructing JSON payloads. Multi-line prompts with quotes, backticks, and special characters break shell-based JSON construction (printf/sed/heredocs). Python's `json.dump` handles escaping correctly every time. For scripts that need to construct and submit JSON payloads, write the construction logic in Python even if the rest of the script is bash.