best-practices/scripting.md

# Scripting Conventions

## Structure

- All scripts live in `scripts/` and run from the repository root
- Scripts should be idempotent and safe to re-run
- Exit non-zero on failure so `&&` chains work naturally

## Verification Scripts

- Automated checks confirming milestone or feature outcomes
- Use colour output (green/red) for pass/fail indicators
- Should be non-destructive and environment-resilient
- Avoid needing sudo — test from the accessible side of a connection instead
- Check for default/insecure credentials and print remediation instructions on failure
- Use `curl --resolve` to bypass DNS/proxy layers when testing direct connectivity
- **Never use `set -e` in verification scripts.** A verify script's job is to run ALL checks and report a summary. `set -e` exits on the first failure, hiding remaining issues. Use explicit conditional checks and a pass/fail counter instead. Note: `((var++))` under `set -e` is a classic bash trap — pre-increment of 0 evaluates to falsy, triggering errexit. Use `var=$((var + 1))`.

## Automation Triggers

If you run the same 3+ commands in sequence more than once, it should become a script. Look for:
- Repeated command sequences in conversation history
- Steps requiring careful ordering
- Multi-step manual processes that are error-prone

## Error Handling by Tool Purpose

Not all scripts need the same error handling strategy:
- **Destructive scripts** (deploy, configure, delete) should use `set -euo pipefail` — fail fast on any error.
- **Reporting/read-only scripts** (status dashboards, aggregation, monitoring) should start without `set -e` — complex data collection from multiple sources is hard to debug under errexit. Use explicit conditional checks instead.
- **The choice depends on the tool's purpose.** A script that writes to production needs strict error handling. A script that reads from 10 sources and aggregates results needs resilience.

## Dryrun Mode

Every script that modifies state should support `--dryrun` / `-n`:
- Makes the script self-documenting about its side effects
- Enables safe testing and review before execution
- Enables test harnesses that verify output without executing changes
- Dryrun output should show exactly what would happen, not a summary

## Shell Gotchas

- `((PASS++))` fails under `set -e` when PASS=0 — the expression evaluates to 0 (false), triggering errexit. Use `PASS=$((PASS + 1))` instead.
- `set -e` silently terminates complex pipelines and subshells with no output — makes debugging extremely difficult. Also kills command substitutions that capture non-zero exit codes (e.g., `result=$(grep "pattern" file)` exits if grep finds nothing).
- `grep` interprets option-like strings (starting with `-`) as flags — use `--` terminator before patterns or input that may start with dashes.
- Always quote variables in conditionals and file paths
- Use `trap` for cleanup of temp files and credentials
- **Use `git diff --numstat` for binary file detection** instead of `file`. The `file` command is unreliable (marks shell scripts as "executable"), while `git diff --numstat` shows `-` for binary files using git's robust binary detection heuristics.
- **Use `cat -A` to diagnose invisible character issues.** Reveals non-printing characters like em dashes, zero-width spaces, and smart quotes that look identical to correct characters but break YAML parsers, config files, and frontmatter. Essential when a file looks correct but tooling rejects it.
- `grep` returns exit code 1 when no lines match — under `set -e`, this kills the script even when zero matches is a valid outcome. Append `|| true` to `grep` commands in pipelines where empty results are expected.
- `while read` in a pipeline creates a subshell — variables modified inside the loop (counters, accumulators) are lost after the loop ends. Use process substitution (`while read line; do ...; done < <(command)`) or here-strings to keep the loop in the current shell.
- **Order matters in sed/regex transformation pipelines.** Process more specific patterns before general ones. For example, if both `![[image.png]]` and `[[page]]` are valid patterns, process the image embed first — otherwise the general wikilink regex matches the inner `[[image.png]]` and the `!` prefix is left orphaned.

## JSON Construction in Scripts

Use Python (not shell) for constructing JSON payloads. Multi-line prompts with quotes, backticks, and special characters break shell-based JSON construction (printf/sed/heredocs). Python's `json.dump` handles escaping correctly every time. For scripts that need to construct and submit JSON payloads, write the construction logic in Python even if the rest of the script is bash. For long agent prompts, `--prompt-file` with temp files is cleaner than heredocs — writing prompts to `/tmp/*.md` files avoids shell escaping issues and enables review before submission.