best-practices/security-architecture.md

Security Architecture

The Server Boundary Rule

No server-side credential may cross the server boundary to the client. Ever.

This is a hard line, not a guideline. The only credentials that cross the boundary between client and server are the client's own identity credentials (MFA tokens, login passwords, OIDC tokens, etc.) — and these flow from client to server, never the reverse.

What this means in practice

• API tokens stay server-side. If a browser-based application needs to call a third-party API (Gitea, S3, database, etc.), it calls a backend proxy that holds the token. The token never appears in JavaScript, localStorage, cookies, or any client-accessible storage.
• Service account credentials stay server-side. Database passwords, S3 access keys, webhook secrets, SMTP credentials — these are mounted into server-side containers via Kubernetes Secrets or environment variables and never exposed to clients.
• OAuth tokens for third-party services stay server-side. If a user authenticates with Service A and the application needs to call Service B on their behalf, the application's backend holds the Service B credentials. The client only ever sees its own session token with the application.
• Per-user tokens mapped server-side. When individual users need distinct third-party access (e.g., per-user Gitea tokens for audit trails), the mapping from user identity to their token lives on the server. The client authenticates with its own identity (e.g., via Authelia MFA), and the server looks up the appropriate third-party token.

The identity exception

The only credentials that legitimately cross from client to server:

• Username + password — the user's own login credentials
• MFA tokens — TOTP codes, WebAuthn assertions, security key responses
• OIDC/OAuth tokens — tokens that represent the user's identity with the application itself (not with third-party services)
• Session cookies/JWTs — issued by the server to represent an authenticated session

These all share the property: they are the user's own identity, flowing from client to server for authentication purposes.

Architecture pattern: proxy with identity mapping

When a client-side application (SPA, CMS, admin UI) needs to interact with a backend service that requires credentials:

Client ──→ Auth Layer (MFA) ──→ API Proxy ──→ Backend Service
                                  │
                                  ├── Reads user identity from auth headers
                                  ├── Looks up user's backend credential
                                  ├── Forwards request with backend credential
                                  └── Returns response (without credential)

The proxy:

1. Sits behind the authentication layer (Authelia, OAuth2 Proxy, etc.)
2. Reads the authenticated user's identity from trusted headers (e.g., Remote-User)
3. Maps the identity to the appropriate backend credential
4. Makes the backend API call with the credential
5. Returns the response — never the credential

Security layers (defense in depth)

A well-designed proxy architecture has multiple independent security layers:

1. Authentication — User must prove their identity (MFA, OIDC)
2. Session validation — Proxy validates the session is current and legitimate
3. Authorization — Proxy checks the user has access to the requested resource
4. Backend permissions — The backend service enforces its own access controls
5. Branch/scope protection — Fine-grained controls prevent privilege escalation (e.g., branch protection rules)

Each layer is independent — compromising one doesn't bypass the others.

Anti-patterns

• Passing API tokens to the browser via OAuth. Even with PKCE, the token ends up in client-accessible storage. Use a backend proxy instead.
• Shared service account tokens. One token for all users means no audit trail and no granular revocation. Map per-user tokens server-side.
• Embedding credentials in client-side config. API keys in config.js, .env files served statically, or hardcoded in HTML — all violate the boundary rule.
• Forwarding backend tokens via API responses. Even "temporarily" returning a token in a response body breaks the rule. The client should never see it.
• Using the same token for client auth and backend calls. The user's session token with your application is distinct from any token your application uses to call backend services.

When credentials must be client-side

Some scenarios genuinely require client-side credentials (e.g., direct S3 uploads for large files, WebRTC signaling). In these cases:

• Use presigned URLs or temporary credentials with the narrowest possible scope and shortest possible lifetime
• The presigning/credential-issuance happens server-side
• The temporary credential is scoped to exactly one operation (e.g., upload one file to one path)
• Log the issuance server-side for audit

Real-world example: CMS editing

Wrong: CMS authenticates directly with Gitea via OAuth popup. Gitea token lands in browser localStorage. CMS makes API calls directly to Gitea with the token.

Right: CMS sits behind Authelia (MFA). A proxy service intercepts the OAuth flow, issues a proxy session token (containing only the user's identity), and forwards all API calls to Gitea using a per-user server-side Gitea token. The Gitea token never leaves the server.

Browser-Side OAuth Popup: COOP Severs window.opener

When a client-side app (CMS admin UI, SPA) runs an OAuth popup flow and the callback redirects through a different origin (e.g., an auth proxy on another subdomain), the Cross-Origin-Opener-Policy (COOP) mismatch severs window.opener. The popup can no longer postMessage its result back to the opener, and the parent's popup-detection logic (commonly window.opener?.origin === location.origin && window.name === 'auth') silently fails — producing zero token exchanges and, if the callback falls through to re-init the login flow, an infinite popup loop.

Mitigations:

• Set COOP: same-origin-allow-popups on the page that opens the popup; keep COOP off the cross-origin callback/proxy page.
• Handle the OAuth callback explicitly: when ?code=&state= are present, always return after processing — never fall through to auto-init/auto-click, which is what amplifies a broken flow into an infinite loop.
• When window.opener is severed, deliver the auth result via a BroadcastChannel fallback. Re-dispatch it as a synthetic MessageEvent so the framework's existing handler fires without patching third-party (often minified) source.

This is the browser-mechanics complement to the server-boundary rule above: even with a correct backend proxy, the popup handshake itself breaks on COOP. Check security headers (COOP/COEP) early when a login popup silently never completes — before investigating sessions, cookies, or backend state.

Multi-Tenant Isolation Guard on Outbound Writes

When a service writes into per-tenant destinations — customer Slack channels, per-customer boards, per-org webhooks, tenant-scoped buckets — route every outbound write through a single IsolationGuard module.

What the guard validates

For every outbound write, the guard checks:

1. Destination ownership — the target channel/board/webhook belongs to the tenant the write is scoped to
2. Payload cross-references — the payload does not reference other tenants by name or ID (string match against the known tenant list)
3. Cross-tenant field stripping — fields known to carry cross-tenant context (internal descriptions, linked-issue titles, audit trails) are removed or redacted before leaving the service

Why centralise it

Per-tenant isolation is only testable if there is one place to exercise. If each call site inlines its own "scope this write" logic, cross-tenant leak tests must cover every call site and every future one. A single guard module:

• Gives tests one surface to fuzz with adversarial payloads
• Makes it impossible to ship a new outbound path that forgets the check (the guard is the only API)
• Centralises logging for any blocked write — leaks become observable, not silent

Pattern

CallSite ──→ IsolationGuard.send(tenant_id, destination, payload)
                  │
                  ├── Validate destination ∈ tenant_id's destinations
                  ├── Scan payload for other tenants' names/IDs
                  ├── Strip cross-tenant fields per schema
                  ├── Log (tenant_id, destination, redacted fields)
                  └── Dispatch to underlying transport

No call site should import the underlying transport directly. Lint or grep for direct imports as a CI check.

File-Only Secret Delivery for Long-Lived Processes

For any container or process running longer than a few minutes, deliver secrets via tmpfs-backed file mounts (mode 0400, owned by root or the secrets operator), never via environment variables. Env vars leak into:

• ps -auxe and /proc/<pid>/environ for any process in the PID namespace
• Log aggregators (anything that captures the process tree)
• Accidentally-echoed error output (echo "config: $DATABASE_URL" leaks the secret to stdout)
• Child processes that inherit the environment by default

Env-var delivery is acceptable only for short-lived (<60s) ephemeral tasks where the exposure window is bounded and no other workloads share the PID namespace.

Defense-in-depth pattern (External Secrets Operator + init):

1. ESO mounts the Secret at /var/run/secrets/... as mode 0400, owned by root (controller's service account)
2. The pod's init container stages a 0600 copy to a path owned by the application's UID
3. The application reads from disk at invocation time, not at startup — so rotation lands without restart

Two-layer scoping: the ESO mount is the source-of-truth; the init copy is the application-readable view. A compromised application can read its copy but cannot read the ESO source. A compromised init container does not survive past startup.

Server Never Holds the Private Key — CSR-Based Flow for Server-Issued Identity

When a server issues identity material to a client (mTLS cert, agent signing key, per-request token), design the protocol so the server signs a CSR rather than generating the keypair.

Rationale: Most managed-runtime languages (CPython, Java, Go via reflection) cannot reliably zeroise private bytes. CPython strings are immutable; there is no memset equivalent; GC residue, copy-on-write, and string interning all leave secret material in addressable memory after "deletion." Once the server has held the private key in process memory, you cannot prove it is gone.

Flow:

1. Client generates the keypair locally via the standard cryptography primitives
2. Client builds a CSR including its identity claims (SAN URI, subject CN)
3. Server receives the CSR over an authenticated transport, validates the identity claims, signs the CSR, returns the certificate
4. The server never sees the private key

Applies to any mTLS issuance, ACL/agent-signing keys, per-request token issuance, and any "server hands the client an identity" protocol. The pattern survives memory-dump forensic analysis of the server.