best-practices/api-design.md

API Design

Best practices for REST/HTTP APIs in internal microservices and platform services. Focused on practical defaults -- not aspirational ideals. Sourced from OWASP API Security Top 10 (2023), RFC 9700 (OAuth 2.0 Security BCP, January 2025), Google AIP, and production experience.

Cross-references: Security Architecture covers the server boundary rule and proxy patterns. Secrets Management covers credential storage and rotation.

---

1. Transport Security

1.1 HTTPS everywhere, no exceptions

Principle: Every API endpoint -- internal or external -- must serve over TLS. Plaintext HTTP must not be available, even on internal networks.

Why it matters: Without TLS, any network hop (load balancer, sidecar, switch) can observe or modify traffic. Internal networks are not trusted in a zero-trust model -- a compromised pod can sniff adjacent traffic.

How to implement:

• Terminate TLS at the ingress controller (e.g., Traefik, NGINX) with certificates from cert-manager / Let's Encrypt.
• For service-to-service within the cluster, use a service mesh (Istio, Linkerd) or cert-manager CSI driver to issue per-pod certificates.
• Set Strict-Transport-Security headers on all responses.
• Redirect HTTP to HTTPS at the ingress layer.

Anti-patterns:

• "Internal traffic doesn't need encryption" -- it does under zero-trust.
• Self-signed certificates with verification disabled (--insecure, verify=False) -- defeats the purpose of TLS.
• Long-lived certificates (years) with no rotation -- use short-lived certs (days to weeks) with automated renewal.

1.2 mTLS between services

Principle: Service-to-service communication must use mutual TLS -- both sides present and verify certificates.

Why it matters: Server-only TLS authenticates the server to the client, but any client can connect. mTLS ensures both parties have a cryptographically verified identity, which is the foundation of zero-trust networking.

How to implement:

• Service mesh (Istio strict mode, Linkerd) handles mTLS transparently via sidecar proxies -- no application code changes.
• Use SPIFFE/SPIRE for standardized workload identity (SVID certificates).
• Default certificate lifetime should be short (24 hours) with automatic rotation.
• Start in permissive mode (allow both plain and mTLS), migrate to strict mode once all services are enrolled.

Anti-patterns:

• Permissive mode as a permanent state -- it must be a migration step, not the end state.
• Disabling mTLS verification for "debugging" and forgetting to re-enable it.
• Using a single shared certificate for all services -- each workload needs its own identity.

1.3 Certificate management

Principle: Certificate issuance and rotation must be fully automated. No manual certificate management in production.

Why it matters: Manual certificate management leads to expired certificates, which cause outages. It also leads to long-lived certificates, which increase blast radius if compromised.

How to implement:

• cert-manager in Kubernetes with ClusterIssuer for ingress certificates.
• Service mesh control plane for workload certificates (Istio Citadel, Linkerd identity).
• Monitor certificate expiry with alerts at 30/14/7 days before expiry.
• Store CA keys in HSM or sealed secrets -- never in plaintext ConfigMaps.

Anti-patterns:

• Certificates stored in Git repos (even encrypted, they need rotation).
• Wildcard certificates shared across trust boundaries.
• No monitoring for certificate expiry -- silent failures at 3am.

---

2. Authentication and Authorization

2.1 OIDC/OAuth2 for user-facing APIs (RFC 9700)

Principle: Use OAuth 2.0 Authorization Code flow with PKCE for all client types. The implicit flow and resource owner password credentials flow are deprecated per RFC 9700 (January 2025).

Why it matters: The implicit flow exposes access tokens in URLs and browser history. The password grant requires users to share credentials directly with the client, bypassing centralized identity providers.

How to implement:

• Authorization Code + PKCE for all clients (web, mobile, CLI). PKCE is now mandatory for all client types, not just public clients.
• Use S256 challenge method (not plain).
• Tokens issued by the authorization server, validated by the resource server.
• Use Authorization Server Metadata (RFC 8414) for automatic discovery of endpoints and supported features.

Anti-patterns:

• Implicit flow (response_type=token) -- deprecated by RFC 9700.
• Resource Owner Password Credentials flow -- deprecated by RFC 9700.
• Storing tokens in localStorage (accessible to XSS) -- use httpOnly cookies or in-memory storage with refresh token rotation.
• Long-lived access tokens without refresh -- use short-lived access tokens (5-15 minutes) with refresh token rotation.

2.2 JWT best practices

Principle: JWTs must be validated completely on every request -- signature, expiry, issuer, audience, and algorithm.

Why it matters: Incomplete JWT validation is a top attack vector. Accepting expired tokens, wrong audiences, or alg: none enables token forgery and replay.

How to implement:

• Validate: signature (asymmetric preferred -- RS256/ES256), exp, iat, iss, aud, nbf.
• Use asymmetric signing (RS256/ES256) so that only the auth server holds the private key. Resource servers only need the public key.
• Set aud claim to the specific API audience -- reject tokens intended for other services.
• Keep tokens small -- put only identity and authorization claims in the token, fetch additional data from a userinfo endpoint.
• Use jti (JWT ID) claim for token revocation checks when needed.

Anti-patterns:

• Accepting alg: none or allowing algorithm switching -- pin the expected algorithm server-side.
• Not validating aud -- allows tokens from one service to be replayed against another.
• Symmetric signing (HS256) with a shared secret across services -- if one service is compromised, all are.
• Treating JWTs as sessions -- JWTs are not revocable by default. Combine with short expiry and token introspection for revocation.

2.3 Service-to-service authentication

Principle: Services authenticate to each other using mTLS identities (SPIFFE) or short-lived JWTs from a token exchange. Never shared static API keys.

Why it matters: Shared API keys have no expiry, no rotation path, no per-service identity, and no audit trail. If one service is compromised, the key works for everything.

How to implement:

• Preferred: mTLS with SPIFFE. The service mesh provides identity automatically. Authorization policies reference SPIFFE IDs (e.g., spiffe://cluster.local/ns/payments/sa/payment-svc).
• Alternative: OAuth2 Client Credentials flow. Each service has its own client_id and client_secret (or asymmetric key pair). Tokens are short-lived and scoped to specific audiences.
• Use asymmetric client authentication (private_key_jwt per RFC 7523) rather than client secrets where possible.
• Implement audience restriction -- tokens minted for service A must not be accepted by service B.

Anti-patterns:

• Shared static API keys passed in headers or query strings.
• One "admin" service account used by all services.
• Service-to-service tokens with no audience claim -- replayable across any internal API.
• Bearer tokens without mTLS -- if the network is compromised, the token can be stolen and replayed from anywhere.

2.4 Authorization: object-level and function-level

Principle: Check authorization at every API endpoint, for every object access, based on the authenticated identity. Never rely on "the client won't send that request."

Why it matters: Broken Object-Level Authorization (BOLA) is the #1 risk in the OWASP API Security Top 10. Broken Function-Level Authorization is #5. These are the most common API vulnerabilities found in penetration tests.

How to implement:

• Every endpoint that accesses a specific resource must verify the caller owns or has access to that resource.
• Use middleware/decorators that enforce authorization before the handler runs.
• Use random UUIDs for resource identifiers, not sequential integers (which are trivially enumerable).
• Separate authorization for data access (BOLA) and function access (admin endpoints, bulk operations).
• Automated tests that verify: user A cannot access user B's resources, non-admin cannot call admin endpoints.

Anti-patterns:

• Authorization only at the API gateway -- must also be enforced at the service level.
• Relying on obscurity of endpoint URLs for access control.
• Sequential/predictable resource IDs without authorization checks.
• Missing authorization on secondary endpoints (e.g., /users/{id}/orders checks user but not order ownership).

---

3. API Design Patterns

3.1 Versioning

Principle: Version your API from day one using URL path versioning (/v1/). Support at most two versions simultaneously.

Why it matters: Breaking changes without versioning cause cascading failures across all consumers simultaneously. Supporting too many versions creates maintenance burden and security risk (old versions may lack patches).

How to implement:

• URL path: /api/v1/resources -- simple, visible, cacheable.
• Deprecation policy: announce deprecation in response headers (Deprecation: true, Sunset: <date>).
• Maximum two active versions. When v3 launches, v1 is removed.
• Internal services can use header-based versioning (Accept: application/vnd.myapi.v2+json) if URL versioning is too rigid for rapid iteration.

Anti-patterns:

• No versioning ("we'll be careful") -- you will break consumers.
• Unlimited version support -- v1 through v7 all still running, each with different bugs.
• Breaking changes in a patch version.
• Versioning individual endpoints instead of the whole API surface.

3.2 Pagination

Principle: All list endpoints must paginate. Use cursor-based pagination for real-time data; offset-based for stable datasets.

Why it matters: Unbounded list responses cause memory exhaustion, slow responses, and database strain. Large offset values cause full table scans.

How to implement:

• Cursor-based (preferred): Return an opaque next_cursor token. Client passes it to get the next page. Stable under concurrent writes.

  { "data": [...], "next_cursor": "abc123", "has_more": true }
  

• Offset-based (simple datasets): ?limit=50&offset=100. Acceptable for admin dashboards or infrequently changing data.
• Set a maximum page size (e.g., 100) enforced server-side. Ignore client requests for larger pages.
• Always return pagination metadata (next_cursor, has_more, or total_count if cheap to compute).

Anti-patterns:

• No pagination on list endpoints -- returns 50,000 records in one response.
• Offset-based pagination on large, frequently-changing datasets -- pages shift as records are inserted/deleted.
• Client-controlled page size with no server-side maximum.
• total_count requiring a full table scan on every request -- make it optional or cached.

3.3 Error handling

Principle: Return structured, machine-readable errors with stable error codes, human-readable messages, and consistent shape across all endpoints.

Why it matters: Inconsistent error formats force every consumer to write custom parsing logic. Missing error codes make automated retry decisions impossible. Leaking stack traces exposes internals to attackers.

How to implement:

• Standard error envelope:

  {
    "error": {
      "code": "RESOURCE_NOT_FOUND",
      "message": "Order 7f3a... not found",
      "details": [{ "field": "order_id", "reason": "not_found" }]
    }
  }
  

• Use HTTP status codes correctly: 400 (bad input), 401 (unauthenticated), 403 (unauthorized), 404 (not found), 409 (conflict), 422 (validation), 429 (rate limited), 500 (server error).
• Error codes are stable strings (not integers) that consumers can switch on.
• Never expose stack traces, SQL errors, or internal paths in error responses.
• Log the full error server-side with a correlation ID. Return only the correlation ID to the client.

Anti-patterns:

• 200 OK with {"success": false} -- use HTTP status codes.
• Returning raw database errors ("duplicate key violates unique constraint on...").
• Different error shapes from different endpoints in the same API.
• Generic "Internal Server Error" with no correlation ID -- impossible to debug.

3.4 Idempotency

Principle: All state-changing operations must be safe to retry. Use idempotency keys for POST requests; PUT and DELETE are idempotent by definition.

Why it matters: Network failures, timeouts, and retries are normal in distributed systems. Without idempotency, retried requests create duplicate orders, double payments, or inconsistent state.

How to implement:

• Accept Idempotency-Key header (IETF draft: draft-ietf-httpapi-idempotency-key-header) on POST endpoints.
• Server stores the response for a given key (TTL 24-48 hours). Duplicate requests return the stored response.
• Use UUIDv4 for idempotency keys -- never sequential or timestamp-based (predictable/guessable).
• Handle concurrent duplicate requests with locking: first request processes, subsequent requests wait then return cached response.
• PUT must be truly idempotent: same request, same result, no side effects on repeat.

Anti-patterns:

• POST endpoints with no idempotency support -- every retry creates a duplicate.
• Idempotency keys stored forever (memory leak) or for too short a period (retries after expiry create duplicates).
• Client-generated sequential keys (integers, timestamps) -- guessable and exploitable.
• "Idempotent" endpoints that still send duplicate emails/webhooks on retry.

3.5 Rate limiting

Principle: Every API must enforce rate limits. Return standard headers so clients can self-throttle.

Why it matters: Without rate limits, a single misbehaving client (or attacker) can exhaust resources for all consumers. Rate limits also protect downstream dependencies.

How to implement:

• Token bucket or sliding window algorithm (token bucket is simplest with good burst handling).
• Return headers: X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset (IETF draft still pending; X- prefix remains de facto standard).
• Return 429 Too Many Requests with Retry-After header (RFC 6585).
• Rate limit checks execute before expensive operations (auth, database queries).
• For distributed deployments, use Redis with atomic Lua scripts for counter operations -- avoid race conditions.
• Different tiers for different consumers (internal services get higher limits than external clients).

Anti-patterns:

• No rate limiting ("it's an internal API") -- a runaway loop in one service takes down the whole platform.
• Rate limiting after expensive operations (database query runs, then rate limit rejects the response).
• No Retry-After header -- clients retry immediately in a tight loop, making the problem worse.
• Per-IP rate limiting only -- bypassed by distributed clients, unfair to NAT'd users.

---

4. Input Validation

4.1 Schema validation at the edge

Principle: Validate all request bodies against a schema (OpenAPI/JSON Schema) at the API gateway or middleware layer. Reject requests that don't conform before they reach business logic.

Why it matters: Invalid input that reaches business logic causes unpredictable behavior -- crashes, data corruption, injection attacks. Edge validation is the first line of defense.

How to implement:

• Define request/response schemas in OpenAPI 3.x. Generate validation middleware from the spec.
• Reject unknown fields (additionalProperties: false) -- attackers probe via unexpected fields.
• Enforce type constraints: string lengths, integer ranges, enum values, date formats.
• Validate Content-Type header -- reject requests with unexpected content types (e.g., reject multipart/form-data on a JSON-only endpoint).

Anti-patterns:

• Validation only in business logic, not at the edge -- invalid data traverses the full call stack before rejection.
• Accepting and silently ignoring unknown fields -- hides bugs and enables mass assignment attacks.
• Validating types but not ranges -- accepting an age field of 99999 or -1.
• No schema at all ("we'll validate manually") -- inconsistent validation across endpoints.

4.2 Injection prevention

Principle: Use parameterized queries for all database access. Never concatenate user input into queries, commands, or templates.

Why it matters: SQL injection remains in the OWASP Top 10 after 20+ years. NoSQL injection, LDAP injection, and command injection follow the same pattern -- unsanitized input in a query language.

How to implement:

• Use an ORM (SQLAlchemy, Prisma, TypeORM) or parameterized queries. All major ORMs parameterize by default.
• For raw SQL (performance-critical paths), use prepared statements exclusively.
• Validate input with allowlists, not denylists. If a field should be a UUID, validate it's a UUID -- don't try to strip "malicious characters."
• For template rendering, use auto-escaping (Jinja2 autoescape, React JSX auto-escaping).

Anti-patterns:

• String concatenation in SQL: f"SELECT * FROM users WHERE id = '{user_input}'".
• Denylisting dangerous characters instead of allowlisting valid patterns.
• Trusting input from "internal" services -- a compromised upstream service sends malicious data.
• Disabling ORM parameterization for "performance" without understanding the security cost.

4.3 Request size and depth limits

Principle: Enforce maximum request body size, JSON nesting depth, and array length at the gateway level.

Why it matters: Deeply nested JSON or extremely large payloads cause CPU exhaustion during parsing (hash collision attacks, recursive descent parsers). This is a denial-of-service vector.

How to implement:

• Set maximum body size at the reverse proxy/ingress (e.g., client_max_body_size 1m in NGINX).
• Limit JSON nesting depth (8-16 levels is generous for any real use case).
• Limit array sizes in request bodies (e.g., batch endpoints accept max 100 items).
• Set request timeouts at the gateway -- don't let slow clients hold connections open.

Anti-patterns:

• No body size limit -- 100MB JSON payload parsed by every middleware layer.
• Accepting arbitrarily nested JSON -- {"a":{"a":{"a":...}}} 1000 levels deep.
• Batch endpoints with no limit -- client sends 1 million items in one request.

---

5. Secrets in APIs

5.1 Never in URLs or query parameters

Principle: Authentication tokens, API keys, and any secret material must be sent in headers (Authorization, custom headers) or request bodies. Never in URLs or query parameters.

Why it matters: URLs are logged everywhere -- web server access logs, proxy logs, browser history, referrer headers, CDN logs, monitoring tools. A token in a URL is a token in every log file in the request path.

How to implement:

• Use Authorization: Bearer <token> header for all token-based auth.
• For webhook signatures, use a signature header (e.g., X-Hub-Signature-256).
• If an API currently accepts tokens in query params, deprecate that path and migrate to header-based auth.
• Configure log scrubbing to redact Authorization headers, but don't rely on it as the primary control.

Anti-patterns:

• GET /api/resources?api_key=sk_live_abc123 -- key in every access log.
• OAuth redirect URIs with tokens in query params (use response_mode=form_post or authorization code flow).
• Webhook URLs with embedded secrets (/webhook?secret=abc) -- logged, cached, shared.

5.2 Token rotation and expiry

Principle: All tokens and API keys must have expiry dates and a documented rotation procedure. No permanent credentials.

Why it matters: Leaked tokens without expiry are valid forever. Rotation limits blast radius -- even if a token is compromised, it expires soon.

How to implement:

• Access tokens: 5-15 minute expiry, refreshed via refresh token.
• Refresh tokens: rotate on use (each refresh issues a new refresh token and invalidates the old one).
• API keys for external integrations: 90-day rotation policy with overlap period (new key valid before old key expires).
• Service account tokens (OAuth2 client credentials): short-lived (1 hour), fetched on demand.
• Track expiry dates in a credential inventory. Alert before expiry (see Secrets Management -- Credential Lifecycle Management).

Anti-patterns:

• API keys that never expire ("we'll rotate them when we need to" -- you won't).
• Refresh tokens that don't rotate -- stolen refresh token provides permanent access.
• No overlap period during rotation -- brief outage while all consumers update.
• Hardcoded tokens in application config deployed via CI -- rotation requires a full redeploy.

5.3 No secrets in logs or error responses

Principle: Scrub all secrets from logs, error responses, and monitoring data. Structured logging with explicit field selection is safer than serializing request objects.

Why it matters: Log aggregation systems (ELK, Loki, Datadog) are often accessible to broader teams than production systems. A token in a log entry has a much wider exposure surface than a token in a running process.

How to implement:

• Use structured logging. Log specific fields, not entire request objects.
• Redact Authorization headers and any field matching token, password, secret, key patterns in log middleware.
• Never log request bodies for auth endpoints (login, token exchange).
• Error responses must not include internal state -- return a correlation ID and log details server-side.

Anti-patterns:

• logger.info(f"Request: {request.headers}") -- logs all headers including Authorization.
• Error responses that include the original request (including auth headers) for "debugging convenience."
• Logging full webhook payloads that contain signing secrets in custom headers.

---

6. Service Mesh and Zero Trust

6.1 Default deny with explicit allow

Principle: Network policies and authorization policies must default to deny-all. Every allowed communication path is explicitly defined.

Why it matters: Default-allow means a compromised service can reach every other service in the cluster. Default-deny contains the blast radius to only the services the compromised workload was authorized to reach.

How to implement:

• Kubernetes NetworkPolicy: deploy a default-deny policy in every namespace, then add specific allow rules.

  apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata:
    name: default-deny-all
  spec:
    podSelector: {}
    policyTypes: [Ingress, Egress]
  

• Service mesh authorization policies: deny by default, allow specific source-to-destination pairs by SPIFFE ID.
• Audit policies periodically -- remove rules for decommissioned services.

Anti-patterns:

• No network policies ("everything's in the cluster, it's fine").
• Overly broad allow rules (allow all from namespace X) -- defeats the purpose.
• Network policies without egress rules -- ingress-only policies still allow compromised pods to exfiltrate data.

6.2 Least-privilege service identities

Principle: Each service gets its own identity (Kubernetes ServiceAccount + SPIFFE SVID) with the minimum permissions needed. No shared service accounts.

Why it matters: Shared identities prevent granular authorization, audit trails, and revocation. If services A and B share an identity, you cannot authorize A without also authorizing B.

How to implement:

• One Kubernetes ServiceAccount per workload (not per namespace).
• RBAC bindings scoped to exactly what the service needs (specific API groups, resources, verbs).
• Authorization policies reference specific service identities: "payment-svc can call order-svc on POST /orders/{id}/payment."
• Regularly audit which identities have access to which services -- prune unused access.

Anti-patterns:

• Default ServiceAccount used by all pods in a namespace.
• Cluster-wide RBAC bindings for convenience.
• Service identities with wildcard permissions ("allow all methods on all paths").
• No audit of identity-to-service mappings.

6.3 Observability as a security control

Principle: Distributed tracing, access logs, and metrics from the service mesh are security controls, not just debugging tools. Monitor them for anomalies.

Why it matters: Zero trust assumes breach. Detection depends on visibility. If you can't see who called what, you can't detect lateral movement.

How to implement:

• Enable access logging in the service mesh (Istio/Envoy access logs, Linkerd tap).
• Distributed tracing (OpenTelemetry, Jaeger) with trace context propagated across all service calls.
• Alert on anomalies: unexpected source-destination pairs, unusual request volumes, authorization denials.
• Retain access logs long enough for incident investigation (30-90 days minimum).

Anti-patterns:

• Disabling access logging for performance -- sample instead of disabling entirely.
• Tracing only in development, not production.
• No alerting on authorization policy denials -- failed access attempts are the signal.

---

OWASP API Security Top 10 (2023) Quick Reference

For context, the current OWASP API Security Top 10 maps to the practices above:

#	Risk	Where addressed
API1	Broken Object-Level Authorization	Section 2.4
API2	Broken Authentication	Sections 2.1, 2.2, 2.3
API3	Broken Object Property-Level Authorization	Section 2.4, 4.1
API4	Unrestricted Resource Consumption	Sections 3.5, 4.3
API5	Broken Function-Level Authorization	Section 2.4
API6	Unrestricted Access to Sensitive Business Flows	Sections 3.4, 3.5
API7	Server-Side Request Forgery	Section 4.2
API8	Security Misconfiguration	Sections 1.1, 6.1
API9	Improper Inventory Management	Section 3.1
API10	Unsafe Consumption of APIs	Section 4.2

---

Sources

• OWASP API Security Top 10
• RFC 9700 - OAuth 2.0 Security Best Current Practice (January 2025)
• OAuth best practices: RFC 9700 summary -- WorkOS
• IETF Idempotency-Key Header Draft
• Google AIP-193: Errors
• ByteByteGo: REST API Design
• Zuplo: Rate Limiting Best Practices
• Zuplo: Input/Output Validation
• OWASP Input Validation Cheat Sheet
• Machine Identity: mTLS + SPIFFE Zero Trust Guide
• Buoyant: Zero Trust, mTLS, and the Service Mesh
• Kong: Zero Trust with Service Mesh
• Microsoft Azure: Web API Design Best Practices