# API Design Best practices for REST/HTTP APIs in internal microservices and platform services. Focused on practical defaults -- not aspirational ideals. Sourced from OWASP API Security Top 10 (2023), RFC 9700 (OAuth 2.0 Security BCP, January 2025), Google AIP, and production experience. Cross-references: [Security Architecture](security-architecture.md) covers the server boundary rule and proxy patterns. [Secrets Management](secrets-management.md) covers credential storage and rotation. --- ## 1. Transport Security ### 1.1 HTTPS everywhere, no exceptions **Principle:** Every API endpoint -- internal or external -- must serve over TLS. Plaintext HTTP must not be available, even on internal networks. **Why it matters:** Without TLS, any network hop (load balancer, sidecar, switch) can observe or modify traffic. Internal networks are not trusted in a zero-trust model -- a compromised pod can sniff adjacent traffic. **How to implement:** - Terminate TLS at the ingress controller (e.g., Traefik, NGINX) with certificates from cert-manager / Let's Encrypt. - For service-to-service within the cluster, use a service mesh (Istio, Linkerd) or cert-manager CSI driver to issue per-pod certificates. - Set `Strict-Transport-Security` headers on all responses. - Redirect HTTP to HTTPS at the ingress layer. **Anti-patterns:** - "Internal traffic doesn't need encryption" -- it does under zero-trust. - Self-signed certificates with verification disabled (`--insecure`, `verify=False`) -- defeats the purpose of TLS. - Long-lived certificates (years) with no rotation -- use short-lived certs (days to weeks) with automated renewal. ### 1.2 mTLS between services **Principle:** Service-to-service communication must use mutual TLS -- both sides present and verify certificates. **Why it matters:** Server-only TLS authenticates the server to the client, but any client can connect. mTLS ensures both parties have a cryptographically verified identity, which is the foundation of zero-trust networking. **How to implement:** - Service mesh (Istio strict mode, Linkerd) handles mTLS transparently via sidecar proxies -- no application code changes. - Use SPIFFE/SPIRE for standardized workload identity (SVID certificates). - Default certificate lifetime should be short (24 hours) with automatic rotation. - Start in permissive mode (allow both plain and mTLS), migrate to strict mode once all services are enrolled. **Anti-patterns:** - Permissive mode as a permanent state -- it must be a migration step, not the end state. - Disabling mTLS verification for "debugging" and forgetting to re-enable it. - Using a single shared certificate for all services -- each workload needs its own identity. ### 1.3 Certificate management **Principle:** Certificate issuance and rotation must be fully automated. No manual certificate management in production. **Why it matters:** Manual certificate management leads to expired certificates, which cause outages. It also leads to long-lived certificates, which increase blast radius if compromised. **How to implement:** - cert-manager in Kubernetes with ClusterIssuer for ingress certificates. - Service mesh control plane for workload certificates (Istio Citadel, Linkerd identity). - Monitor certificate expiry with alerts at 30/14/7 days before expiry. - Store CA keys in HSM or sealed secrets -- never in plaintext ConfigMaps. **Anti-patterns:** - Certificates stored in Git repos (even encrypted, they need rotation). - Wildcard certificates shared across trust boundaries. - No monitoring for certificate expiry -- silent failures at 3am. --- ## 2. Authentication and Authorization ### 2.1 OIDC/OAuth2 for user-facing APIs (RFC 9700) **Principle:** Use OAuth 2.0 Authorization Code flow with PKCE for all client types. The implicit flow and resource owner password credentials flow are deprecated per RFC 9700 (January 2025). **Why it matters:** The implicit flow exposes access tokens in URLs and browser history. The password grant requires users to share credentials directly with the client, bypassing centralized identity providers. **How to implement:** - Authorization Code + PKCE for all clients (web, mobile, CLI). PKCE is now mandatory for all client types, not just public clients. - Use `S256` challenge method (not `plain`). - Tokens issued by the authorization server, validated by the resource server. - Use Authorization Server Metadata (RFC 8414) for automatic discovery of endpoints and supported features. **Anti-patterns:** - Implicit flow (`response_type=token`) -- deprecated by RFC 9700. - Resource Owner Password Credentials flow -- deprecated by RFC 9700. - Storing tokens in localStorage (accessible to XSS) -- use httpOnly cookies or in-memory storage with refresh token rotation. - Long-lived access tokens without refresh -- use short-lived access tokens (5-15 minutes) with refresh token rotation. ### 2.2 JWT best practices **Principle:** JWTs must be validated completely on every request -- signature, expiry, issuer, audience, and algorithm. **Why it matters:** Incomplete JWT validation is a top attack vector. Accepting expired tokens, wrong audiences, or `alg: none` enables token forgery and replay. **How to implement:** - Validate: signature (asymmetric preferred -- RS256/ES256), `exp`, `iat`, `iss`, `aud`, `nbf`. - Use asymmetric signing (RS256/ES256) so that only the auth server holds the private key. Resource servers only need the public key. - Set `aud` claim to the specific API audience -- reject tokens intended for other services. - Keep tokens small -- put only identity and authorization claims in the token, fetch additional data from a userinfo endpoint. - Use `jti` (JWT ID) claim for token revocation checks when needed. **Anti-patterns:** - Accepting `alg: none` or allowing algorithm switching -- pin the expected algorithm server-side. - Not validating `aud` -- allows tokens from one service to be replayed against another. - Symmetric signing (HS256) with a shared secret across services -- if one service is compromised, all are. - Treating JWTs as sessions -- JWTs are not revocable by default. Combine with short expiry and token introspection for revocation. ### 2.3 Service-to-service authentication **Principle:** Services authenticate to each other using mTLS identities (SPIFFE) or short-lived JWTs from a token exchange. Never shared static API keys. **Why it matters:** Shared API keys have no expiry, no rotation path, no per-service identity, and no audit trail. If one service is compromised, the key works for everything. **How to implement:** - **Preferred: mTLS with SPIFFE.** The service mesh provides identity automatically. Authorization policies reference SPIFFE IDs (e.g., `spiffe://cluster.local/ns/payments/sa/payment-svc`). - **Alternative: OAuth2 Client Credentials flow.** Each service has its own `client_id` and `client_secret` (or asymmetric key pair). Tokens are short-lived and scoped to specific audiences. - Use asymmetric client authentication (private_key_jwt per RFC 7523) rather than client secrets where possible. - Implement audience restriction -- tokens minted for service A must not be accepted by service B. **Anti-patterns:** - Shared static API keys passed in headers or query strings. - One "admin" service account used by all services. - Service-to-service tokens with no audience claim -- replayable across any internal API. - Bearer tokens without mTLS -- if the network is compromised, the token can be stolen and replayed from anywhere. ### 2.4 Authorization: object-level and function-level **Principle:** Check authorization at every API endpoint, for every object access, based on the authenticated identity. Never rely on "the client won't send that request." **Why it matters:** Broken Object-Level Authorization (BOLA) is the #1 risk in the OWASP API Security Top 10. Broken Function-Level Authorization is #5. These are the most common API vulnerabilities found in penetration tests. **How to implement:** - Every endpoint that accesses a specific resource must verify the caller owns or has access to that resource. - Use middleware/decorators that enforce authorization before the handler runs. - Use random UUIDs for resource identifiers, not sequential integers (which are trivially enumerable). - Separate authorization for data access (BOLA) and function access (admin endpoints, bulk operations). - Automated tests that verify: user A cannot access user B's resources, non-admin cannot call admin endpoints. **Anti-patterns:** - Authorization only at the API gateway -- must also be enforced at the service level. - Relying on obscurity of endpoint URLs for access control. - Sequential/predictable resource IDs without authorization checks. - Missing authorization on secondary endpoints (e.g., `/users/{id}/orders` checks user but not order ownership). --- ## 3. API Design Patterns ### 3.1 Versioning **Principle:** Version your API from day one using URL path versioning (`/v1/`). Support at most two versions simultaneously. **Why it matters:** Breaking changes without versioning cause cascading failures across all consumers simultaneously. Supporting too many versions creates maintenance burden and security risk (old versions may lack patches). **How to implement:** - URL path: `/api/v1/resources` -- simple, visible, cacheable. - Deprecation policy: announce deprecation in response headers (`Deprecation: true`, `Sunset: `). - Maximum two active versions. When v3 launches, v1 is removed. - Internal services can use header-based versioning (`Accept: application/vnd.myapi.v2+json`) if URL versioning is too rigid for rapid iteration. **Anti-patterns:** - No versioning ("we'll be careful") -- you will break consumers. - Unlimited version support -- v1 through v7 all still running, each with different bugs. - Breaking changes in a patch version. - Versioning individual endpoints instead of the whole API surface. ### 3.2 Pagination **Principle:** All list endpoints must paginate. Use cursor-based pagination for real-time data; offset-based for stable datasets. **Why it matters:** Unbounded list responses cause memory exhaustion, slow responses, and database strain. Large offset values cause full table scans. **How to implement:** - **Cursor-based (preferred):** Return an opaque `next_cursor` token. Client passes it to get the next page. Stable under concurrent writes. ```json { "data": [...], "next_cursor": "abc123", "has_more": true } ``` - **Offset-based (simple datasets):** `?limit=50&offset=100`. Acceptable for admin dashboards or infrequently changing data. - Set a maximum page size (e.g., 100) enforced server-side. Ignore client requests for larger pages. - Always return pagination metadata (`next_cursor`, `has_more`, or `total_count` if cheap to compute). **Anti-patterns:** - No pagination on list endpoints -- returns 50,000 records in one response. - Offset-based pagination on large, frequently-changing datasets -- pages shift as records are inserted/deleted. - Client-controlled page size with no server-side maximum. - `total_count` requiring a full table scan on every request -- make it optional or cached. ### 3.3 Error handling **Principle:** Return structured, machine-readable errors with stable error codes, human-readable messages, and consistent shape across all endpoints. **Why it matters:** Inconsistent error formats force every consumer to write custom parsing logic. Missing error codes make automated retry decisions impossible. Leaking stack traces exposes internals to attackers. **How to implement:** - Standard error envelope: ```json { "error": { "code": "RESOURCE_NOT_FOUND", "message": "Order 7f3a... not found", "details": [{ "field": "order_id", "reason": "not_found" }] } } ``` - Use HTTP status codes correctly: 400 (bad input), 401 (unauthenticated), 403 (unauthorized), 404 (not found), 409 (conflict), 422 (validation), 429 (rate limited), 500 (server error). - Error codes are stable strings (not integers) that consumers can switch on. - Never expose stack traces, SQL errors, or internal paths in error responses. - Log the full error server-side with a correlation ID. Return only the correlation ID to the client. **Anti-patterns:** - 200 OK with `{"success": false}` -- use HTTP status codes. - Returning raw database errors ("duplicate key violates unique constraint on..."). - Different error shapes from different endpoints in the same API. - Generic "Internal Server Error" with no correlation ID -- impossible to debug. ### 3.4 Idempotency **Principle:** All state-changing operations must be safe to retry. Use idempotency keys for POST requests; PUT and DELETE are idempotent by definition. **Why it matters:** Network failures, timeouts, and retries are normal in distributed systems. Without idempotency, retried requests create duplicate orders, double payments, or inconsistent state. **How to implement:** - Accept `Idempotency-Key` header (IETF draft: draft-ietf-httpapi-idempotency-key-header) on POST endpoints. - Server stores the response for a given key (TTL 24-48 hours). Duplicate requests return the stored response. - Use UUIDv4 for idempotency keys -- never sequential or timestamp-based (predictable/guessable). - Handle concurrent duplicate requests with locking: first request processes, subsequent requests wait then return cached response. - PUT must be truly idempotent: same request, same result, no side effects on repeat. **Anti-patterns:** - POST endpoints with no idempotency support -- every retry creates a duplicate. - Idempotency keys stored forever (memory leak) or for too short a period (retries after expiry create duplicates). - Client-generated sequential keys (integers, timestamps) -- guessable and exploitable. - "Idempotent" endpoints that still send duplicate emails/webhooks on retry. ### 3.5 Rate limiting **Principle:** Every API must enforce rate limits. Return standard headers so clients can self-throttle. **Why it matters:** Without rate limits, a single misbehaving client (or attacker) can exhaust resources for all consumers. Rate limits also protect downstream dependencies. **How to implement:** - Token bucket or sliding window algorithm (token bucket is simplest with good burst handling). - Return headers: `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset` (IETF draft still pending; `X-` prefix remains de facto standard). - Return `429 Too Many Requests` with `Retry-After` header (RFC 6585). - Rate limit checks execute before expensive operations (auth, database queries). - For distributed deployments, use Redis with atomic Lua scripts for counter operations -- avoid race conditions. - Different tiers for different consumers (internal services get higher limits than external clients). **Anti-patterns:** - No rate limiting ("it's an internal API") -- a runaway loop in one service takes down the whole platform. - Rate limiting after expensive operations (database query runs, then rate limit rejects the response). - No `Retry-After` header -- clients retry immediately in a tight loop, making the problem worse. - Per-IP rate limiting only -- bypassed by distributed clients, unfair to NAT'd users. --- ## 4. Input Validation ### 4.1 Schema validation at the edge **Principle:** Validate all request bodies against a schema (OpenAPI/JSON Schema) at the API gateway or middleware layer. Reject requests that don't conform before they reach business logic. **Why it matters:** Invalid input that reaches business logic causes unpredictable behavior -- crashes, data corruption, injection attacks. Edge validation is the first line of defense. **How to implement:** - Define request/response schemas in OpenAPI 3.x. Generate validation middleware from the spec. - Reject unknown fields (additionalProperties: false) -- attackers probe via unexpected fields. - Enforce type constraints: string lengths, integer ranges, enum values, date formats. - Validate `Content-Type` header -- reject requests with unexpected content types (e.g., reject `multipart/form-data` on a JSON-only endpoint). **Anti-patterns:** - Validation only in business logic, not at the edge -- invalid data traverses the full call stack before rejection. - Accepting and silently ignoring unknown fields -- hides bugs and enables mass assignment attacks. - Validating types but not ranges -- accepting an `age` field of 99999 or -1. - No schema at all ("we'll validate manually") -- inconsistent validation across endpoints. ### 4.2 Injection prevention **Principle:** Use parameterized queries for all database access. Never concatenate user input into queries, commands, or templates. **Why it matters:** SQL injection remains in the OWASP Top 10 after 20+ years. NoSQL injection, LDAP injection, and command injection follow the same pattern -- unsanitized input in a query language. **How to implement:** - Use an ORM (SQLAlchemy, Prisma, TypeORM) or parameterized queries. All major ORMs parameterize by default. - For raw SQL (performance-critical paths), use prepared statements exclusively. - Validate input with allowlists, not denylists. If a field should be a UUID, validate it's a UUID -- don't try to strip "malicious characters." - For template rendering, use auto-escaping (Jinja2 autoescape, React JSX auto-escaping). **Anti-patterns:** - String concatenation in SQL: `f"SELECT * FROM users WHERE id = '{user_input}'"`. - Denylisting dangerous characters instead of allowlisting valid patterns. - Trusting input from "internal" services -- a compromised upstream service sends malicious data. - Disabling ORM parameterization for "performance" without understanding the security cost. ### 4.3 Request size and depth limits **Principle:** Enforce maximum request body size, JSON nesting depth, and array length at the gateway level. **Why it matters:** Deeply nested JSON or extremely large payloads cause CPU exhaustion during parsing (hash collision attacks, recursive descent parsers). This is a denial-of-service vector. **How to implement:** - Set maximum body size at the reverse proxy/ingress (e.g., `client_max_body_size 1m` in NGINX). - Limit JSON nesting depth (8-16 levels is generous for any real use case). - Limit array sizes in request bodies (e.g., batch endpoints accept max 100 items). - Set request timeouts at the gateway -- don't let slow clients hold connections open. **Anti-patterns:** - No body size limit -- 100MB JSON payload parsed by every middleware layer. - Accepting arbitrarily nested JSON -- `{"a":{"a":{"a":...}}}` 1000 levels deep. - Batch endpoints with no limit -- client sends 1 million items in one request. --- ## 5. Secrets in APIs ### 5.1 Never in URLs or query parameters **Principle:** Authentication tokens, API keys, and any secret material must be sent in headers (Authorization, custom headers) or request bodies. Never in URLs or query parameters. **Why it matters:** URLs are logged everywhere -- web server access logs, proxy logs, browser history, referrer headers, CDN logs, monitoring tools. A token in a URL is a token in every log file in the request path. **How to implement:** - Use `Authorization: Bearer ` header for all token-based auth. - For webhook signatures, use a signature header (e.g., `X-Hub-Signature-256`). - If an API currently accepts tokens in query params, deprecate that path and migrate to header-based auth. - Configure log scrubbing to redact Authorization headers, but don't rely on it as the primary control. **Anti-patterns:** - `GET /api/resources?api_key=sk_live_abc123` -- key in every access log. - OAuth redirect URIs with tokens in query params (use `response_mode=form_post` or authorization code flow). - Webhook URLs with embedded secrets (`/webhook?secret=abc`) -- logged, cached, shared. ### 5.2 Token rotation and expiry **Principle:** All tokens and API keys must have expiry dates and a documented rotation procedure. No permanent credentials. **Why it matters:** Leaked tokens without expiry are valid forever. Rotation limits blast radius -- even if a token is compromised, it expires soon. **How to implement:** - Access tokens: 5-15 minute expiry, refreshed via refresh token. - Refresh tokens: rotate on use (each refresh issues a new refresh token and invalidates the old one). - API keys for external integrations: 90-day rotation policy with overlap period (new key valid before old key expires). - Service account tokens (OAuth2 client credentials): short-lived (1 hour), fetched on demand. - Track expiry dates in a credential inventory. Alert before expiry (see [Secrets Management](secrets-management.md) -- Credential Lifecycle Management). **Anti-patterns:** - API keys that never expire ("we'll rotate them when we need to" -- you won't). - Refresh tokens that don't rotate -- stolen refresh token provides permanent access. - No overlap period during rotation -- brief outage while all consumers update. - Hardcoded tokens in application config deployed via CI -- rotation requires a full redeploy. ### 5.3 No secrets in logs or error responses **Principle:** Scrub all secrets from logs, error responses, and monitoring data. Structured logging with explicit field selection is safer than serializing request objects. **Why it matters:** Log aggregation systems (ELK, Loki, Datadog) are often accessible to broader teams than production systems. A token in a log entry has a much wider exposure surface than a token in a running process. **How to implement:** - Use structured logging. Log specific fields, not entire request objects. - Redact `Authorization` headers and any field matching `token`, `password`, `secret`, `key` patterns in log middleware. - Never log request bodies for auth endpoints (login, token exchange). - Error responses must not include internal state -- return a correlation ID and log details server-side. **Anti-patterns:** - `logger.info(f"Request: {request.headers}")` -- logs all headers including Authorization. - Error responses that include the original request (including auth headers) for "debugging convenience." - Logging full webhook payloads that contain signing secrets in custom headers. --- ## 6. Service Mesh and Zero Trust ### 6.1 Default deny with explicit allow **Principle:** Network policies and authorization policies must default to deny-all. Every allowed communication path is explicitly defined. **Why it matters:** Default-allow means a compromised service can reach every other service in the cluster. Default-deny contains the blast radius to only the services the compromised workload was authorized to reach. **How to implement:** - Kubernetes NetworkPolicy: deploy a default-deny policy in every namespace, then add specific allow rules. ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all spec: podSelector: {} policyTypes: [Ingress, Egress] ``` - Service mesh authorization policies: deny by default, allow specific source-to-destination pairs by SPIFFE ID. - Audit policies periodically -- remove rules for decommissioned services. **Anti-patterns:** - No network policies ("everything's in the cluster, it's fine"). - Overly broad allow rules (`allow all from namespace X`) -- defeats the purpose. - Network policies without egress rules -- ingress-only policies still allow compromised pods to exfiltrate data. ### 6.2 Least-privilege service identities **Principle:** Each service gets its own identity (Kubernetes ServiceAccount + SPIFFE SVID) with the minimum permissions needed. No shared service accounts. **Why it matters:** Shared identities prevent granular authorization, audit trails, and revocation. If services A and B share an identity, you cannot authorize A without also authorizing B. **How to implement:** - One Kubernetes ServiceAccount per workload (not per namespace). - RBAC bindings scoped to exactly what the service needs (specific API groups, resources, verbs). - Authorization policies reference specific service identities: "payment-svc can call order-svc on POST /orders/{id}/payment." - Regularly audit which identities have access to which services -- prune unused access. **Anti-patterns:** - Default ServiceAccount used by all pods in a namespace. - Cluster-wide RBAC bindings for convenience. - Service identities with wildcard permissions ("allow all methods on all paths"). - No audit of identity-to-service mappings. ### 6.3 Observability as a security control **Principle:** Distributed tracing, access logs, and metrics from the service mesh are security controls, not just debugging tools. Monitor them for anomalies. **Why it matters:** Zero trust assumes breach. Detection depends on visibility. If you can't see who called what, you can't detect lateral movement. **How to implement:** - Enable access logging in the service mesh (Istio/Envoy access logs, Linkerd tap). - Distributed tracing (OpenTelemetry, Jaeger) with trace context propagated across all service calls. - Alert on anomalies: unexpected source-destination pairs, unusual request volumes, authorization denials. - Retain access logs long enough for incident investigation (30-90 days minimum). **Anti-patterns:** - Disabling access logging for performance -- sample instead of disabling entirely. - Tracing only in development, not production. - No alerting on authorization policy denials -- failed access attempts are the signal. --- ## OWASP API Security Top 10 (2023) Quick Reference For context, the current OWASP API Security Top 10 maps to the practices above: | # | Risk | Where addressed | |---|------|----------------| | API1 | Broken Object-Level Authorization | Section 2.4 | | API2 | Broken Authentication | Sections 2.1, 2.2, 2.3 | | API3 | Broken Object Property-Level Authorization | Section 2.4, 4.1 | | API4 | Unrestricted Resource Consumption | Sections 3.5, 4.3 | | API5 | Broken Function-Level Authorization | Section 2.4 | | API6 | Unrestricted Access to Sensitive Business Flows | Sections 3.4, 3.5 | | API7 | Server-Side Request Forgery | Section 4.2 | | API8 | Security Misconfiguration | Sections 1.1, 6.1 | | API9 | Improper Inventory Management | Section 3.1 | | API10 | Unsafe Consumption of APIs | Section 4.2 | --- ## Sources - [OWASP API Security Top 10](https://owasp.org/API-Security/) - [RFC 9700 - OAuth 2.0 Security Best Current Practice (January 2025)](https://datatracker.ietf.org/doc/rfc9700/) - [OAuth best practices: RFC 9700 summary -- WorkOS](https://workos.com/blog/oauth-best-practices) - [IETF Idempotency-Key Header Draft](https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/) - [Google AIP-193: Errors](https://google.aip.dev/193) - [ByteByteGo: REST API Design](https://blog.bytebytego.com/p/the-art-of-rest-api-design-idempotency) - [Zuplo: Rate Limiting Best Practices](https://zuplo.com/learning-center/10-best-practices-for-api-rate-limiting-in-2025) - [Zuplo: Input/Output Validation](https://zuplo.com/blog/2025/03/25/input-output-validation-best-practices) - [OWASP Input Validation Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Input_Validation_Cheat_Sheet.html) - [Machine Identity: mTLS + SPIFFE Zero Trust Guide](https://petronellatech.com/blog/machine-identity-is-the-new-perimeter-mtls-spiffe-for-zero-trust/) - [Buoyant: Zero Trust, mTLS, and the Service Mesh](https://www.buoyant.io/blog/zero-trust-mtls-and-the-service-mesh-explained) - [Kong: Zero Trust with Service Mesh](https://konghq.com/blog/engineering/zero-trust-service-mesh-security) - [Microsoft Azure: Web API Design Best Practices](https://learn.microsoft.com/en-us/azure/architecture/best-practices/api-design)