best-practices/networking.md

Networking & Infrastructure

nftables Flush Ruleset on Remote Hosts

On remote hosts, nftables flush ruleset followed by a failed rule load leaves the host with NO firewall. SSH survives only on existing connections — new connections are blocked or allowed depending on the default policy.

Always validate rules before applying: nft -c -f <rulefile> does a dry-run parse. For extra safety, deploy a cron-based auto-rollback timer that reverts rules unless explicitly confirmed (similar to shutdown -c pattern).

systemd Socket Activation Overrides Config File Ports

On modern Linux systems (Ubuntu 24.04+), systemd socket activation controls the listening port for services like SSH. Editing the service config file alone (e.g., sshd_config Port 2222) has no effect — the socket unit still binds the original port.

Check socket activation first: systemctl cat <service>.socket shows whether socket activation is in play. If so, override the socket unit's ListenStream directive, not the service config.

Docker Sets iptables FORWARD Policy to DROP

Docker sets the iptables FORWARD chain default policy to DROP. This affects ALL forwarding on the host, not just Docker traffic. Non-Docker forwarding (VPN, VM bridges, custom NAT) silently breaks.

Fix: Add explicit ACCEPT rules in the DOCKER-USER chain for non-Docker forwarding needs. This chain is processed before Docker's own rules and persists across Docker restarts.

HTTP Host Header vs TLS SNI Are Different Layers

When proxying to a backend over HTTPS, two independent identifiers must be set correctly:

• TLS SNI — sent during the TLS handshake, used for certificate selection. Missing SNI causes x509: cannot validate certificate for <IP>.
• HTTP Host header — sent after TLS is established, used for virtual host routing. Missing or wrong Host header causes 404 from the backend.

A reverse proxy must set both. They often need to be the same value, but they're configured independently.

Wildcard Certs in Auto-Renewing Proxies

Auto-renewing proxies (Caddy, Traefik with Let's Encrypt, etc.) that also support file-loaded certificates treat file-loaded certs as globally available. A wildcard cert loaded for one site block will match ALL matching subdomains, silently preventing automatic certificate issuance for other sites.

Rule: Use automatic certificate management for all sites. Don't mix file-loaded and automatic certs unless you understand the matching priority.

Reverse Proxies Ignore Labels on Stopped Containers

Docker-label-based routing (Traefik, Caddy-docker-proxy, nginx-proxy) silently drops routes whose target containers are not running. Flags like allowEmptyServices do not help — the router only sees the labels of running containers.

This breaks on-demand and "scale-to-zero" backends: the proxy has no route to the stopped container, so the wake-up request never reaches whatever is meant to start it. Requests 404 (or worse, go to the wrong backend) until the container happens to be up.

Rule: For any backend that may not always be running, declare the route in dynamic file config, not container labels. File-config routes exist regardless of container state — the router can then proxy to a "wake" handler, return a holding page, or queue the request.

Validation: always test routing with the backend container stopped, not just running. If the route disappears when the container stops, the config is wrong for on-demand use.

Cilium DNAT Resolves LB VIP Before NetworkPolicy Evaluation

Cilium performs DNAT on LoadBalancer VIP traffic before evaluating NetworkPolicy. Traffic to a VIP is rewritten to a backend pod IP before the policy check. For egress to LoadBalancer services in CiliumNetworkPolicy, use toEndpoints targeting the backend pods (by namespace/label), not toCIDR targeting the VIP.