# Networking & Infrastructure ## nftables Flush Ruleset on Remote Hosts On remote hosts, `nftables flush ruleset` followed by a failed rule load leaves the host with NO firewall. SSH survives only on existing connections — new connections are blocked or allowed depending on the default policy. **Always validate rules before applying:** `nft -c -f ` does a dry-run parse. For extra safety, deploy a cron-based auto-rollback timer that reverts rules unless explicitly confirmed (similar to `shutdown -c` pattern). ## systemd Socket Activation Overrides Config File Ports On modern Linux systems (Ubuntu 24.04+), systemd socket activation controls the listening port for services like SSH. Editing the service config file alone (e.g., `sshd_config Port 2222`) has no effect — the socket unit still binds the original port. **Check socket activation first:** `systemctl cat .socket` shows whether socket activation is in play. If so, override the socket unit's `ListenStream` directive, not the service config. ## Docker Sets iptables FORWARD Policy to DROP Docker sets the iptables FORWARD chain default policy to DROP. This affects ALL forwarding on the host, not just Docker traffic. Non-Docker forwarding (VPN, VM bridges, custom NAT) silently breaks. **Fix:** Add explicit ACCEPT rules in the `DOCKER-USER` chain for non-Docker forwarding needs. This chain is processed before Docker's own rules and persists across Docker restarts. ## HTTP Host Header vs TLS SNI Are Different Layers When proxying to a backend over HTTPS, two independent identifiers must be set correctly: - **TLS SNI** — sent during the TLS handshake, used for certificate selection. Missing SNI causes `x509: cannot validate certificate for `. - **HTTP Host header** — sent after TLS is established, used for virtual host routing. Missing or wrong Host header causes 404 from the backend. A reverse proxy must set both. They often need to be the same value, but they're configured independently. ## Wildcard Certs in Auto-Renewing Proxies Auto-renewing proxies (Caddy, Traefik with Let's Encrypt, etc.) that also support file-loaded certificates treat file-loaded certs as globally available. A wildcard cert loaded for one site block will match ALL matching subdomains, silently preventing automatic certificate issuance for other sites. **Rule:** Use automatic certificate management for all sites. Don't mix file-loaded and automatic certs unless you understand the matching priority. ## Reverse Proxies Ignore Labels on Stopped Containers Docker-label-based routing (Traefik, Caddy-docker-proxy, nginx-proxy) silently drops routes whose target containers are not running. Flags like `allowEmptyServices` do not help — the router only sees the labels of *running* containers. This breaks on-demand and "scale-to-zero" backends: the proxy has no route to the stopped container, so the wake-up request never reaches whatever is meant to start it. Requests 404 (or worse, go to the wrong backend) until the container happens to be up. **Rule:** For any backend that may not always be running, declare the route in **dynamic file config**, not container labels. File-config routes exist regardless of container state — the router can then proxy to a "wake" handler, return a holding page, or queue the request. **Validation:** always test routing with the backend container **stopped**, not just running. If the route disappears when the container stops, the config is wrong for on-demand use. ## Cilium DNAT Resolves LB VIP Before NetworkPolicy Evaluation Cilium performs DNAT on LoadBalancer VIP traffic before evaluating NetworkPolicy. Traffic to a VIP is rewritten to a backend pod IP before the policy check. For egress to LoadBalancer services in CiliumNetworkPolicy, use `toEndpoints` targeting the backend pods (by namespace/label), not `toCIDR` targeting the VIP. ## Don't Use ICMP/ping to Test eBPF LoadBalancer VIPs Cilium (and other eBPF LB datapaths) only program TCP/UDP for a LoadBalancer VIP; ICMP is not handled by the eBPF LB and falls through to the kernel, which can generate redirect loops. Symptom: `ping ` returns "TTL exceeded" or "connection refused" even from a host on the same network, tempting a false "the load balancer is broken" conclusion — while TCP/HTTPS to the same VIP works fine. **Rule:** test service-VIP reachability with `curl`/`nc` (TCP), never `ping`. A failed ping to an eBPF LB VIP is meaningless. ## VPN Subnet/Exit Routing Needs Return-Path SNAT For any mesh/overlay VPN (Tailscale/Headscale, WireGuard subnet routers) that advertises LAN subnets or acts as an exit node, outbound reachability proves nothing on its own. LAN hosts have no route back to the VPN's client address range (e.g. Tailscale's `100.64.0.0/10` CGNAT range), so return traffic is silently dropped — the symptom is partial connectivity where some destinations answer and others don't. **Fix:** MASQUERADE/SNAT traffic sourced from the VPN client range out each physical LAN interface on the router node, and persist it (`netfilter-persistent` or equivalent). **When testing any VPN or subnet-routing change, verify both directions** — confirm the SNAT rule exists before concluding a route works. Outbound success alone is a false positive. ## VLAN-Aware Bridge vs Raw Sub-Interface Coexistence (Proxmox/Linux bridges) On a host that runs BOTH a VLAN-aware bridge (`vmbr0` on a bond/uplink) AND raw `vlanN@bondX` sub-interfaces on the same uplink, the kernel delivers tagged frames for VLAN N to the raw sub-interface, NOT to the vlan-aware bridge's VLAN handling. A VM NIC attached as `vmbr0,tag=N` then receives nothing. Symptom: guest has the right IP/routes and the switch port is healthy, but there is zero L2 connectivity (no DHCP lease → APIPA `169.254.x`, gateway un-ARPable) — the guest config looks perfect while frames go nowhere. **Rules:** - A genuinely-tagged VLAN that also has a raw `vlanN` sub-interface must attach to its **dedicated named bridge** (whose port is `vlanN`), never `vmbr0,tag=N`. - A VM on the trunk's **native/untagged** VLAN must use untagged `vmbr0` (no `tag=`); a `tag=` NIC emits tagged frames that don't match the native VLAN and are dropped both directions. - For a multi-VLAN VM, one NIC per VLAN on per-VLAN bridges is more reliable than Proxmox `trunks=` (which silently fails to route on some `vmbr0` configs). **Diagnostic shortcut:** a guest with correct IP/routes but unreachable from a same-subnet host is an L2/VLAN problem, not guest config. Check which bridge the tap landed on (`ls /sys/class/net//brif/`, `bridge vlan show dev `) and whether the VLAN is native(untagged) vs tagged on the uplink. ## Forward-Auth Proxy Redirects Break CORS Preflight (302 on OPTIONS → ERR_INVALID_REDIRECT) A forward-auth layer (Authelia, oauth2-proxy, etc.) sitting in front of a backend intercepts *unauthenticated* requests and 302-redirects them to its login portal — including CORS OPTIONS preflight requests. The browser reports `ERR_INVALID_REDIRECT` (not `ERR_FAILED`), pointing at a 302 whose target is the auth portal. **This is an auth-bypass gap, not a CORS bug.** The backend's CORS middleware handles OPTIONS correctly once the auth layer stops intercepting. Do not touch the CORS config. - **Signature:** `ERR_INVALID_REDIRECT` + a 302 whose `Location` is the auth portal ⇒ bypass gap. A true CORS bug shows `ERR_FAILED` / missing `Access-Control-*` headers with no redirect. - **Fix:** add the endpoint path to the auth layer's bypass/allow list, then reload the auth layer. - **Bypass rules are path-scoped.** Every new backend endpoint needs its own bypass entry — one that works does not cover a sibling. Enumerate all public paths and confirm each is bypassed, including root-level paths a client library may call outside the expected prefix (e.g. `/schemas`, `/health`). ## `tlsv1 alert internal error` = DNS Resolves to the Ingress, but No Route Matches the Host A domain resolves to the ingress IP, but `https://` aborts during the TLS handshake with `tlsv1 alert internal error`. Root cause: the ingress controller (Traefik, and equivalents) has no route matching `Host()` — only related hosts exist (a different subdomain, a sibling domain). With no matching route there is no certificate to present, so the handshake aborts before any HTTP layer. This is the expected "nothing is served here" behaviour, not a cert or reverse-proxy misconfiguration. - Diagnostic: if `curl -v https://` fails at TLS but a known-good sibling host on the same ingress succeeds, suspect a missing route, not a broken cert. - Fix (if the site should be served): add an ingress route/IngressRoute matching that exact Host. Otherwise it is working as intended.