Load Balancing

Load balancing maps demand to healthy capacity while preserving transport and application semantics. Placement, health, retry, and overload policy form one system: an algorithm cannot recover capacity that health checks misclassify or retries consume.

Placement Layers #

DNS placement returns regional or service addresses. Resolver and client caching makes changes coarse-grained, so DNS cannot drain individual connections or react to per-request load.
Anycast placement advertises the same address from several sites and lets routing select a path. Route changes can move flows, and the nearest network path is not necessarily the site with spare application capacity.
Transport load balancing selects a backend from TCP, UDP, or QUIC flow metadata without interpreting the application request. Direct server return and connection tracking affect topology and failure behavior.
Reverse proxies terminate an application protocol and can route HTTP by authority, path, headers, method, or cookies. Termination enables retries and observability but consumes proxy capacity and creates another failure domain.
Application placement assigns work using domain state, such as tenant, shard ownership, queue partition, or accelerator availability. The application must then handle ownership changes and stale routing metadata.

Large systems combine layers: DNS or anycast chooses a site, an L4 load balancer chooses a proxy, and the proxy chooses a workload.

Selection Algorithms #

Round robin fits endpoints with similar capacity and request cost. Weighted round robin reflects known capacity differences. Least-connections uses active connection count as a load signal, but long-lived idle connections and multiplexed HTTP/2 or HTTP/3 streams can make one connection an inaccurate unit. Least-request or latency-aware variants need stable measurements and damping to avoid sending every request toward a temporarily fast endpoint.

Hashing by a stable key provides affinity. Consistent or rendezvous hashing reduces remapping when membership changes, but it can concentrate a hot key and does not eliminate failover. Locality-aware algorithms prefer the same zone or region to reduce latency and transfer cost while retaining spillover capacity for a local shortage.

Health, Outliers, and Draining #

An active health check sends synthetic probes. A passive check observes real connection errors, resets, timeouts, or status codes. Readiness should verify the dependency boundary needed to serve traffic without turning every transient downstream failure into fleet-wide removal.

Use consecutive-success and consecutive-failure thresholds to avoid flapping. Outlier detection temporarily ejects an endpoint whose observed behavior differs from peers; bound the ejected percentage so one shared dependency failure does not remove the entire service.

Draining removes an endpoint from new selection while allowing established requests or connections to finish within a deadline. Coordinate load-balancer drain time, application shutdown hooks, orchestration readiness, and the protocol’s keep-alive behavior.

Affinity, Retries, and Failover #

Cookie, source-address, or hash affinity can improve cache locality or retain legacy session state. Affinity reduces balancing freedom and must define what happens when the selected backend disappears. Prefer external or replicated session state when requests need transparent failover.

A timeout means the caller did not receive a result; it does not prove the backend performed no work. Retry only within the caller’s deadline and a bounded retry budget. Safe and idempotent operations are candidates; non-idempotent operations require an application idempotency key and durable deduplication. Spread retries with backoff and jitter, and avoid retrying overload responses onto an already saturated peer set.

Cross-zone or cross-region failover needs reserved capacity, compatible data, and an explicit consistency model. A routing switch alone cannot make a remote site ready to accept writes.

Capacity and Overload #

Endpoint count is not capacity. Track concurrent requests, queue depth, utilization of the constrained resource, latency percentiles, errors, and admission rejections. Autoscaling reacts after measurement and provisioning delay, so preserve headroom for bursts and failures.

When demand exceeds capacity, bound queues, shed lower-priority work, enforce per-tenant limits, and return a clear retry signal where the protocol supports one. Backpressure should reach callers before memory or connection pools are exhausted. Load-balancer metrics must distinguish no healthy endpoint, local proxy saturation, backend overload, and policy rejection because each requires a different response.