Load Balancing Strategies: Round Robin, Least Connections, Consistent Hashing
Fast recall
Quick revision
Core answer
Algorithm vocabulary, trade-offs, L4 vs L7 distinction, and the role of health checks.
Defaulting to round-robin everywhere (poor fit for variable-cost requests), or assuming sticky sessions are required for stateful apps (usually a sign the app should be made stateless).
Discuss power-of-two-choices (P2C) as a near-optimal alternative to least-connections at scale, slow-start for new instances, and outlier detection in service mesh proxies.
Interview-ready guide
Interview answer
Match algorithm to traffic shape
- Stateless, uniform requests — round-robin
- Heterogeneous backends — weighted round-robin
- Variable-cost requests (uploads, long-poll) — least connections or P2C
- Latency-sensitive — least response time or P2C
- Cache affinity / sticky session — consistent hashing
- Massive fleets (1000+) — P2C, never global least-connections
Round-robin and weighted RR
RR: trivial, no state. Blind to actual load. OK for uniform. WRR: handles heterogeneous backends (instance types, phased rollouts). Still blind to current load. Smooth WRR (NGINX): spreads picks evenly over time, not in bursts.
Least connections and least response time
Least connections: route to backend with fewest active connections. Naturally handles variable-cost — a backend doing a long upload accumulates connections, gets less new traffic. Needs per-backend state at the LB.
Least response time: exponentially smoothed average; directly optimizes for user-visible latency. Can ping-pong if noisy.
Power-of-two-choices — modern workhorse
Problem with least-connections at scale: global per-backend state across hundreds of LB instances is expensive and stale.
P2C trick: pick two random backends, route to the one with fewer connections. Math from queueing theory says this is exponentially better than random and only marginally worse than full least-connections, with O(1) decision and no global state.
Used by HAProxy, Envoy default LEAST_REQUEST, Linkerd, Finagle. My default for L7 in modern systems.
Hash-based and consistent
Hash-based: backend = hash(key) mod N. Same client → same backend. Issue: N changes reshuffle everything.
Consistent hashing: only K/N keys move on backend change. Use for cache affinity that survives autoscaling/deploys. Envoy RING_HASH, NGINX hash consistent.
Maglev: precomputed table, bounded disruption, O(1) lookup. Google's L4 LB.
Bounded-load consistent hashing: caps per-backend load at (1+ε) × average; spills excess. Combines affinity with hard load guarantees.
Sticky sessions = hash on cookie or IP. Often a smell — the app should be stateless. Use only when in-memory session can't move to Redis, WebSocket, or migration constraints.
L4 vs L7
L4 (TCP): connection metadata only, no TLS termination, no HTTP awareness. Extremely fast. AWS NLB, Google Maglev.
L7 (HTTP): understands path, header, cookie. Terminates TLS. Can retry, route, manipulate headers. AWS ALB, Envoy, NGINX, HAProxy.
Common topology: L4 at edge (TLS pass-through, DDoS) → L7 behind (per-route routing, retries) → backends.
gRPC: L4 doesn't balance well because gRPC multiplexes calls over one TCP connection. Use L7 with HTTP/2 awareness (Envoy) or client-side LB (gRPC's xDS).
Health checks
- Active: periodic /healthz probe; ejects unhealthy
- Passive (outlier detection): observes failed responses; ejects high-error backends
- Use both — active for degradation, passive for issues active misses
- Use readiness, not liveness — readiness means 'can serve traffic now'
- Frequency: 5-15s active typical; instantaneous passive
Slow start
Fresh backend has empty caches. Without ramp-up, P2C/least-conn sends a flood, cold cache crashes. Slow start ramps traffic over 30s. Envoy, AWS ALB, NGINX Plus support it.
Recommendation by workload
- Stateless uniform — round-robin or P2C
- Variable-cost — least connections or P2C
- Latency-sensitive — least response time or P2C
- Cache affinity — consistent hashing
- Heterogeneous — weighted variant
- Geo-distributed — DNS/anycast routing first, LB second
- gRPC — L7 (Envoy)
- Massive fleets — P2C, never global state
Recommendation
For most modern microservices: L4 at the edge (NLB or similar for TLS pass-through and DDoS), L7 behind it (Envoy or ALB) using P2C as the default algorithm with active + passive health checks and slow-start for new instances. Use consistent hashing only when cache affinity or sticky sessions are genuinely needed. Sticky sessions are usually a sign the app should be stateless — push session state to Redis and use round-robin or P2C instead.
Load Balancing Strategies: Round Robin, Least Connections, Consistent Hashing
Direct Answer
Pick by traffic shape: round-robin for stateless, similar-cost requests; weighted round-robin when nodes have different capacity; least connections when request duration varies (long-poll, file uploads); least response time for latency-sensitive APIs; consistent hashing for cache affinity or sticky sessions; random / power-of-two-choices for very large fleets where global state is expensive. Layer 4 (TCP) load balancing is faster but blind to HTTP; layer 7 (HTTP) understands paths and headers and enables routing, retries, and TLS termination.
Common Mistakes & Pitfalls
Defaulting to round-robin everywhere (poor fit for variable-cost requests), or assuming sticky sessions are required for stateful apps (usually a sign the app should be made stateless).
Check your understanding
Answer these without looking back at the article.
- Walk through what happens to a backend with cold caches when it joins a P2C-balanced fleet without slow-start.
- Compare consistent hashing vs Maglev hashing for an L4 load balancer.
- How do you handle gradual rollouts / canary deploys at the load balancer level?
- Design the load balancing for a system that does both REST API and gRPC traffic.
Related interview prep categories
2Want a guided curriculum order?
These modules are also sequenced in our flagship backend interview track — open it when you prefer an ordered path instead of browsing by topic.