Load balancing
- Lesson 5 of 40
- 12 min
- Beginner
- 12 steps
Having several servers is not enough: something has to decide where each request goes. You will see what a load balancer does, which rules it uses and what happens when something fails.
- Users
- 200,000
- Requests/s
- 6,000 req/s
- p95 latency
- 80 ms
- Errors
- 0%
- Healthy servers
- 3
- request
- slow request
- waiting in queue
- health check
Step 1 of 12
Problem
Three servers, all traffic on one
We now have three servers, but the app's DNS only publishes the IP of Server B. All 200,000 users land there while A and C sit idle.
In this example each server handles about 5,500 req/s. 6,000 arrive, so B rejects around 500 per second (≈ 8 %) and the rest wait in line.
Step 2 of 12
Solution
One front door for the whole pool
We add a load balancer (LB). Its IP is the only one clients know: it takes each request, picks a server from the pool, hands the request over and returns the response.
The 6,000 req/s are now shared: about 2,000 per server, around 36 % of its capacity. And since clients never see the servers, you can add or remove one without telling anybody.
What changed
- Load balancer
- Clients talk to the LB
- CPU: 2 / 100 / 2 % → ≈ 36 % each
- Errors: 8 % → 0 %
Step 3 of 12
Technique
Round robin: everyone takes a turn
Who does the LB give each request to? The simplest rule is round robin: the first to A, the second to B, the third to C, then start again.
It needs no knowledge of the servers, so it costs almost nothing. It works well when every server is the same and every request costs about the same.
What changed
- Strategy: round robin
- Slower animation so you can follow each turn
Step 4 of 12
Problem
Not every request weighs the same
In real life, opening a profile takes milliseconds and building a yearly report takes seconds. Round robin shares out requests, not work.
Here one request in three is heavy (the slow dots). With three servers and fixed turns, every heavy one ends up on the same server: B saturates and builds a queue while A and C barely work.
What changed
- Heavy requests
- B's CPU: 37 % → 95 %
- p95: 80 ms → 1.4 s
Step 5 of 12
Technique
Least connections: to whoever is freest
With least connections, the LB counts how many requests each server has open and sends the new one to the server with the fewest.
A heavy request keeps its connection open longer, so the server handling it gets fewer new requests until it finishes. The work evens out by itself.
What changed
- Strategy: least connections
- B's CPU: 95 % → 55 %
- p95: 1.4 s → 150 ms
Step 6 of 12
Technique
Health checks: drop whoever stops answering
The LB probes each server every few seconds, for example with GET /health (the rings). If one fails several times in a row, it stops getting traffic; once it answers properly again, it is added back.
Server C just went down and nobody noticed: A and B absorb its 2,000 req/s and move to about 3,000 each, around 55 %.
What changed
- Server C down
- Health checks
- C leaves the pool
- Errors: still 0 %
Step 7 of 12
Problem
What if the balancer is the one that fails
We removed one single point of failure and added another: now every request goes through one LB. If it fails, everything fails.
The three servers are healthy and idle. Nobody can reach them: requests get no answer until they time out.
What changed
- Load balancer down
- Errors: 0 % → 100 %
Step 8 of 12
Solution
A second balancer on standby
We run two balancers: an active one that serves traffic and a passive one on standby. Clients point at neither; they use a virtual IP (VIP) held by the active one.
They send each other a heartbeat about once per second. As long as the passive one keeps receiving them, it knows the active one is alive.
What changed
- Standby load balancer
- Heartbeats between the two
- The active one holds the VIP
Step 9 of 12
Solution
Taking over in seconds
The active one stops responding. When the heartbeats stop, the standby takes the VIP and starts sharing out the traffic.
Clients keep using the same address and change nothing. At worst, requests that were halfway through fail and get retried.
What changed
- Active load balancer down
- The standby becomes active
- The VIP moves to the other machine
Step 10 of 12
Technique
Layer 4 or layer 7: looking inside the request
A layer 4 balancer only sees IPs and ports, and spreads whole connections without reading them. A layer 7 one understands HTTP and reads the request.
So it can decide by path: /api/* goes to the API servers and /images/* to a dedicated image service. Each pool grows and ships on its own.
What changed
- The LB decides by URL path
- Server C → image service
Step 11 of 12
Technique
Shipping a version without an outage
Version 2 is ready. A rolling deploy updates one server at a time: the LB drains it (no new requests, the ones in flight may finish), it gets updated, passes its health check and rejoins the pool.
C is already on v2, B is draining and A waits its turn. Meanwhile A and C share the 6,000 req/s and nobody sees an error.
What changed
- C updated to v2
- B draining before its update
- A still on v1
Step 12 of 12
Summary
Summary
A load balancer gives a pool of servers one stable address.
Round robin is simple; least connections adapts when work is uneven.
Health checks take failing servers out of the pool with no one stepping in.
The LB itself must be duplicated: active-passive with a shared virtual IP.
Layer 7 routes by path, and draining servers lets you deploy without outages.
← → step · R replay · Space pause
You will learn
- Why a pool of servers needs a single entry point (covered)
- Round robin versus least connections (covered)
- Health checks that route around dead servers (covered)
- Duplicating the balancer with a virtual IP (covered)
- Layer 7 routing and deploys without outages (covered)