Load balancing

  • Lesson 5 of 40
  • 12 min
  • Beginner
  • 12 steps

Having several servers is not enough: something has to decide where each request goes. You will see what a load balancer does, which rules it uses and what happens when something fails.

Users
200,000
Requests/s
6,000 req/s
p95 latency
80 ms
Errors
0%
Healthy servers
3
200,000 usersLoadbalancerLoad balancerStandby LBStandby LBServer AAPI AServer AServer BAPI BServer BServer CServer CImagesImages2%100%2%DNS only points here36%37%35%single public IP36%37%35%round robin: A → B → C24%95%25%round robinqueue growing48%55%50%least connections55%55%out of the pool2%2%2%down36%37%35%active · VIPstandby36%37%35%now active · VIP40%42%30%layer 7: routes by path/api/*/images/*55%55%v1 · waits its turndraining → v2v2 · readyrolling deploy36%37%35%VIP
  • request
  • slow request
  • waiting in queue
  • health check

Step 1 of 12

Problem

Three servers, all traffic on one

We now have three servers, but the app's DNS only publishes the IP of Server B. All 200,000 users land there while A and C sit idle.

In this example each server handles about 5,500 req/s. 6,000 arrive, so B rejects around 500 per second (≈ 8 %) and the rest wait in line.

Step 2 of 12

Solution

One front door for the whole pool

We add a load balancer (LB). Its IP is the only one clients know: it takes each request, picks a server from the pool, hands the request over and returns the response.

The 6,000 req/s are now shared: about 2,000 per server, around 36 % of its capacity. And since clients never see the servers, you can add or remove one without telling anybody.

What changed

  • Load balancer
  • Clients talk to the LB
  • CPU: 2 / 100 / 2 % → ≈ 36 % each
  • Errors: 8 % → 0 %

Step 3 of 12

Technique

Round robin: everyone takes a turn

Who does the LB give each request to? The simplest rule is round robin: the first to A, the second to B, the third to C, then start again.

It needs no knowledge of the servers, so it costs almost nothing. It works well when every server is the same and every request costs about the same.

What changed

  • Strategy: round robin
  • Slower animation so you can follow each turn

Step 4 of 12

Problem

Not every request weighs the same

In real life, opening a profile takes milliseconds and building a yearly report takes seconds. Round robin shares out requests, not work.

Here one request in three is heavy (the slow dots). With three servers and fixed turns, every heavy one ends up on the same server: B saturates and builds a queue while A and C barely work.

What changed

  • Heavy requests
  • B's CPU: 37 % → 95 %
  • p95: 80 ms → 1.4 s

Step 5 of 12

Technique

Least connections: to whoever is freest

With least connections, the LB counts how many requests each server has open and sends the new one to the server with the fewest.

A heavy request keeps its connection open longer, so the server handling it gets fewer new requests until it finishes. The work evens out by itself.

What changed

  • Strategy: least connections
  • B's CPU: 95 % → 55 %
  • p95: 1.4 s → 150 ms

Step 6 of 12

Technique

Health checks: drop whoever stops answering

The LB probes each server every few seconds, for example with GET /health (the rings). If one fails several times in a row, it stops getting traffic; once it answers properly again, it is added back.

Server C just went down and nobody noticed: A and B absorb its 2,000 req/s and move to about 3,000 each, around 55 %.

What changed

  • Server C down
  • Health checks
  • C leaves the pool
  • Errors: still 0 %

Step 7 of 12

Problem

What if the balancer is the one that fails

We removed one single point of failure and added another: now every request goes through one LB. If it fails, everything fails.

The three servers are healthy and idle. Nobody can reach them: requests get no answer until they time out.

What changed

  • Load balancer down
  • Errors: 0 % → 100 %

Step 8 of 12

Solution

A second balancer on standby

We run two balancers: an active one that serves traffic and a passive one on standby. Clients point at neither; they use a virtual IP (VIP) held by the active one.

They send each other a heartbeat about once per second. As long as the passive one keeps receiving them, it knows the active one is alive.

What changed

  • Standby load balancer
  • Heartbeats between the two
  • The active one holds the VIP

Step 9 of 12

Solution

Taking over in seconds

The active one stops responding. When the heartbeats stop, the standby takes the VIP and starts sharing out the traffic.

Clients keep using the same address and change nothing. At worst, requests that were halfway through fail and get retried.

What changed

  • Active load balancer down
  • The standby becomes active
  • The VIP moves to the other machine

Step 10 of 12

Technique

Layer 4 or layer 7: looking inside the request

A layer 4 balancer only sees IPs and ports, and spreads whole connections without reading them. A layer 7 one understands HTTP and reads the request.

So it can decide by path: /api/* goes to the API servers and /images/* to a dedicated image service. Each pool grows and ships on its own.

What changed

  • The LB decides by URL path
  • Server C → image service

Step 11 of 12

Technique

Shipping a version without an outage

Version 2 is ready. A rolling deploy updates one server at a time: the LB drains it (no new requests, the ones in flight may finish), it gets updated, passes its health check and rejoins the pool.

C is already on v2, B is draining and A waits its turn. Meanwhile A and C share the 6,000 req/s and nobody sees an error.

What changed

  • C updated to v2
  • B draining before its update
  • A still on v1

Step 12 of 12

Summary

Summary

A load balancer gives a pool of servers one stable address.

Round robin is simple; least connections adapts when work is uneven.

Health checks take failing servers out of the pool with no one stepping in.

The LB itself must be duplicated: active-passive with a shared virtual IP.

Layer 7 routes by path, and draining servers lets you deploy without outages.

You will learn

  • Why a pool of servers needs a single entry point
  • Round robin versus least connections
  • Health checks that route around dead servers
  • Duplicating the balancer with a virtual IP
  • Layer 7 routing and deploys without outages