Latency and throughput

  • Lesson 2 of 40
  • 11 min
  • Beginner
  • 10 steps

Two questions that sound like one: how long a request takes and how many the system can handle. You will see why they differ and what happens when the server fills up.

p50 latency
120 ms
p99 latency
190 ms
Throughput
350 req/s
Utilisation
35%
7,000 usersApp serverApp serverSearchfan-outSearchEventwriterEvent writerDatabaseIndex 1Index 2Index 3Index 4Index 5Index 610%≈ 120 ms round trip70%Latency vs utilisationlatencyutilisation →70%96%Latency vs utilisationlatencyutilisation →96%70%Percentilesaverage180 mslooks healthyp50160 mshalf are fasterp99600 ms1 in 100 is slowerthe slow 1% does not show in the average35%Latency vs utilisationlatencyutilisation →35%35%Madrid≥ 90 ms more per round trip35%≈ 42 in flightLittle's lawin flight = throughput × latency350 req/s × 0.12 s ≈ 42at 1 s of latency → 35040%slow this timeWaiting for the slowesteach server is slow 1 time in 1001 server1% slow100 servers1 − 0.99¹⁰⁰ ≈ 63%35%batch: 100 rows / 50 msINSERT of 100 rowsRows per second1 row per INSERT2,000/s100 rows per INSERT50,000/seach row waits ≤ 50 ms35%Latency vs utilisationlatencyutilisation →35%
  • request
  • response
  • slow request
  • waiting in queue
  • batch

Step 1 of 10

Concept

Latency and throughput are not the same

Latency: how long one request takes to go and come back. Follow one dot from the crowd to the server and back.

Throughput: how many requests the system finishes per second. Count the dots that arrive.

Step 2 of 10

Problem

Same server, seven times the people

Now 7,000 users arrive. Throughput rises to 350 req/s because the server can still cope: in this example it tops out around 500 req/s, so it is busy 70% of the time.

Look at the server's entrance: sometimes a request arrives, finds it busy and waits. Time in the queue is latency too.

What changed

  • Users: 1,000 → 7,000
  • Throughput: 50 → 350 req/s
  • A short queue at the entrance

Step 3 of 10

Problem

Near 100%, latency explodes

A third more users and utilisation reaches 96%. Throughput barely moves (480 req/s, almost the ceiling), but latency multiplies: p99 goes from 600 ms to 4.2 s.

Requests do not arrive in a tidy line but in bursts. With headroom, a burst is served right away; without it, each burst piles onto the queue already there and takes ages to drain. That is why the curve looks like a hockey stick.

What changed

  • Users: 7,000 → 9,600
  • Utilisation: 70% → 96%
  • Queue: 2 → 6 waiting
  • p99: 600 ms → 4.2 s

Step 4 of 10

Concept

The average lies

Back to 70% utilisation. Almost every request is fast, but a few get stuck behind slow work (the dark dots). The average comes out around 180 ms and looks healthy: the few slow ones get diluted.

That is why we measure percentiles. p50 is the time half the requests stay under; p99, the time only 1 in 100 exceeds.

What changed

  • Back to 7,000 users
  • Some slow requests
  • p99: 4.2 s → 600 ms

Step 5 of 10

Solution

More headroom, less waiting

We double the server's cores. The same 350 req/s are now only 35% of its capacity: the queue disappears and both p50 and p99 drop.

Throughput does not change because demand did not change. What we gain is headroom so bursts do not pile up.

What changed

  • Server: 4 → 8 cores
  • Utilisation: 70% → 35%
  • The queue
  • p99: 600 → 190 ms

Step 6 of 10

Concept

Distance is latency too

Now the server is in Madrid and the users in Mexico City. The server did not get any slower, yet every trip takes longer: watch how long the dots take to cross.

Light in fibre covers about 200 km per millisecond. The two cities are about 9,000 km apart, so each round trip adds at least ~90 ms, and a bit more in practice because cables do not run in a straight line.

What changed

  • Server moved to Madrid
  • ≥ 90 ms per round trip
  • p50: 120 → 210 ms

Step 7 of 10

Technique

How many requests are in flight

One rule ties the three numbers together: in flight = throughput × latency. At 350 req/s and 0.12 s each, about 42 requests are half-done at any moment.

Each one holds a thread, a connection or some memory. If latency rises to 1 s at the same throughput, 350 are in flight, and a pool sized for 50 runs out.

What changed

  • ≈ 42 requests in flight

Step 8 of 10

Problem

One search, many servers

A search asks 100 index servers at once (6 shown here) and has to wait for the slowest. Each one answers quickly 99% of the time, but with 100 of them one is almost always slow.

1 − 0.99¹⁰⁰ ≈ 63%: almost two out of three searches wait for someone's p99. What is rare for one server becomes normal for the page.

What changed

  • 100 index servers (6 shown)
  • The search waits for the slowest
  • p99: 190 → 900 ms

Step 9 of 10

Trade-off

Batching: more throughput for a little latency

Writing analytics events one row at a time leaves the database at about 2,000 inserts per second in this example: almost all the cost is each statement's overhead.

If we collect events for up to 50 ms and write 100 rows per statement (the squares), throughput jumps to 50,000 rows/s. In exchange, each event waits up to 50 ms longer.

What changed

  • Batches of 100 rows every 50 ms
  • 2,000 → 50,000 rows/s
  • Each event waits up to 50 ms

Step 10 of 10

Summary

Summary

Latency is the time of one request; throughput, how many finish per second.

Near 100% utilisation latency explodes: keep headroom.

Watch p99, not the average: the tail is what users feel.

Distance sets a floor no server can remove.

In flight = throughput × latency; fan-out turns the rare into the usual and batching trades latency for throughput.

You will learn

  • Tell latency and throughput apart
  • Why latency explodes near 100% utilisation
  • Read p50 against p99
  • Why distance sets a latency floor
  • Little's law, fan-out and batching