Vertical or horizontal
- Lesson 4 of 40
- 11 min
- Beginner
- 11 steps
Your app lives on a single server and traffic keeps growing. There are two ways to get more capacity, and each one has its price.
- Users
- 500,000
- Requests
- 20,000 req/s
- p95 latency
- 95 ms
- Errors
- 0%
- Cost $/month
- 14,400
- request
- response
- waiting in queue
Step 1 of 11
Concept
One machine does everything
At first your app runs on a single server: every time someone opens a screen or saves something, the request leaves their device, reaches that machine and comes back with the response.
With 100 users online there are about 4 requests per second. The machine has 4 vCPUs and handles about 400 req/s, so it runs at 1%.
Step 2 of 11
Problem
A video goes viral
Overnight you have 120 times more users: 12,000 online and 480 req/s. The server only handles about 400, so 80 per second are too many.
CPU is stuck at 100%, requests wait in a queue, p95 latency climbs to 2.4 s and about 17% (80 out of every 480) end in a timeout.
What changed
- Users 100 → 12,000
- CPU 1% → 100%
- Timeouts: ≈ 17%
Step 3 of 11
Solution
Path 1: a bigger machine
The most direct way out is moving to a server with more vCPUs, more memory and faster disks. Going from 4 to 16 vCPUs gives about 1,600 req/s of capacity.
The code does not change at all. This is called vertical scaling (scaling up). At 480 req/s CPU drops to 30%.
What changed
- Server 4 → 16 vCPUs
- CPU 100% → 30%
- Cost $150 → $600 a month
Step 4 of 11
Trade-off
Until there is no bigger machine
The product keeps growing to 500,000 users and 20,000 req/s. You move up a size again and again until the biggest machine in the catalogue: 192 vCPUs. On paper that is 19,200 req/s, but in practice it delivers about 17,600.
Software does not use 192 cores as well as 16: locks, shared memory and a single disk become bottlenecks. 12% of requests fail. And every size jump is a migration with a restart.
What changed
- Users 12,000 → 500,000
- Biggest machine reached
- Cost $600 → $7,200 a month
Step 5 of 11
Problem
Everything hangs on one box
There is a worse problem than the ceiling: if that machine goes down, everything goes down. A buggy deploy, a full disk or a dead power supply is enough.
A component whose failure takes the whole system down is a single point of failure (SPOF).
What changed
- App server down
- Errors 12% → 100%
Step 6 of 11
Solution
Path 2: more identical machines
Instead of one huge machine, you run several medium ones in parallel with the same code: here, four with 96 vCPUs (about 9,600 req/s each).
Need more capacity? Add another server. There is no hard ceiling and each machine is only a slice of the total. This is horizontal scaling (scaling out), and it is how almost every large system grows.
What changed
- 1 giant machine → 4 medium ones
- Cost $7,200 → $14,400 a month
- CPU ≈ 52% on each
- Errors 100% → 0%
Step 7 of 11
Problem
Peak capacity, asleep
Traffic follows people's schedules. At 8 p.m. you need all four servers; at 4 a.m. there are a tenth of the users and each one runs at 5%.
You are paying for peak capacity 24 hours a day, even though most of the day it is barely used.
What changed
- 4 a.m.: users 500,000 → 50,000
- CPU ≈ 52% → 5%
Step 8 of 11
Solution
Let the fleet follow demand
Since all the servers are the same, an autoscaler can start and stop machines for you, for example to keep average CPU near 50%.
In the small hours it keeps two (the minimum, so it survives one going down) and in the evening it goes back to four. If four run half the day and two the other half, the average cost drops to $10,800 a month.
What changed
- 2 servers stopped in the small hours
- CPU 5% → 10% on the ones left
- Average cost $14,400 → $10,800 a month
Step 9 of 11
Problem
Case: a lightning deal
At noon there are 150,000 users and two servers have room to spare (6,000 req/s, 31% CPU). A deal starts and within two minutes traffic grows 5×: 750,000 users, 30,000 req/s.
The autoscaler asks for 4 more servers, but in this example each takes about 3 minutes to boot. Meanwhile the two running ones serve 19,200 req/s: almost two thirds. The rest, 36%, fails. The fix is to scale ahead: schedule it for known events and keep headroom for surprises.
What changed
- Users × 5 at noon
- 4 servers booting (≈ 3 min)
- Errors 0% → 36% while they boot
Step 10 of 11
Problem
So who sends each request to a server?
So far each group of users points at the address of one specific server. If Server B goes down, a quarter of the users get errors even though three healthy servers sit right next to it.
Clients do not know which servers are alive or saturated. Something has to sit in front, take every request and send it to a healthy server.
What changed
- Server B down
- Errors 0% → 25%
Step 11 of 11
Summary
Up versus out
Vertical: a bigger machine. No code changes and nothing new to run, but it has a ceiling, a restart at every jump and a single point of failure.
Horizontal: more identical machines. No practical ceiling, a failure only hits a slice and you can autoscale. In exchange, servers cannot keep state and you need a load balancer.
← → step · R replay · Space pause
You will learn
- Recognise when a server is saturated (covered)
- Scale vertically and know where it ends (covered)
- Scale horizontally with several identical machines (covered)
- Autoscale with the daily rhythm and get ahead of peaks (covered)
- Understand why scaling out needs a load balancer (covered)