Redundancy and failover
- Lesson 24 of 40
- 12 min
- Intermediate
- 12 steps
Every component fails sooner or later: disks, servers, whole buildings and, above all, your own deploys. The question is not whether it will happen, but what stops working when it does.
- Availability
- 99.99%
- Downtime per year
- 53 min
- Single points of failure
- 0
- Survives losing
- a zone
- request
- response
- replication
- failed request
Step 1 of 12
Concept
How much can a stack of single parts take?
We start with the bare minimum: a load balancer, an application server and a database. Say (example value) each one works 99.9% of the time.
Every request needs all three at once, so their availabilities multiply: 0.999 × 0.999 × 0.999 ≈ 0.997. That is 99.7%, about 26 hours without service a year.
Step 2 of 12
Problem
Any part can bring it all down
A single point of failure (SPOF) is any part that, failing on its own, leaves the system without service. In this stack all three are.
One damaged disk, one hung process or one badly applied change in any of them is enough for a million people to see an error page.
What changed
- Three single points of failure marked
Step 3 of 12
Solution
Redundancy: a spare for every part
Double every part. The load balancer becomes an active-passive pair: if the active one dies, the other takes over its address. Two application servers share the traffic, and the leader database replicates every change to a follower ready to step in.
Now both copies have to fail at once: 0.1% × 0.1% = 0.0001%. Losing a single machine is no longer news.
What changed
- Active-passive load balancer pair
- Second application server
- Follower database
- Single points of failure: 3 to 0
Step 4 of 12
Problem
Copies that go down together
All those copies live in the same data center. A power cut (or a flood, a severed fibre cable, a cooling failure) switches off the whole building and, with it, every “redundant” copy.
When one cause takes out many parts at once we call them correlated failures. Redundancy only helps if the copies fail independently.
What changed
- Everything in one data center
- The data center loses power
- Every copy goes down at once
Step 5 of 12
Solution
Spread across availability zones
Cloud providers split each region into availability zones: data centers kilometres apart, with their own power and network, joined by low-latency links.
Place one application server and one database copy in each zone. A disaster that takes a whole zone now takes only half.
What changed
- A single data center
- Zone A and zone B
- One copy of each tier per zone
Step 6 of 12
Solution
Zone A goes dark: failover
Zone A loses power. Health checks stop answering, the load balancer stops sending it traffic and the follower in zone B is promoted to leader.
Anyone in the middle of a request may see an error and retry; within about a minute everything runs from zone B.
What changed
- Zone A out of service
- All traffic goes to zone B
- The follower becomes leader
Step 7 of 12
Technique
How long does it take, and how much is lost?
Two numbers sum up a failover. RTO (recovery time objective): how long until the service is back. RPO (recovery point objective): how much recent data you can afford to lose.
With asynchronous replication the follower may be a few seconds behind, and those writes are lost. Synchronous replication brings RPO to zero, in exchange for slower writes.
Step 8 of 12
Concept
A failover that is never tested does not work
A fallback path that never runs decays quietly: config drifts, the spare falls short, a script has a bug. You find out in the middle of a real outage.
That is why some teams break things on purpose, in production and during working hours: chaos engineering. Here a test switches off an application server and the team watches how the system copes.
What changed
- Zone A is back
- One server switched off on purpose
- Traffic keeps flowing through the other
Step 9 of 12
Problem
Case: the bug that is everywhere
A new version with a bug ships to every server in both zones at the same time. Every copy crashes on the same request: redundancy does not help when they all run the same broken code.
This is the most common cause of real outages: not hardware, but changes to code, configuration or infrastructure.
What changed
- v2.4 on every server at once
- Both application servers crash
Step 10 of 12
Solution
Canary deployments
Deploy in stages. First a canary: one server gets v2.4 and the load balancer sends it only 1% of the traffic. For a while you compare its error rate and latency with the rest.
If they get worse, you roll back and only 1% of users noticed, briefly. If all goes well, you move to 10%, 50% and 100%. Many teams also deploy one zone at a time.
What changed
- v2.4 on a single server
- 1% of the traffic
- Automatic rollback if errors rise
Step 11 of 12
Technique
Beyond zones: several regions
Zones protect you from a building going down. A whole region can go down too (a regional network or cloud control-plane failure), and it is also far from users on other continents.
With two regions and a global DNS (or anycast) in front, each user goes to the nearest healthy region. The hard part is the data: crossing an ocean adds around 100 ms of lag to replication.
What changed
- Second region
- A global DNS picks the nearest healthy region
- Cross-region replication: ~100 ms of lag
Step 12 of 12
Summary
Recap
Parts in a chain multiply their availability.
Remove single points of failure with redundant copies.
Copies must fail separately: spread them across zones.
Set RTO and RPO, and rehearse failover so it works when needed.
Most outages come from changes: deploy with canaries.
← → step · R replay · Space pause
You will learn
- Measure availability in nines (covered)
- Spot and remove single points of failure (covered)
- Spread copies across zones that fail separately (covered)
- Set RTO and RPO and rehearse failover (covered)
- Contain bad deploys with canaries and regions (covered)