Timeouts and retries
- Lesson 26 of 40
- 11 min
- Intermediate
- 11 steps
When a dependency slows down or fails, the way you call it decides whether the problem stays there or takes your whole system with it. The numbers in the scene are example values.
- Charged on time
- 99%
- Payment call p99
- 180 ms
- Load on payments
- 1×
- Busy threads (of 200)
- 30
- request
- response
- slow request
- failed request
- health check
Step 1 of 11
Problem
A slow service drags another one down
The payments service usually answers in 50 ms, but today it takes 30 seconds. The orders service calls it and keeps waiting, with no time limit at all.
Every waiting call holds a thread. Orders has 200 and in under a minute all of them are stuck: it can no longer serve even the pages that never touch payments. A problem in one dependency turns into a full outage.
Step 2 of 11
Solution
Add a timeout and a fallback
Every outgoing call gets a timeout: if payments has not answered in 800 ms, orders stops waiting and frees the thread. In this example 90 % of calls arrive in time and the other 10 % are cut off.
Those 10 % get a fallback: the order is accepted as “payment pending” and charged a little later. Orders stays up for everything else.
What changed
- 800 ms timeout on payment calls
- Fallback: “payment pending”
- Busy threads: 200 → 36
- Charged on time: 2 % → 90 %
Step 3 of 11
Technique
Retry the passing glitches
Many failures last an instant: a lost packet, an instance restarting, a short spike. Try again a moment later and it usually works.
With one retry after the timeout, 8 out of 10 cut-off calls succeed on the second go: charged on time rises from 90 % to 98 % (90 + 10 × 0.8). The cost: payments gets 10 % more calls (1.1×) and p99 grows to about 950 ms (800 of waiting + about 150 for the second attempt).
What changed
- One retry on timeouts and 503s
- Charged on time: 90 % → 98 %
- Load on payments: 1× → 1.1×
Step 4 of 11
Problem
Retries turn into a storm
Now payments is truly overloaded. Every client retries three times with no pause, so each failing request turns into four: payments gets 4 times its normal load.
It can never recover: the retries themselves keep it down, and your clients behave like a denial-of-service attack. Every attempt uses up its 800 ms, so p99 reaches 3.2 s (4 × 800 ms) and the 200 threads fill up again.
What changed
- Payments overloaded
- 3 immediate retries per failure
- Load on payments: 1.1× → 4×
Step 5 of 11
Solution
Wait longer each time, with jitter
Before retrying, wait, and double the wait each time: 100 ms, then 200, then 400. That is exponential backoff. Add a random part (jitter) so thousands of clients do not retry at the same instant.
Cap the attempts and also the share of traffic that may be retries: with a retry budget of 10 %, load never goes above 1.1×. Payments gets room to recover.
What changed
- Waits of 100 → 200 → 400 ms (+ jitter)
- Retry budget: 10 %
- Load on payments: 4× → 1.1×
Step 6 of 11
Technique
Circuit breaker: stop calling for a while
Payments goes down completely. Even with a timeout, every call wastes 800 ms finding out. A circuit breaker counts failures: past a threshold it opens and rejects calls instantly, without trying them.
Every so often (here, 30 s) it lets one trial call through: that is the half-open state. If it succeeds, it closes again. Meanwhile, orders keep coming in as “payment pending”.
What changed
- Payments down
- Circuit breaker: OPEN
- A failed call takes 1 ms, not 800 ms
Step 7 of 11
Solution
Half-open, then closed
Payments recovers. The next trial call succeeds, the breaker closes and traffic comes back gradually, so the service is not crushed the moment it returns.
The “payment pending” orders are charged from a queue. None was lost and the rest of the site never stopped working.
What changed
- Payments recovered
- Breaker: HALF-OPEN → CLOSED
- Pending orders charged
Step 8 of 11
Technique
A retry must not charge twice
A tricky case: payments charges the card, but the response gets lost on the way back. Orders hits its timeout, retries… and the customer pays twice.
The fix is an idempotency key: a unique id per order that travels with every attempt, for example in an Idempotency-Key header. Payments stores the result for each key and, when a duplicate arrives, returns that result instead of charging again.
What changed
- Idempotency-Key on payment calls
- A repeated attempt returns the stored result
Step 9 of 11
Technique
Pass the deadline down
The shopper's browser gives up after 2 seconds. If orders has already spent 1.9 s, there is no point in payments starting 800 ms of work: nobody will see the answer.
So every request carries a deadline: “answer before 10:30:02.000”. Each service checks how much time is left and passes that budget on; with nothing left, it stops right away. In the scene, orders spent 0.8 s and payments has 1.2 s left.
What changed
- Deadline on every request
- Nothing starts if it cannot finish in time
Step 10 of 11
Technique
Bulkheads: one pool per dependency
Remember step 1, when slow payments grabbed all 200 threads? Orders can also protect itself by giving each dependency its own pool: for example, 40 threads for payments, 40 for inventory and 120 for everything else.
Now, when payments turns slow, only its 40 threads fill up. About 50 are busy in total (40 for payments and about 10 for the rest), and browsing, search and the cart keep working. It is the idea behind a ship's watertight compartments: the bulkhead pattern.
What changed
- One thread pool per dependency
- Slow payments fills only its 40 threads
- The rest of the site is unaffected
Step 11 of 11
Summary
Summary
Every remote call needs a timeout and a fallback.
Retry only passing errors, with exponential backoff, jitter and a cap.
A circuit breaker stops hammering a service that is down.
Anything that gets retried must be idempotent.
Propagate deadlines and isolate dependencies with bulkheads.
← → step · R replay · Space pause
You will learn
- How one slow service spreads failure (covered)
- Timeouts and fallbacks (covered)
- Retries, retry storms and backoff with jitter (covered)
- Circuit breakers and idempotency keys (covered)
- Deadlines and bulkheads (covered)