Rate limiting
- Lesson 25 of 40
- 10 min
- Intermediate
- 9 steps
One client gone wild can leave everyone else without service. You will see how to share capacity with per-client limits and what to do when even that is not enough.
- Buggy script
- 10 req/s
- Users served
- 100%
- API CPU
- 40%
- Rejected at the gate
- 0 req/s
- request
- slow request
- waiting in queue
- failed request
Step 1 of 9
Problem
One hasty script, everyone waiting
A customer deploys a script with a bug: it calls your API inside a loop with no pause, 4,000 requests per second. Almost all of the API's CPU goes into serving it.
Real users' requests wait behind that flood and many end in a timeout. Nobody is attacking you: a single client is taking the capacity that belonged to everyone.
Step 2 of 9
Solution
A doorman at the entrance
Put a rate limiter in a gateway in front of the API. Each client is identified (by API key, user id or IP) and gets a budget: in this example, 10 requests per second.
Whatever fits in the budget goes through; the rest is turned away on the spot with 429 Too Many Requests, without touching the API. The script loses 3,990 of its 4,000 requests per second.
What changed
- Gateway with a rate limiter
- Script limited to 10 req/s
- API CPU: 98% → 40%
- Users served: 58% → 100%
Step 3 of 9
Technique
How it counts: the token bucket
The most widely used algorithm is the token bucket. Each client has a bucket of up to 10 tokens that refills at 10 per second. Every request spends one token; with the bucket empty, the answer is 429.
A normal user hardly ever empties it: right now they have 8 of 10 left. The script drains it the moment new tokens arrive, so almost everything it sends is turned away.
What changed
- One bucket per client
Step 4 of 9
Technique
Windows versus buckets
A fixed window counts requests per clock minute and goes back to zero on every exact minute. It is simple, but it has an edge: with a limit of 60 per minute, a client can send 60 at 0:59 and another 60 at 1:00, 120 in about 2 seconds.
A sliding window always looks at the last 60 seconds and removes that edge. The token bucket allows controlled bursts. That is why gateways usually offer one of these two.
Step 5 of 9
Problem
Two gateways, one budget
One gateway is no longer enough, so you put two behind a load balancer. If each counts on its own, a client that spreads its requests over both gets twice its limit; with 10 gateways, ten times.
The way out is to keep the counters in a fast shared store, such as Redis: each request does an atomic increment (INCR) and compares the total with the limit.
What changed
- Second gateway
- Shared counter store
- Each client's counters live in the store
Step 6 of 9
Technique
Tell the client when to come back
A good 429 does not just say no: it says when to come back with the Retry-After: 30 header (in seconds) and usually reports the limit and what is left with headers such as X-RateLimit-Remaining, widely used though not standard.
A well-built client waits that long before retrying. The script's owner fixes the loop and it now uses 10 req/s, its fair share, without a single rejection.
What changed
- Retry-After and limit headers
- The script drops to 10 req/s
- Rejected: 3,990/s → 0
Step 7 of 9
Problem
Case: one company with 8,000 users
You sell your API to companies. One of them runs a nightly import with 8,000 users; none goes over their personal limit, but together they add up to 2,400 req/s, 60% of a 4,000 capacity (example values). Every night, every other customer gets slow.
A per-user limit cannot see the problem. Add a per-tenant quota (the company account) based on its plan, on top of the per-user limit: with a 1,500 req/s quota, the 900 extra get a 429.
What changed
- Per-tenant quota based on the plan
- The import stays at 1,500 req/s
Step 8 of 9
Technique
When everyone arrives at once: shed load
Limits stop one client from hogging. But on launch day everyone arrives at once, each within their limit, and the total exceeds what you can serve: 6,000 req/s for a capacity of 4,000 (example values).
That is where load shedding comes in: the gateway drops background work first (prefetching, recommendations, analytics), 2,400 req/s. That leaves 3,600, 90% of capacity, and logins and payments keep working.
What changed
- Peak of real users
- The gateway drops low-priority work
- Logins and payments: 100% served
Step 9 of 9
Summary
Summary
Give each client a budget and turn the excess away early, with a 429.
The token bucket tolerates short bursts and caps the average; the sliding window avoids the fixed window's edge.
With several gateways, keep the counters in a shared store.
Stack limits per user, per tenant and per endpoint, and use Retry-After to say when to come back.
If the whole system is saturated, drop low-priority work first.
← → step · R replay · Space pause
You will learn
- Why a single client can leave everyone else without service (covered)
- How the token bucket and windows count (covered)
- How to share one limit across several gateways (covered)
- 429 responses, retries and per-tenant quotas (covered)
- Shedding load when everyone arrives at once (covered)