Message queues
- Lesson 21 of 40
- 12 min
- Beginner
- 11 steps
Answer straight away, do the heavy work later, and lose no task on the way. You will see how a queue separates whoever asks for the work from whoever does it, and what to watch when something fails.
- Upload response
- 80 ms
- Jobs queued
- 60
- Workers
- 4
- Jobs lost
- 0
- request
- slow request
- waiting in queue
Step 1 of 11
Problem
Heavy work inside the request
Creators upload videos and each one has to be converted into several resolutions: about 30 seconds of CPU per video (example value). Today the web server does that conversion inside the upload request itself.
The result: the creator watches a spinner for 30 s, many uploads exceed the client's or the proxy's timeout and, while the CPU is busy converting, every other page on the site slows down too.
Step 2 of 11
Solution
A queue in the middle
Now the web server stores the file, drops a small message on a message queue ("convert video 123") and answers at once with 202 Accepted: we are on it and will let you know.
Other machines, the workers, take jobs from the queue and convert them at their own pace. The web server is the producer and the workers are the consumers: neither needs to know how many there are on the other side.
What changed
- Video queue
- 2 workers
- Upload response: 30 s → 80 ms
Step 3 of 11
Concept
A spike turns into waiting
A creator with a big following posts and uploads go up tenfold. In this example each worker converts 2 videos per minute, so 2 workers handle 4; during the spike 40 arrive per minute. The queue grows by 36 a minute: in half an hour it holds 1,080 jobs.
Without the queue, that spike would have knocked over the web servers. With it, the web tier only enqueues and keeps answering in about 85 ms; jobs wait their turn and the workers keep their pace.
What changed
- Uploads: ×10 (40 per minute)
- Jobs queued: 2 → 1,080
- Upload response: still about 85 ms
Step 4 of 11
Technique
More workers to drain it
The spike is over and 4 videos per minute arrive again. With 2 workers, 4 also leave: the queue would never go down. We start 3 more: 5 workers handle 10 per minute, the queue shrinks by 6 a minute and goes from 1,080 to 300 in about 130 minutes.
They all read from the same queue and each job is handed to only one worker at a time: the competing consumers pattern. Since workers keep no state, adding or removing them is easy, and they are often scaled automatically on queue length.
What changed
- 3 more workers (5 in total)
- Jobs queued: 1,080 → 300 and falling
Step 5 of 11
Problem
A worker dies halfway through
Worker 5 crashes. To avoid asking for jobs one by one, each worker reserves 12 in advance (example value). If the queue deletes each message as soon as it hands it out, those 12 videos vanish with the worker and their creators wait forever.
Machines crash, restart and get replaced all the time. The queue has to take for granted that it will happen.
What changed
- Worker 5 down
- 12 jobs lost
Step 6 of 11
Solution
Confirm when done
The fix: when it hands out a message, the queue does not delete it, it only hides it. When the worker finishes it sends an ack (acknowledgement) and only then is it deleted. If no ack arrives within 60 s, the message becomes visible again and another worker takes it. That timeout must be longer than a job takes (here, 30 s).
The price: a job may now run twice, for instance if the worker finishes but dies right before sending the ack. That is why jobs must be idempotent: repeating them leaves the same result.
What changed
- Ack after each job
- No ack in 60 s: the message is delivered again
- Jobs lost: 12 → 0 (back in the queue: 300 → 312)
Step 7 of 11
Technique
Poison messages and the dead-letter queue
One upload is a corrupt file. The worker that takes it fails, the message goes back to the queue, the next worker fails the same way… and so on forever, wasting workers and holding up the rest.
So queues count the attempts of each message. After, say, 3 failures, it is moved to a dead-letter queue (DLQ) for a person or a script to look at.
What changed
- Worker 5 removed
- Dead-letter queue (DLQ)
- After 3 failures, the message is set aside
- Jobs queued: 312 → 60 an hour later
Step 8 of 11
Problem
Case: messages out of order
Another queue carries account events: created, email changed, deleted. With several workers in parallel, a slow one may finish "email changed" after another has already processed "deleted", and the deleted account has data again.
Many queues guarantee no order at all, and even those that do lose it as soon as several consumers work at once.
What changed
- Events: created → email changed → deleted
- Processed: created → deleted → email changed
Step 9 of 11
Solution
Order per key with partitions
Log-based messaging systems split a topic into partitions. Messages with the same key, for instance the account id, always go to the same partition, and each partition is read by a single worker of the group, in order.
So the events of one account are processed in order, while different accounts are still processed in parallel on different partitions.
What changed
- 4 partitions by account id
- Each worker reads one partition
- Each account's events keep their order
Step 10 of 11
Technique
When the queue never drains
A queue absorbs spikes, but if for hours jobs arrive faster than they finish, it grows without end. In this example 12 arrive per minute and the 4 workers handle 8: the queue rises by 4 a minute and in 10 hours holds 2,400. At 8 per minute, the newest arrival waits about 5 hours.
Watch the age of the oldest message, not just how many there are, and apply backpressure: scale the workers, cap the queue and ask producers to slow down (or reject new jobs with 503 and Retry-After).
What changed
- Alert: oldest message older than 10 min
- Queue capped at 2,400: the web answers 503 and asks to retry in 60 s
Step 11 of 11
Summary
Summary
Move slow work out of the request: enqueue a message and answer right away.
The queue absorbs spikes; more workers drain it, as long as they take out more than comes in.
Ack when done; whatever is not acknowledged is retried, so jobs must be idempotent.
Set aside in a dead-letter queue whatever fails again and again.
Keep order per key with partitions and watch the age of the queue to push back in time.
← → step · R replay · Space pause
You will learn
- Why slow work does not belong inside a request (covered)
- Producers, queues and consumers (covered)
- How a queue absorbs a spike (covered)
- Acks, retries and dead-letter queues (covered)
- Order per key, partitions and backpressure (covered)