Observability
- Lesson 27 of 40
- 11 min
- Intermediate
- 9 steps
A system split across many services fails in ways nobody can see at a glance. Observability is what lets you ask it what is wrong and get an answer in minutes.
- Time to notice
- 2 min
- Time to the cause
- 5 min
- p99 latency
- 430 ms
- Signals
- metrics + logs + traces
- request
- response
- slow request
- failed request
Step 1 of 9
Problem
“Checkout takes forever.” Where?
Complaints arrive: paying for the cart takes several seconds. Every purchase goes through the gateway, the orders service, inventory, payments, a database and an external anti-fraud service.
In the diagram you can see a request stuck somewhere, but the team sees none of it. CPU looks normal on every server and nobody knows where to start looking.
Step 2 of 9
Technique
Metrics: numbers over time
Every service records a few numbers about the requests it serves, following the RED method: Rate (requests per second), Errors (share that fails) and Duration (latency percentiles). It ships them to a time-series database and a dashboard draws them.
With the dashboard in front of you the map changes: inventory and the database answer fast; the slowness lives on the payments path.
What changed
- RED metrics on every service
- Dashboard with each service’s p99
Step 3 of 9
Technique
Alert on what users notice
Waking someone because CPU went over 80% creates alerts at all hours that almost never mean anything. It is better to alert on symptoms users actually notice.
Pick an SLI (say, the share of checkouts that finish in under 1 s), set a target or SLO (99.9% over 30 days) and alert when you burn the remaining error budget too fast. Here, 1.5% slow checkouts in the last hour is 15 times the 0.1% allowed: page now.
What changed
- SLO: 99.9% of checkouts under 1 s
- Alert on error-budget burn
Step 4 of 9
Technique
Logs: what happened, line by line
Metrics tell you where; logs tell you what happened. Every service writes timestamped events, but at thousands of requests per second lines from different requests mix together and nothing makes sense.
The fix: the gateway assigns a request id, every service forwards it on its calls and writes it on every line. Filter by that id and you read the whole story of a single purchase.
What changed
- Request id on every log line
Step 5 of 9
Solution
Traces: follow one request through the whole system
A distributed trace records every hop of a request as a span, with its start and duration, all tied together by the same trace id. Drawn as a waterfall, it shows exactly where the time goes.
In this purchase, 2.2 of its 2.6 seconds (about 85%) are spent waiting for the external anti-fraud service.
What changed
- Trace id propagated across services
- Cause: anti-fraud, ~85% of the time
Step 6 of 9
Solution
Fix it and prove it worked
With the cause in sight the fix is quick: call anti-fraud with a 300 ms timeout, cache recent scores and, if it does not answer in time, approve the payment and review it later.
The same dashboards confirm the result: p99 drops from 2.6 s to 430 ms and the SLO alert clears. Even when the cache misses and the full timeout is spent, the purchase stays under 1 s.
What changed
- 300 ms timeout and cache on anti-fraud
- Checkout p99: 2.6 s → 430 ms
Step 7 of 9
Problem
Case: errors jump at 3 a.m.
The alert fires: failed checkouts go from 0.1% to 6% at 03:12. The dashboard points at inventory, and the deploy log shows a configuration change to that service at 03:10: its timeout went from 2 s to 20 ms by mistake.
The right move is to mitigate first: revert the change and restore service. The underlying cause is investigated afterwards, calmly, not the other way round.
What changed
- Config change to inventory at 03:10
- Failed checkouts: 0.1% → 6%
- Reverted at 03:17
Step 8 of 9
Concept
Learn from it: the post-mortem
Once the incident is over, the team writes a post-mortem: the timeline, the impact, the root cause and the actions that stop it from happening again.
It is blameless: the question is how the system let the mistake through, not who made it. Here, configuration changes now roll out in stages (canary), just like code.
What changed
- Post-mortem written
- Action: canary rollout for config too
Step 9 of 9
Summary
Recap
RED metrics (rate, errors, duration) show where it hurts.
Alert on SLOs, what users notice, not on CPU.
Logs with a request id tell the story of one request.
Traces follow a request across every service.
In an incident, mitigate first; then write a blameless post-mortem.
Instrument before you need it: in the middle of an outage it is too late to add logs.
← → step · R replay · Space pause
You will learn
- Why “it is slow” is so hard to investigate (covered)
- Metrics and dashboards with the RED method (covered)
- Alerts on SLOs, not on CPU (covered)
- Logs with a request id and distributed traces (covered)
- How to run an incident and write the post-mortem (covered)