Testing

  • Lesson 35 of 40
  • 12 min
  • Beginner
  • 11 steps

Testing by hand is slow and only covers what someone remembers to check. A good test suite tells you within minutes, on every change, whether you broke something.

Feedback in
20 s → 12 min
Tests cover
whole system
Bugs reaching users
rare
shoppersTestsTestsTestsTestsTestsTestsTestsTestsTestsCheckoutyour codeCheckoutyour codeCheckoutyour codeCheckoutyour codeCheckoutyour codeCheckoutyour codeCheckoutyour codeCheckoutyour codeCheckoutExternalpaymentsExternalpaymentsPaymentstubPaymentsandboxPaymentsandboxPaymentsandboxPaymentsandboxExternalpaymentsPayment sandboxInventoryInventoryStock stubStock stubInventoryInventoryInventoryInventoryInventoryorders db$12.73 · should be $12.72overcharged3,000 tests · 20 sA unit testcart3 × $4.99couponOFF15 · 15%total(cart, "OFF15")$12.72 · passesno db, no network · ~1 msstub: “approved”stub: “in stock”in memory · no disk250 tests · 5 mincreated for each runcontractverifies the contractThe contractGET /items/A12/stock200 { "id": "A12", "units": 4 }rename units → Inventory’s build failseach side checks it separately15 journeys · 12 minThree layersend-to-end15 · 12 minwhole journeysintegration250 · 5 minreal neighboursunit3,000 · 20 sone piece at a timemany and fast at the bottom, few at the topfixed 2 s wait for the emailfails 1 in 20 · “retry”85%p99 shoots up at 4.5×65%injected delay: 5 s
  • request
  • response
  • slow request
  • failed request

Step 1 of 11

Concept

Does this change break anything?

The checkout service totals each purchase and applies coupons. It stores orders in its database, charges through an external payment provider and asks the Inventory service, owned by another team, whether items are in stock.

Someone has just changed how discounts are rounded. To check it, the team makes a couple of purchases by hand on a test server. Everything looks right, and it ships.

Step 2 of 11

Problem

Manual checks leave gaps

Nobody tried a cart of 3 items at $4.99 with a 15% coupon. The discount is 14.97 × 0.15 = 2.2455, which should become $2.25; the new code truncates it to $2.24 and charges $12.73 instead of $12.72. One cent too much on thousands of orders. Every figure in this lesson is an example value.

Because manual checks are slow, they happen rarely and only cover the cases someone thought of. Customers found this bug.

What changed

  • The rounding bug reaches production
  • Customers find it, not the team

Step 3 of 11

Solution

Unit tests: one piece in isolation

A unit test calls a single function with known inputs and compares the result: here, total(cart, "OFF15") must return $12.72. It touches no database and no network, so it takes about a millisecond.

They are cheap enough to keep thousands and run them on every change: 3,000 tests in 20 s. The rounding case is now written down and fails the moment someone breaks it again.

What changed

  • Tests call checkout directly
  • No real database, payments or Inventory

Step 4 of 11

Technique

Test doubles for everything else

Checkout cannot run without its dependencies, so tests replace them with doubles: a fake in-memory database, a payment stub that always answers “approved”, and a mock that records whether the card was charged exactly once.

The catch: a double only knows what you told it. If the real provider renames a field, every unit test stays green.

What changed

  • Fake db, payment stub and stock stub

Step 5 of 11

Solution

Integration tests: real neighbours

An integration test runs checkout against a real database, created from scratch in a container for each run, and checks what actually ends up in the tables. SQL mistakes, broken migrations and mismatched types show up here instead of in production.

Payments stay simulated (you cannot charge real cards), though a separate, smaller group of tests can call the provider’s sandbox. That is 250 tests in 5 min.

What changed

  • Fake db → real db in a container
  • Tests take seconds, not milliseconds

Step 6 of 11

Technique

Contract tests between teams

Inventory belongs to another team, who deploy whenever they like. If they rename the field units to quantity, checkout’s stub never notices and the tests keep passing.

A contract test writes down what checkout expects (this request, a response with these fields), and Inventory’s pipeline checks every change against that contract. An incompatible change fails in their build, before it ships.

What changed

  • What checkout expects, saved as a contract
  • Inventory verifies it on every change

Step 7 of 11

Solution

End-to-end: the whole journey

An end-to-end test uses the real system the way a user would: it opens the shop in an automated browser, adds three items, applies the coupon, pays with a sandbox card and waits for the confirmation.

It is the only test that proves everything works together, and also the slowest and most fragile. About 15 tests for the journeys that bring in revenue are enough: 12 min per run.

What changed

  • A test browser walks through the whole system
  • Minutes per run

Step 8 of 11

Concept

The test pyramid

Together they form the test pyramid: thousands of unit tests at the base, hundreds of integration and contract tests in the middle and a few end-to-end tests at the top.

Lower layers are fast, precise and cheap to run on every change; upper ones are slow but closer to what a user does. Push every check down to the lowest layer that can catch it.

Step 9 of 11

Problem

Case: the flaky test

One end-to-end test fails once every 20 runs. It waits a fixed 2 seconds for the confirmation email, and sometimes the email takes 3. The team gets used to hitting “retry”, and when a real bug turns it red, they retry that too.

A flaky test is worse than no test at all: it teaches the team to ignore failures.

What changed

  • Fails 1 run in 20 for no reason
  • Real failures get retried and ignored

Step 10 of 11

Technique

Load tests and failure tests

Working is not enough. A load test sends realistic traffic to a copy of the system (staging) and finds the point where latency starts to climb, before your users do. Here, at 6 times normal traffic, p99 shoots up from about 4.5 times onwards.

Failure tests break things on purpose (stop a replica, make the payment provider take 5 s) and check that timeouts, retries and failover behave as designed.

What changed

  • Load test: 6× normal traffic on staging
  • Failure test: payments slowed to 5 s

Step 11 of 11

Summary

Recap

Unit tests check one piece with doubles for the rest: fast enough to run by the thousand on every change.

Integration tests use real neighbours, such as a database in a container, to catch what doubles cannot see.

Contract tests catch incompatible API changes between teams, in the provider’s build.

A few end-to-end tests cover the key journeys: keep the pyramid the right way up.

Fix flaky tests, and test load and failure as well, not just whether it works.

You will learn

  • Unit tests and test doubles
  • Integration and contract tests
  • End-to-end tests and the test pyramid
  • Flaky tests, load tests and failure tests