Resilience and stability patterns
How services keep working when their neighbours fail.
key idea / coordinatorservice / good pathin progress / watch outproblem / failureneutral
What resiliency means
Resiliency is more than adding retries. Newman separates four ideas. From Building Microservices, ch. 12.
Rebound needs tested backups and an outage playbook: who leads, how fast to tell users, and how. Plan it before the stress hits.
Stability patterns
| Pattern | What it does | Watch out |
|---|---|---|
| Time-out | Give up on slow calls; also cap the whole operation | No time-out hangs threads; a 30 s one made users refresh and add load |
| Retry | Repeat temporary failures (503, 504), not a 404, after a delay | Retries add to the wait: 3 tries plus pauses can reach 3.5 s |
| Bulkhead | Separate resources, e.g. a connection pool per downstream | Costs resources, but stops one failure spreading |
| Circuit breaker | After enough failures, fail fast, then probe for recovery | Mandate it for synchronous downstream calls |
| Isolation | Own machines, or middleware that buffers calls | More infrastructure to run |
| Redundancy | Several instances, across availability zones | Needs capacity for the extra copies |
| Idempotency | Make repeats safe so messages can be replayed | Needs a unique reason or id per operation |
| Load shedding | Reject some requests so the service survives | Decide which work matters most |
Availability and the CAP trade-off
Scaling and availability meet in distributed data. Spread instances over several availability zones, because a cloud provider promises uptime for a region, not for one node. Then accept what the CAP theorem says: when the network between nodes fails, you keep consistency or availability, not both (the sections above cover resiliency in depth).
Newman advises against writing your own consistent store; use an off-the-shelf one. Aim for eventually consistent AP designs where the business allows (a rare stock error can be fixed with an apology). The choice is per capability: a loyalty-points service may show stale balances but require consistency when spending them.
From Building Microservices, ch. 12, "Spreading Your Risk" and "CAP Theorem", as it applies to scaling.
CAP theorem, chaos and game days
The CAP theorem says a distributed data store cannot give consistency, availability and partition tolerance all at once. Networks do fail, so the real choice during a partition is between the first two. From Building Microservices, ch. 12.
Do not invent multi-node consistency yourself. Use a data store or lock service built for it, such as Consul. It is not all or nothing: one system can mix AP and CP services.
| Practice | What it tests | Example |
|---|---|---|
| Chaos Monkey | Instance loss | Turns off random machines in production |
| Chaos Gorilla | Zone loss | Takes out an availability zone |
| Latency Monkey | Slow network | Simulates slow links between machines |
| Game day | People and process | Planned outage drill; the key expert is kept out of the room |
The resilience toolkit
At-least-once delivery means duplicates and replays. The book borrows ACID 2.0 (associative, commutative, idempotent, distributed) from the CALM theorem: facts that only accumulate are safe on an eventually consistent store. From Practical Event-Driven Microservices Architecture, ch. 7.
| Topic | Rule from the book |
|---|---|
| Acknowledgement | Acknowledge after processing, not on receipt. Auto-commit can lose a message in memory (message leak) on a crash or deploy. |
| Poison events | Ignore ones that fail validation. For a bug, fix and rewind the offset. A dead letter queue helps when many good events followed the bad one. |
| Retries | Cap attempts, back off exponentially, add jitter, retry only errors that can succeed. Google found retrying after the 95th percentile latency added about 2 percent load. |
Recap
- Stability patterns: time-outs, retries, bulkheads, circuit breakers, isolation, redundancy, idempotency. Degrade gracefully, choose CP or AP per service, and practice with chaos experiments and game days.
- Use the outbox for save-and-publish, acknowledge after processing, back off retries, add circuit breakers and bulkheads, and rewind offsets to repair state.
Quick check
Show answer
Slow calls tie up threads and refreshes and retries add load, so a sick Payment drags Order down. Use a shorter time-out within an overall budget, retry only temporary errors after a delay, add a circuit breaker so calls fail fast, and give Payment its own bulkhead pool.