Scaling microservices
Scale for load or for robustness: the four axes, databases, caching and autoscaling.
key idea / coordinatorservice / good pathin progress / watch outproblem / failureneutral
Why scale, and along which axis
We scale for two reasons: to perform better (more load, lower latency) and to be robust (survive a failure). Different techniques help with different goals, so name the goal before picking a tool.
Newman uses a running example, FoodCo: food delivery in many countries, with growing write contention on its main database. We follow it through each axis.
| Axis | Helps load? | Helps robustness? | Main cost |
|---|---|---|---|
| Vertical | Yes, quickly | Hardly | Money; a ceiling on machine size |
| Horizontal duplication | Yes, if work spreads easily | Yes | More infrastructure; needs a way to split work |
| Data partitioning | Yes, especially writes | Little on its own | Hard key choice; cross-shard queries |
| Functional decomposition | Yes, per workload | Opens the door | Biggest code change; more services |
From Building Microservices, ch. 13, "The Four Axes of Scaling".
Vertical scaling: a bigger box
Get a faster CPU, more memory, better I/O. On virtual machines or a public cloud this takes minutes and rarely needs code changes, so it is a cheap experiment to try first.
Horizontal duplication
Run several copies of the same thing and spread the work. The consumer sees one service: the load balancer or queue is part of its logical boundary. Instances come and go, so consumers need DNS or a registry to find them (service discovery, lesson 04).
Duplication is simple because the application seldom changes. Its limits: it costs infrastructure, copies the whole monolith even when only one part is hot, and relies on your load-distribution mechanism (balancer, broker, replicas) working as designed. Systems that need a user always routed to the same replica (sticky sessions) are prone to other problems; Newman avoids building them.
From Building Microservices, ch. 13, "Horizontal Duplication".
Data partitioning and sharding
Apply a function to a key on each request; the result picks the partition (shard). Unlike duplication, each node holds only part of the data, which is why it scales writes so well.
Partitioning can happen in the database (Cassandra, Kafka topics) or in front of dedicated service instances, useful when instances keep in-memory caches. The hard part is the key.
| Benefit | Limit |
|---|---|
| Scales write-heavy workloads well | A bad key is painful to change; one client needed three days of downtime to re-partition |
| Smaller blast radius for maintenance; per-region rollouts | One failed shard fails its share of requests (a quarter, with four shards) |
| Keeps data inside a jurisdiction (geographic shards) | Queries across shards must visit each shard or use a separate read store |
From Building Microservices, ch. 13, "Data Partitioning".
Functional decomposition and combining axes
Extract a piece of functionality so it can scale on its own. This is the classic microservice split (lessons 02 and 23). Each workload can get right-sized hardware, be switched off when idle, or move to a better-suited language or database.
The goal is not to use every axis, only to know them. Newman tries the cheap, low-impact ones first (vertical, duplication) and leaves decomposition for later unless it also solves organisational scaling. That is FoodCo: splitting out deliveries and menus also lets a growing team own services.
From Building Microservices, ch. 13, "Functional Decomposition" and "Combining Models".
Scaling databases: reads, writes and CQRS
Reads and writes scale differently. Reads are easy to copy; writes are the hard part.
| Problem | Try first | Then |
|---|---|---|
| Read-heavy | Read replicas behind a balancer | Caching, or CQRS if reads need a different shape |
| Write-heavy | Bigger database machine | Shard by key; consider a database built for your write pattern |
| One hot workload | Move it to its own service and database | Scale that service on its own |
When a database cannot keep up with writes, teams often change database technology. A bigger box is the fastest relief while you evaluate options.
From Building Microservices, ch. 13, "Horizontal Duplication", "Data Partitioning" and the CQRS box in "Start Small".
Caching: where, how, and how fresh
Lesson 07 covers caching reads. Here is the scaling view: a cache cuts latency, shields a contended origin, and can keep a service answering when the origin is down (if you keep stale entries). It always trades freshness for speed.
| Invalidation | How it works | Watch out |
|---|---|---|
| TTL | Entry valid for a set time (Cache-Control, Expires) | Blunt: stale until expiry. Shorter TTL means more origin calls |
| Conditional GET | Send If-None-Match with the ETag; 304 means unchanged | Still a round trip; saves rebuilding the response |
| Notification | Origin publishes a change event (Stock Change) | Needs a broker; send heartbeats so silence is not mistaken for no change |
| Write-through | Cache and origin updated together | Practical mainly on the server side |
| Write-behind | Update the cache first, origin later | Data loss risk; unclear source of truth. Rare in microservices |
Expires: Never header, and those pages stuck in browser caches the team could not clear; the only fix was changing URLs. Know the full path from source to screen, including CDNs and browsers.From Building Microservices, ch. 13, "Caching" to "Cache Poisoning: A Cautionary Tale".
Autoscaling
If provisioning and deployment are fully automated, instances can be added and removed automatically. Two triggers: predictive (known daily pattern) and reactive (observed load or failure).
- Autoscale for failure first ("at least five instances"), while you collect load data.
- Know how fast you can scale up. If you get two minutes of notice and need ten, keep spare capacity.
- Test your rules with load tests; otherwise you learn in production.
- Scale down slowly. Extra capacity is cheaper than too little.
- Beware forgotten rules: a minimum-instances rule will respawn the instances you try to stop for maintenance.
From Building Microservices, ch. 13, "Autoscaling".
Start small, and start again
Scaling adds complexity: every new service or cache is another thing to run and secure. Treat each change as an experiment: find the bottleneck, build a small test, change one thing, measure.
The architecture that gets you started may not carry you to the next level. Gilt ran a Rails monolith for two years before load forced a redesign. That is no reason to build for huge scale on day one: you would spend months on load that may never come and learn too late whether anyone wants the product. Needing to re-architect for scale is a sign of success, not failure.
From Building Microservices, ch. 13, "Start Small" and "Starting Again".
Recap
- Scale for load or for robustness; choose the technique by the goal.
- Four axes: vertical, horizontal duplication, data partitioning, functional decomposition. Try the cheap ones first.
- Reads scale with replicas and caches; writes need bigger machines, sharding or a better-suited database.
- Pick shard keys for even, stable load; sharding alone does not give robustness.
- Cache in as few places as you can, with an invalidation strategy you can explain.
- Autoscale for failure first; test rules; scale down slowly.
- Make each scaling step an experiment; re-architecting later is a sign of success.
Quick check
Show answer
A read replica (or a cache): low risk and quick, with no change to the write model. CQRS is one of the harder options and raises cognitive load.
Show answer
Names are unevenly distributed (and vary by culture), so one shard gets far more load. Names can also change. Use a unique customer id.
Show answer
Up to two minutes: the client may refresh from a server cache that is already one minute old. Hence cache in as few places as possible.