Chapter 09 of 27 · Part 5: Data management

Managing transactions with sagas

How to keep data consistent across services without distributed transactions.

key idea / coordinatorservice / good pathin progress / watch outproblem / failureneutral

The problem: one operation, many databases

In a monolith one ACID transaction keeps everything consistent. With a database per service you need another way.

Sagas instead of 2PC

Two-phase commit needs every participant online and many modern tools do not support it. A saga splits the work into local transactions.

Undoing a saga

A saga cannot simply roll back. It runs compensating transactions for the steps that already committed, in reverse order.

Who runs the saga?

Two styles: choreography (everyone listens to events) and orchestration (one orchestrator tells each service what to do).

No isolation, so plan for it

Because each step commits at once, other sagas can see half-finished data.

Countermeasures

The most common is the semantic lock: mark records as pending so other sagas know to wait or retry.

What splitting a database costs

ACID means Atomicity, Consistency, Isolation, Durability. A service can still use an ACID transaction on its own database. What you lose is atomicity across services.

So when data must change together, you have four options, from simplest to most work.

Two-phase commit (2PC) is the classic way to get atomicity across databases. It has a vote phase and a commit phase:

From Building Microservices (ch. 6, Workflow) and Decomposing the Database (ch. 4, Transactions).

Transactions: what splitting costs you

ACID means atomicity (all or nothing), consistency, isolation (no half-done reads) and durability. Each local transaction stays ACID after a split, but atomicity across databases is the first thing lost. Many teams then reach for distributed transactions.

Two-phase commit

A coordinator asks all workers to vote; if all agree, it tells them to commit. A worker that votes yes promises to make the change later, so it must lock the record until then.

From Monolith to Microservices, ch. 4: "Transactions", "ACID Transactions", "Two-Phase Commits" and "Distributed Transactions: Just Say No".

Failing backward and forward

A saga can recover in two ways. You can mix them in one saga.

A compensating action is a semantic rollback. You cannot unsend an email, so the undo sends a second email saying the order was cancelled. Keep a record of the aborted order, because that history is useful.

From Building Microservices (ch. 6, Saga Failure Modes) and Decomposing the Database (ch. 4, Sagas).

Choosing and mixing saga styles

Choreographed sagas give up the central view. A common fix is a tracker that listens to every event. Each event carries the same correlation id, a unique id created for the saga.

From Building Microservices (ch. 6, Implementing Sagas).

Sagas

A saga breaks a long process into steps, each a local ACID transaction in one service, without holding locks across the whole thing. It does not give saga-level atomicity; it gives you enough state to reason about where you are. Full treatment in the saga sections above.

Orchestration or choreography

Pat Helland's point: with distributed transactions the more nodes you add, the more likely the whole thing stalls. Modeling the process as a saga also makes the business process explicit.

From Monolith to Microservices, ch. 4: "Sagas", "Saga Failure Modes", "Implementing Sagas" and "Sagas Versus Distributed Transactions".

Living with eventual consistency

Asynchronous events mean one service may read data before another has applied the same event. This stale period is the inconsistency window. Agree with the business where it is acceptable. In CAP terms, event-driven design chooses availability, which pushes stale-data handling onto consumers.

Five ways to deal with it:

From Practical Event-Driven Microservices Architecture (ch. 5) and Decomposing the Database (ch. 4).

What eventual consistency costs

In a monolith a change is visible everywhere at once. With events, services read stale data until the change propagates. Two properties help you reason about it: safety (nothing invalid happens) and liveness (good things happen, mainly higher availability). Eventual consistency gives no safety guarantee by itself. From Practical Event-Driven Microservices Architecture, ch. 5.

The CAP theorem is often used to justify eventual consistency. The book urges caution: CAP availability is about network partitions, not overall uptime. Google reported only 7.6 percent of Spanner incidents were network related. Eventual consistency also pushes the complexity of stale data onto every client, so confirm the business can live with it. One team lost trust in a product-visibility feature after delays and went back to manual work.

The practical goal is a small inconsistency window. Studies the book cites for Cassandra and MongoDB found windows of a millisecond or a few milliseconds for most requests. If eventual consistency is fast enough, nobody notices. For services that means horizontal scaling and a lag metric.

Five ways to handle stale data

The running example: a subscription service must know the product total stock, but the inventory event only says one XS unit was bought. From Practical Event-Driven Microservices Architecture, ch. 5.

Local state can also live in memory: either a time window of recent events (good for bulk jobs) or the whole stream loaded at startup (only for small, stable data sets such as a few hundred categories).

Recap

• A saga is a series of local transactions linked by messages.
• Failures are undone by compensating transactions in reverse order.
• Prefer orchestration for anything beyond a simple saga.
• Sagas lack isolation. Use countermeasures such as semantic locks.
• Use the transactional outbox (chapter 06) so each step's update and message happen together.
• Avoid two-phase commit: it holds locks across services. Keep data together if it truly needs ACID.
• A saga can fail backward (compensate) or forward (retry). Reorder steps so likely failures come first.
• Mix saga styles if needed, and track a choreographed saga with a correlation id.
• Event-driven systems are eventually consistent. Use richer events, versions, local state or end-to-end checks.
• Splitting tables loses joins, foreign keys and transactions; handle them in code and with sagas, not two-phase commit.
• Eventual consistency is manageable: richer events, boundaries, versioned retries, local state, end-to-end checks, and a small inconsistency window kept small by autoscaling on lag.

Quick check