Chapter 08 of 27 · Part 5: Data management

Database per service and the shared database

Why each service should own its data, what a shared database costs, and the coping patterns while you get there.

key idea / coordinatorservice / good pathin progress / watch outproblem / failureneutral

Split the data along the same lines

A service boundary is only real if it owns its data. The Decomposing the Database chapter of Monolith to Microservices shows how to prepare the data before any service exists.

Splitting the data along the same linesFirst split data access code, then the schemas, then the services, each with its own data.1 Repositoriesone app, one schemaFinanceCatalogOrdersOne shared schema2 Schemasone app, three schemasFinanceDBCatalogDBOrdersDB3 Servicesthree services, own dataFinanceDBCatalogDBOrdersDBStage 2 is a hedge: a modular monolith with separate schemas can stay a monolith.Logical split can share one database engine. A physical split adds robustness but costs more.
Based on the database decomposition chapter of Monolith to Microservices.

Lesson 13 covers what happens next, when a service must take its tables and the monolith still needs them.

The shared database and why it hurts

Sharing one database between services is the most common form of implementation coupling. The core problem: you no longer decide what is shared and what is hidden, so nobody knows which parts of the schema are safe to change.

Three services writing to one order schemaWeb shopchanges orderBillingchanges orderFulfilmentchanges orderShared order schemaone state machine, three rule setsWho owns the rules? If behaviour differs between services, the data breaks.

Cohesion suffers too. When three services can change an order, the rules for its state machine are spread out, and every rule change must be repeated in each service. (Here "database" means a logically isolated schema; one engine can host many.)

ProblemWhat it means
Hidden consumersYou know others read the schema but not which tables or columns they use.
Unclear controlBusiness logic that changes the data lives in several places.
Schema as public contractEven unmaintained clients freeze your table design.
The bank story. A credit-derivative platform needed faster writes. Over 20 outside applications used one shared login, so nobody knew who read or wrote what. The team resolved the writers, then built a schema of views that looked like the old one. Lesson: give each actor its own credentials, so access is visible and revocable.

When sharing is acceptable

CaseWhy it is safe enough
Read-only static reference dataCountry codes, postal lookups: very stable, changed as an admin task.
A database exposed on purpose as an endpointDesigned and managed for many consumers (see the database-as-a-service section).

Everything else should move toward each service owning its data. Where that cannot happen yet, the coping patterns below stop things getting worse. This builds on split the data along the same lines above and living with a shared database below.

From Monolith to Microservices, ch. 4: "Pattern: The Shared Database" and "Schemas and Databases".

Living with a shared database

Sharing a database means you cannot tell what is shared or hidden, and no one controls the business rules. The Decomposing the Database chapter lists coping patterns to use while the real split is not yet possible.

Four ways to live with a shared databaseA view, a wrapper service, a read-only copy database and an API on the monolith.1 ViewNew serviceDatabase viewMonolith schema2 WrapperOther appsWrapper serviceTangled schema3 Read-only DBReporting clientRead-only copyInternal DB4 Monolith APINew serviceMonolith APIMonolith DBAll four are stepping stones. They limit or hide the shared schema; they do not remove it.Arrows show calls or reads. In option 3 the copy is filled from the internal DB.
PatternIdeaLimit
Database viewGive clients a read-only projection of the schemaRead-only, usually the same database engine, and someone must keep it in step with the schema
Wrapping serviceHide a schema that is too tangled behind a thin service and stop it growingCallers must change to API calls. It is a bandage
Database as a service (reporting database)Fill a separate read-only database from the internal one with a mapping engine, ideally change data captureData lags. More work, but it can use a different database technology
Aggregate exposing monolithThe monolith offers an API for the data a new service needs. The same idea is called monolith as data access layerWorks while the monolith still owns the state changes

The book tells of a bank trading system where more than 20 unknown applications shared one login. The schema had become a public contract that could not change. Give each client its own credentials from the start. A secret store can make that practical.

Coping pattern: view and wrapping service

Two ways to regain control of what is shared without splitting the data yet.

Database view

A view is a limited projection of the underlying schema. It hides columns and tables, and lets the owner change tables as long as the view keeps working.

A view projects only part of a tableCustomer tablename, address, loyalty IDDatabase viewcustomer ID + loyalty IDLoyalty servicereads the view onlyOther tables stay invisible. The consumer sees a projection, not your schema.
AspectDetail
LimitsUsually read-only; the view and source often must sit on the same engine; support varies by database.
Materialized viewPrecomputed, so reads are cheaper but may be stale.
OwnershipTreat a published view like a service interface, kept current by the team owning the source schema.
Use whenFull decomposition is impractical now. Prefer real schema decomposition if the goal is a service interface.

Database wrapping service

When the schema is too tangled to touch, hide it behind a thin service. In the book, a bank kept its entitlements logic, tangled in stored procedures, behind a new service so other teams stopped adding data and behaviour to it.

Wrapping a hard-to-split database in a serviceBeforeAfterApp 1App 2App 3Entitlementsschema(stored procs)App 1App 2App 3Wrapper servicethin API in frontEntitlements schemaNew data goes in app-local storesno more growthEveryone writes straight inDatabase dependencies become service dependencies. Later you can split the schema.
ViewWrapping service
Shape of dataMust map onto existing tablesAny projection you can code
WritesTypically noYes, through API calls
Consumer changePoint at another schemaSwitch from DB access to API calls
Best withRead-only consumersSame team owns schema and wrapper
Honest framing. Both are bandages. They are worth it as stepping stones, and when they let you say clearly what is yours and what is not.

From Monolith to Microservices, ch. 4: "Pattern: Database View" and "Pattern: Database Wrapping Service".

Database-as-a-service interface (reporting database)

Sometimes clients just need a SQL endpoint: large reads, or tools such as BI dashboards. Do not hand them your working database. Build a separate, read-only one and fill it from the internal one.

Exposing a read-only database through a mapping engineInternal DBprivate to serviceMapping enginecopies changesExposed DBread-only, own techBI / SQL toolsad hoc queriesThe service also offers a read/write API and events. The external DB is one more endpoint.The engine always lags behind writes, so say when the data was last updated.

Mapping engine options: change data capture is the most robust and freshest (for example Debezium); listening to service events works if you already emit them; a batch copy lags longer, is hard to scope, and can fail silently. The author now prefers CDC.

Compared to a viewDetail
TechnologyThe exposed DB can be a different engine (Cassandra inside, SQL outside).
CostMore work, but more freedom to evolve the internal schema.
PathStart with a view; move to a dedicated reporting database later if needed.
FitsRead-only clients, large joins over one service data. Can feed a larger warehouse.

The name differs from Fowler's reporting database pattern because reporting is only one use; the same idea serves any ad hoc query. See lesson 11 for CQRS views, and feeding the reporting database below. Do not underestimate the work of keeping the projection current.

From Monolith to Microservices, ch. 4: "Pattern: Database-as-a-Service Interface".

Read-only databases for others

Sometimes a consumer just needs a database to query: ad hoc SQL, large joins, or a BI tool that expects a SQL endpoint. You can offer this without exposing your internal tables. Three techniques, from lightest to heaviest:

TechniqueWhat it isLimits
Database viewA read-only projection of a schema. A materialized view is precomputedSame database engine; materialized views can be stale; tied to the schema owner
Wrapping serviceA service in front of the schema that can build richer projections and take writesConsumers must switch from SQL to API calls
Reporting databaseA separate database, filled from the internal one, exposed read-onlyYou must keep it up to date; it lags the source
The reporting database patternOrders serviceRead/write APIInternal database (hidden)Mapping engineCDC, batch or eventsReporting DBread-only, own schemaBI and ad hoc SQLThe owning team keeps this endpoint compatible, like any API.

Two points matter. Expose only the minimum, so the reporting schema can be shaped for its readers, even on a different database technology. And treat it as part of the service interface: when the internals change, the mapping changes, not the consumers.

From Building Microservices (ch. 3, Reporting Database) and Decomposing the Database (ch. 4).

Feeding the reporting database

The mapping engine copies changes from the internal database to the external one. It always lags a little, so readers see slightly old data.

Ways to build the mapping engineChange data captureMost robustFreshest viewe.g. DebeziumService eventsService fires eventsA listener updates the DBDoubles as public eventsBatch copyLongest lagHard to pick changed rowsJobs can silently failTip: expose when the external database was last updated, so readers can judge freshness.
Move away from batch. Newman says he has been bitten too often by batch jobs that did not run or ran too long, and would now reach for change data capture. Do not underestimate the work of keeping the projection correct.

From Decomposing the Database (ch. 4, Database-as-a-Service Interface) and Building Microservices (ch. 3).

Data lakes and warehouses

To query across many services, data can be pushed into one shared store. This is "communication through common data": one side writes, others read later, usually by polling. The two best-known forms sit at opposite ends of a coupling scale.

Data lake versus data warehouseOne-way flow: services write, others readData lakeraw data in any formatData warehousestructured store with a schemacouplingLooser: readers must know how to parse itTighter: writers must know the structure
Data lakeData warehouse
Who knows the formatThe readersThe writers
If the structure changesReaders adaptWriters break unless updated
Good forLarge volumes, not low latencyLarge volumes and structured reports
Avoid the shared read/write database. The pattern works because data flows one way. If several services both read and write the same records, that is common coupling. For near-real-time data, a stream such as Kafka suits better than polling files.

From Building Microservices (ch. 4, Communication Through Common Data).

Recap

Quick check

Two services read and write the same customers table because it was quicker. What goes wrong, and what is a safe first step?
Show answer

Nothing is hidden: any schema change can break the other service, and the rules for a customer live in two places. Give the table one owner, then let the other service read through a view, a wrapping service or the owner's API until it has its own data.