Microservices

Chapter 15 of 27 · Part 7: Testing

Integration, contract and end-to-end tests

Tests wider than a unit: integration, consumer-driven contract, service component and end-to-end, plus testing in production.

key idea / coordinatorservice / good pathin progress / watch outproblem / failureneutral

Integration tests

Test the adapters, not whole services: one adapter at a time, and contracts between services.

InteractionFTGO exampleContract holds
REST request/responseAPI gateway → Order svcHTTP request + response
Publish/subscribeOrder svc → Order HistoryOne event
Async request/responseOrder svc → Kitchen svcCommand + reply message
Persistence integration testStart DBMySQL in DockerTxn 1: savenew OrderTxn 2: loadfindById()Assertfields matchUnit tests only use objects in memory. This proves the ORM mapping works against a real database.

Integration tests with a real database

Bootstrapping Microservices tests a whole microservice through its real HTTP interface, with no mocks. The database is real too, and it is loaded with known test data called a fixture.

Integration test of one microserviceCompose upstarts MongoDBLoad fixtureknown recordsHTTP requestAxios to the APIAssertcheck the replyThe fixture helper empties the test database first: never aim it at production.

Integration tests are slower than unit tests because a real server and database start, but they find problems that hide between modules, and one test covers a lot of code. Docker Compose gives every developer the same database.

Contract tests for REST

One contract becomes a provider test and a consumer stub.

REST contract testsContractGET /orders/{id}Spring Cloudgenerates + stubsProvider testcontroller + mocksConsumer stubWireMock serverIf both sides pass against the same contract, they agree on the API.
Consumer side: the API gateway's proxyTestOrderSvcProxyWireMockfindOrderById()GET /orders/1223232200 + contract JSONassert fields

Contract tests for messaging

The same idea works for events and for commands with replies.

Publish/subscribe contract: consumer sideTestMessaging stubEvent handlersMock DAOtrigger event labelOrderCreatedaddOrder()verify eventuallyProvider side: a generated test checks Order Service publishes an event matching the contract.
Async request/response contract: consumer sideTestKitchenProxyMessaging stubsend CreateTicketcommand messagematch contract inputCreateTicketReplyassert replyProvider side: Kitchen's command handler gets the input message and must send the output message.

Consumer-driven contracts and Pact

A schema catches structural breakage, such as a missing field. It cannot catch semantic breakage, where behavior changes but the shape does not. End-to-end tests catch both but at high cost. Consumer-driven contracts (CDCs) catch semantic breakage with a small scope. From Building Microservices, ch. 9.

Contracts from two consumers run against one producerHelpdesk testsWeb shop testsCustomer buildruns both sets, aloneLoyalty stubEach consumer writes its expectations. The producer runs them on every build.A failure shows exactly which consumer would break.
How Pact worksConsumer testexpectationsPact fileJSON contractPact Brokerstores versionsProducer buildverifies the fileThe mock server that writes the Pact file also works as a local stub for the consumer.
CheckCatchesScope
SchemaStructural breaks (field removed)Tiny
Consumer-driven contractSemantic breaks (behavior changed)One service, like a service test
End-to-end testBoth, plus wiring problemsMany services, slow

The Pact Broker also records when a contract was last verified and which services depend on which. Spring Cloud Contract is an alternative, but it only suits an all-JVM stack. Good practice: people from both teams write the contracts together. A CDC is a record of a conversation, and a broken contract is the cue for the next one.

When trust is low: CDCs need good communication. For a third party you cannot talk to, keep a few larger tests around that component. For a public API with thousands of consumers, play the consumer yourself or work with a subset of them.

Contract tests and test categories

End-to-end tests cover the whole system but are slow, hard to keep stable and need a full copy of production. Contracts check the seams cheaply. From Practical Event-Driven Microservices Architecture, ch. 10.

Consumer-driven contractThe consumer states what it needs. The producer must keep passing it.Consumerwrites expectationsContractfields and typesProducerverifies in its buildA breaking change fails the producer build before release, with the consumer team named in thefailure.
Marick quadrantsTests sorted by business or technology focus, and by checking or exploring.Q2 Business, verifyfunctional and acceptanceQ1 Technology, verifyunit, component, integrationQ3 Business, exploremanual exploratory testingQ4 Technology, exploreload, security, reliability
End-to-end quality from many small checksEach service validates itself and its seams, and the sum covers the flow.Order svcunit, componentContractsschema and consumersInventory svcunit, componentA failing flow usually maps to one component, which can check that condition by itself.

Component tests

Acceptance tests for one whole service, treated as a black box, with its neighbors stubbed.

Acceptance tests written in GherkinGherkin stepGiven a valid consumer and cardWhen I place an orderThen the order is APPROVEDAnd OrderAuthorized is publishedTest phaseSetupExecuteVerifyVerifyCucumber runs these near-English scenarios as real tests.
Two ways to run a component testIn-processSame JVM as the testIn-memory DB and messagingFast and simpleDoes not test the real buildOut-of-processService runs in DockerReal MySQL and KafkaTests what you shipSlower, more complex
Component test for Order ServiceStep defsOrder svcKafkaAccounting stubreply success (Given)POST /orders (When)AuthorizeCommandsuccess replyGET /orders/{id}check OrderAuthorizedOrder Service is real. Its saga partners are stubs that reply over Kafka.
Running component tests with Docker ComposeBuildservice JARStartsvc, MySQL, KafkaRunCucumber scenariosStopcontainers down

Which tests does an event-driven service need?

Pick tests by what they prove and how much they cost. Each layer of the service gets unit tests, and the service as a whole gets a few broader ones. From Practical Event-Driven Microservices Architecture, ch. 10.

Test pyramid for one serviceMany fast unit tests, fewer component tests, a few integration and contract tests.End-to-end (avoid)Contract and consumer-driven contractsComponent and half-breed integrationUnit tests in every layerThe higher the level, the slower, costlier and flakier the test.
TestScopeBroker and database
UnitOne class or function in a layer.None
ComponentThe whole service, in memory.Fakes or in-memory stand-ins
Half-breed integrationThe service with real infrastructure.Real broker and database, other services faked
ContractThe event schema between two services.None needed
Component test with fakesFeed an event in, check the state change and the events published.Fake eventOrderCreatedServicereal logic, fake I/OAssertstate and outputFast and deterministic. Good for idempotency and error paths (lesson 06).

End-to-end tests

Test the whole application, but only a few key journeys.

One end-to-end user journeyPlace orderAPPROVED, $16.33Revise order+2 samosas, $20.97Cancel orderCANCELLEDOne journey test instead of three separate tests means less setup and fewer slow tests.
End-to-end testsWhy fewMany moving partsSlow: minutes even for FTGOBrittle and hard to debugHowWrite user-journey testsGherkin + CucumberPush testing down the pyramid

Browser tests with Playwright

Bootstrapping Microservices boots the application with Docker Compose and drives its UI with Playwright. The test sees the app the way a customer does. From Bootstrapping Microservices, ch. 9.

Playwright against an app running under Docker ComposePlaywrightheadlessGatewayserves the UIMetadatavideo recordsMongoDBtest databaseFixture APIload test datadocker compose up--buildA cut-down app is enough. A small REST service loads fixtures for the tests.

A test loads a fixture, opens a page, then asserts what it shows, for example that the video list has two entries. Playwright runs headless, can try several browsers, and gives good reports, so it fits a CI pipeline via npm test. It can also mock the backend to test a frontend alone, though that is then not truly end-to-end.

Fast loops: live reload (nodemon plus Docker volumes) restarts services as you edit, so you do not rebuild the whole application per change. Also test production mode locally now and then, so Dockerfile problems do not pile up.

Why end-to-end tests hurt

One service changes, and you want to know it did not break the helpdesk or web shop. The usual answer is a shared end-to-end stage that every pipeline feeds into. From Building Microservices.

Fan-in: many pipelines share one end-to-end stageCustomer buildLoyalty buildHelpdesk buildWeb shop buildShared e2e stageall services deployedDeployWhich versions of the others do you test with? One slow stage now blocks every team.
PitfallWhy it hurts
Flaky and brittleMore moving parts mean failures that say nothing about the feature. People rerun and stop trusting the suite.
SlowSuites of a day, or six weeks for a regression run. Failures surface after you have moved on.
Great pile-upA broken stage blocks check-ins, changes accumulate and each release gets riskier. The fix is a faster suite.
Metaversion"These versions work together" slides into one version number and deploying everything at once.
No independent testingShared test environments and shared suites force teams to coordinate. High performers test on demand.
Who owns the end-to-end tests?Free-for-allEveryone adds testsNobody owns failuresDedicated test teamFar from the codeService owners waitShared ownershipTeams co-own healthRare, needs disciplineOne variant: assign groups of tests to a team, but another team can still break them.

Newman is convinced that at a certain organizational scale you need to move away from cross-team end-to-end tests.

Shrinking the end-to-end suite

Think of end-to-end tests as training wheels. While you learn contracts and production monitoring they trade speed for safety. As the other safeguards improve, you need fewer of them. From Building Microservices.

Training wheels for releasesE2E netslow but safeAdd contractsschemas + CDCsAdd prod checkscanary, monitoringShrink E2Eonly what is leftDo not drop them before you know what you would lose.
QuestionBest tool
Will my change break a consumer?Consumer-driven contract
Does the service work with neighbors stubbed?Service test
Does one key user journey work?A few end-to-end tests
Is it working for real users right now?Testing in production (next)

Testing in production

Pre-production tests cannot drive the chance of failure to zero. A distributed system has too many surprises, so testing where the software really runs gives better feedback. Much of it is safe, and you may already do some. From Building Microservices, ch. 9 and 10.

From deploy to release with testsDeploynot released yetSmoke testdoes it work?Canarysome usersReleaseeveryoneSynthetic transactions and chaos tests keep running after release.
TechniqueWhat happensReal users affected?
Ping / health checkIs the instance alive?No
Smoke testAutomated check of a new deployment, before releaseNo
Synthetic transactionFake user with known input and expected output, run regularlyNo (fake data)
Canary releaseNew version for a small slice, widen if healthyA few
A/B testTwo versions side by side, see which performs betterSplit
Parallel runRequest goes to old and new; user sees one, you compareNo
Chaos engineeringInject faults, e.g. Chaos Monkey stops VMsSystem under stress
Two ways to reduce production painLower failure rate (MTBF)More pre-production testsNever reaches zeroDiminishing returnsFaster recovery (MTTR)Very fast rollbackGood monitoringCanary limits the damageTeams that pour effort into tests often neglect monitoring and recovery.
Make it safe: separate deployment from release so you can test deployed but unreleased code. Use fake users and data. Watch side effects: one company ran tests against live ordering and a pile of washing machines arrived at head office. If the product idea is unproven, skipping pre-production tests can even be sensible.

Testing event-driven services in production

Pre-production environments never match live traffic. Safe production techniques complement them. From Practical Event-Driven Microservices Architecture, ch. 10.

Shadowing with a separate consumer groupA new version reads the same events but its output is not used.Topicorder eventsLive v1group: liveShadow v2group: shadowCompareno side effectsBoth groups get every event. Disable external effects such as emails in the shadow.
Canary release and feature flagSend a small share of traffic to the new version, and switch behavior by flag.5% canarywatch error rate25%watch latency and lag100%retire old versionA feature flag turns behavior off without a redeploy, so rollback takes seconds.
TechniqueWhat it provesRisk to manage
ShadowingNew code gives sane results on real events.Duplicate side effects, extra load
CanaryReal users are fine on the new version.A small share sees the bug first
Feature flagsCode can ship dark and be enabled slowly.Old flags pile up; remove them
Automated production testsKey flows work now, using test accounts.Test data must stay out of reports
Deployment note: the book has no deployment chapter. For pipelines and rollout mechanics see lessons 19 and 21. For rewinding offsets, scaling on lag and the outbox, see lesson 06.

Performance and robustness tests

Newman prefers cross-functional requirements to the name non-functional: they emerge from work across the system. Many can only be fully met in production, but tests show whether you are heading the right way. From Building Microservices.

Cross-functional requirementsSpeedLatencypage response timeCapacityUsershow many at onceAccessAccessibilityusable by allSafetySecuritycustomer dataFollow the pyramid: a load test finds the bottleneck, a small test keeps it fixed.

Splitting a system adds network calls. One database call may become several service calls, and one slow link in a synchronous chain slows everything. So test performance early, not just before launch.

A useful performance testProd-like setupdata and machinesRamp up usersgraduallyWatch latencyat each load levelComparevs target or lastRun a subset daily and a bigger set weekly. Then actually read the numbers.

Without a target from your SLOs, fail the build when results change too much from the previous one. Robustness tests re-create failures, for example an artificial timeout between a service and its stub, to check that retries or circuit breakers work. They are worth it for shared code such as a service mesh default.

Recap

Quick check

One REST contract (GET /orders/1223232 → 200 + JSON) is used in two ways. What are they?
Show answer

It generates a provider-side test that calls OrderController, and it configures a WireMock stub that the API gateway's proxy test calls instead of the real Order Service.

A schema says the Customer API still returns a points field, but the value is now in a different unit. Which test type can catch this before production?
Show answer

A consumer-driven contract. Schemas only check structure. A contract holds a sample request and the expected response, and the producer runs it on every build.

What does a consumer-driven contract catch that unit tests miss?
Show answer

A producer change that breaks what a consumer relies on, such as a renamed field. The producer build fails before the release.

How can you try a new service version on real events without affecting users?
Show answer

Run it as a shadow in its own consumer group, with side effects disabled, and compare its output with the live version.