Microservices

Chapter 17 of 27 · Part 8: Cross-cutting concerns

Observability

See what your services are doing: logs, metrics, traces, health checks and alerts on what users feel.

key idea / coordinatorservice / good pathin progress / watch outproblem / failureneutral

Observability

Make the running system visible to developers and operations.

Six observability patternsObservabilitysee inside a running systemHealthGET /healthLogssearchableTracingone requestMetricsnumbersExceptionsdedupe + alertAudit logwho did what
Health check APIInvokermonitor, K8s, registryGET /healthhealth handlerChecksDB, broker200 or 503send traffic or notA service can be running yet broken. The infrastructure uses this to route traffic or restart it.
Log aggregation (ELK)Serviceslog to stdoutLog pipelineLogstash, FluentdLog serverElasticsearchYouKibana: search, alert
Distributed tracingUserAPI gatewayOrder svcZipkinGET /orders/1trace XYZ, span AX-B3-TraceId: XYZchild span Breport spansview the timelineA trace is one request. A span is one timed step. The trace id also links all the log lines.
Application metricsOrderServiceplaced_orders++Metrics endpoint/actuator/prom..Prometheuspulls metricsGrafanagraphs, alertsA sample has a name, value, timestamp and dimensions. Push (CloudWatch) or pull (Prometheus).
Three ways to write an audit logIn business logicSimplebut tangled, easy to missAOPReliablebut little business contextEvent sourcingAutomaticbut harder to query

Observability versus monitoring

Monitoring and observabilityMonitoringAn activity you doAlerts on problems you predictedWeak against new failuresObservabilityA property of the systemAsk questions you never plannedBuilt from external outputsThe more observable a system, the faster you find what went wrong.

Newman distrusts the "three pillars" (metrics, logs, traces) as too reductive: they overlap, and a tool per pillar can be a sales pitch. Think of every output as an event. From a stream of events you can project a trace, a searchable index or aggregated numbers. Aim for a system you can understand, not a pile of tools. From Building Microservices, ch. 10.

Building blockAnswers
Log aggregationWhat happened on every instance?
Metrics aggregationWhat are the trends? When do we need capacity?
Distributed tracingWhere did one request spend its time?
SLOs and alertingAre we doing OK, and who gets woken?
Semantic monitoringIs the business behaving as expected?
Testing in productionCan we catch problems before users do?

Aggregating logs, metrics and traces

How log aggregation worksServicelogs to a fileLocal agentships the logsCentral storeindexed, queryableOperatorssearch and alertServices need no special API. Know how logs can get lost in shipping.

Newman treats a log aggregation tool as a prerequisite for microservices. Logs can hold sensitive data, so limit access and avoid logging what you do not need.

Metrics: group and resolutionGroup by (needs metadata)Whole systemOne serviceOne instanceKeep at different resolutionLast 30 min: every 10 sLast month: hourly averageLong history shows trends and drives capacity planning and autoscaling.
Distributed tracing pipelineInstrumentOpenTelemetry APILocal agentbuffers, adds tagsCollectorbuilds the traceViewerJaeger or similarSampling: keep every error, but maybe only 1 in 100 normal successes.

Spans from each service share an identifier and are joined into one trace. Pick tools that support the open OpenTelemetry API: many libraries already emit spans, and you can switch vendors later.

SLOs, alerts and semantic monitoring

"Up" or "down" means little for a distributed system: one sick bee does not make a sick hive. Instead define what acceptable behavior is. From Building Microservices, ch. 10.

From promise to measurementSLApromise to usersSLOteam targetSLIwhat you measureError budgetallowed misses99.9% availability per quarter leaves about 2 h 11 min of downtime to spend.

With budget left, try risky changes. With the budget gone, put reliability first. Budgets give teams room to experiment.

What makes an alert goodAlerts should beRelevantUniqueTimelyPrioritizedand alsoUnderstandableDiagnosticAdvisory (what to do)Focusing attentionCriteria from Steven Shorrock's alarm design work, as quoted by Newman.

Too many alerts overwhelm people, as seen at Three Mile Island. Ask: should this wake someone at 3 a.m.? A routine disk failure can wait for working hours.

Semantic monitoringState the modelsign-ups, sales per hourMeasure itreal users + syntheticAlert on the modelnot on every errorExample: new customers can register, and we sell $20,000 an hour at peak.

Logging and restarts in practice

Bootstrapping Microservices keeps it simple: log to standard output, and let the platform collect it.

Where to read logsDocker Composeone merged streamkubectl logsone pod at a timeAggregationFluentd + ElasticDashboardsKibana, GrafanaKubernetes has no built-in log aggregation. Use third-party tools, not your own.
What to do when a service hits an errorAbort and restart (default)Let the process exitKubernetes restarts the podRisk: in-flight work is lostResume operation (rare)Keep running after the errorUse when aborting loses dataExample: a video upload serviceKubernetes also lets you define readiness and liveness probes for finer health checks.

Code defensively: expect bad input, hidden bugs and flaky dependencies such as the message broker. Test the error paths too, not only the happy path.

Recap

Quick check

CPU and memory dashboards look fine, but users say checkout is slow. What should you have had in place?
Show answer

Signals about what users feel: an SLO on checkout latency, with distributed tracing across the services in the request so you can see which hop is slow. Host metrics alone rarely show a problem in the call chain.