Observability and failure

Key points

• Three signals: metrics say whether it is healthy, logs say what happened, traces say where the time went.
• Track RED for every service - Rate, Errors, Duration. Alert on what users feel, like error rate and p99 latency, not on CPU.
• Percentiles, never averages. An average of 80 ms hides the p99 of 4 seconds, and the p99 is the customer about to leave.
• Every remote call needs a timeout. Without one, a slow dependency parks all your threads and the outage becomes yours.
• Retry only idempotent calls, with exponential backoff plus jitter and a hard cap. Synchronised retries turn a blip into an outage.
• A circuit breaker (Resilience4j) fails fast while a dependency is down; a bulkhead keeps that dependency from consuming every thread.

Example

# Spring Boot 3 + Actuator + micrometer-registry-prometheus
management.endpoints.web.exposure.include=health,info,metrics,prometheus
management.endpoint.health.probes.enabled=true
management.endpoint.health.show-details=when-authorized
management.metrics.tags.application=${spring.application.name}
# Sample 10% of traces: enough signal, affordable volume
management.tracing.sampling.probability=0.1

# Resilience4j: fail fast instead of piling up on a sick dependency
resilience4j.circuitbreaker.instances.payments.sliding-window-size=50
resilience4j.circuitbreaker.instances.payments.failure-rate-threshold=50
resilience4j.circuitbreaker.instances.payments.wait-duration-in-open-state=10s
resilience4j.timelimiter.instances.payments.timeout-duration=2s
resilience4j.retry.instances.payments.max-attempts=3
resilience4j.retry.instances.payments.wait-duration=200ms
resilience4j.retry.instances.payments.enable-exponential-backoff=true

Timeouts, capped retries with jitter, and circuit breakers are what turn an incident into a blip.