Debugging a failure across services

Key points

• Start from user impact, not from a dashboard. Which endpoint, which error, since when, and what fraction of requests - that scopes everything that follows.
• Use the trace before the logs. A trace shows where the time or the error actually is; logs tell you why once you know where. Searching logs first is how an hour disappears.
• Correlate by change. Most incidents follow a deploy, a configuration change or a traffic shift, so the first question is what changed in the window before the symptom started.
• Distinguish saturation from failure. Saturation shows rising latency and queue depth across many endpoints; failure shows errors concentrated in one. They need opposite responses - shed load versus fix or roll back.
• Beware the loudest service. In a cascade the service that reports the most errors is usually the victim, not the cause - it is the one whose timeouts fired first.
• Check the boring things before the interesting ones: connection pool exhaustion, disk full, certificate expiry, DNS, and a config value that differs in one environment.

Example

/*
 * A workable order of operations. Roughly cheapest and most-likely first.
 *
 * 1. SCOPE      which endpoint, what error, since when, what percentage
 * 2. CHANGE     deploys, config, feature flags, traffic in the last hour
 * 3. TRACE      one failing trace end to end - where is the time or error
 * 4. SATURATE   pool usage, queue depth, CPU, GC on the implicated service
 * 5. LOGS       filtered BY TRACE ID, not by keyword
 * 6. DEPENDS    is the downstream healthy, or reporting our calls as errors
 */

# The trace id makes cross-service correlation a single query.
$ curl -H "X-Trace-Id: $(uuidgen)" https://api.internal/orders/42

# Then in the log platform - every service, one request:
#   traceId:"8f3c1a" | sort by @timestamp

# Saturation vs failure, from metrics alone:
#
#   SATURATION                       FAILURE
#   latency up across many endpoints  errors on one endpoint
#   queue depth / pool usage rising   pool usage normal
#   error rate rises LATE (timeouts)  error rate rises FIRST
#   -> shed load, scale, throttle     -> roll back, fix, fail over

# The pool exhaustion check - unglamorous and very often the answer:
$ curl -s localhost:8080/actuator/metrics/hikaricp.connections.pending
#   value > 0 sustained -> requests are queuing for a connection,
#   and every symptom upstream is a consequence of that.

Scope, then what changed, then one trace - and remember the service shouting loudest in a cascade is usually the victim.