Retry amplification and metastable failure

Key points

• Retries multiply through a call chain. Three layers each retrying three times is up to 27 calls for one request - exponential in the depth of the chain.
• That amplification arrives exactly when the downstream is least able to absorb it, which is how a recoverable degradation becomes a total one.
• The discipline is to retry at one layer only, normally the one closest to the failure, and let every other layer fail fast.
• A retry budget is stronger than a per-call limit: cap retries as a fraction of total traffic, say ten percent, so no failure pattern can amplify beyond a known factor however many callers there are.
• Metastable failure is the endgame. A spike fills queues and triggers retries; the retries generate enough load to keep the system saturated even after the original trigger has gone. The system is now sustaining its own outage.
• Metastable states do not recover on their own. Getting out needs load shedding, a deliberate drain, or turning traffic off and back on - which is why load shedding must exist before you need it.

Example

/*
 * AMPLIFICATION - why "retry everywhere" is not defensive.
 *
 *   gateway   retries 3x
 *     +-- orders    retries 3x
 *           +-- inventory  retries 3x
 *
 *   1 user request -> 3 x 3 x 3 = 27 calls to inventory
 *
 * At 1,000 req/s that is 27,000 req/s hitting a service that is already
 * struggling. The retries ARE the outage now.
 */

// Retry at ONE layer, with a budget and jitter.
@Bean
Retry inventoryRetry() {
    return Retry.of("inventory", RetryConfig.custom()
            .maxAttempts(3)
            // Jitter matters: without it, every client that failed together
            // retries together, producing repeated synchronised bursts.
            .intervalFunction(IntervalFunction
                    .ofExponentialRandomBackoff(Duration.ofMillis(100), 2.0, 0.5))
            // Never retry what is not safe to repeat, or what says "stop".
            .retryOnException(e -> e instanceof TimeoutException
                    || e instanceof ConnectException)
            .build());
}

// Inner layers fail fast - the retry belongs to exactly one of them.
@Bean
Retry ordersRetry() {
    return Retry.of("orders", RetryConfig.custom().maxAttempts(1).build());
}

/*
 * ESCAPING A METASTABLE STATE - the trigger is already gone:
 *
 *   t=0   downstream slows for 10s (GC pause, deploy, whatever)
 *   t=10  the cause is gone, but queues are full and retries are in flight
 *   t=60  still saturated - the retry load alone sustains it
 *
 * Waiting does not help. Shed load, drain the queues, then let traffic back.
 */

Retry at one layer with a budget and jitter - and build load shedding before you need it, because a metastable system will not recover on its own.