Resilience: timeout, retry, breaker

Key points

• Timeout comes first. Every remote call needs one, or a hung dependency quietly eats your thread pool.
• Retry only what is safe and transient: connection resets, 503, 429. Retrying a non-idempotent POST charges someone twice.
• Retry with exponential backoff and jitter. Synchronised retries turn a two-second blip into a stampede.
• Circuit breaker watches the failure rate. It opens and fails fast for a cool-off, then half-opens to test the water.
• Bulkhead caps concurrent calls per dependency, so one slow neighbour cannot consume every thread you have.
• Resilience4j is the standard library on Spring Boot 3+ (Hystrix is retired). A service mesh can do the same at the network layer.

Example

// Dependency: io.github.resilience4j:resilience4j-spring-boot3
@Service
class InventoryGateway {

    private final RestClient inventory;

    InventoryGateway(RestClient inventory) { this.inventory = inventory; }

    // Default Resilience4j aspect order:
    // Retry ( CircuitBreaker ( Bulkhead ( call ) ) )
    @Retry(name = "inventory")
    @CircuitBreaker(name = "inventory", fallbackMethod = "unknownStock")
    @Bulkhead(name = "inventory")
    Stock stockFor(String sku) {
        return inventory.get()
                .uri("/api/stock/{sku}", sku)
                .retrieve()
                .body(Stock.class);
    }

    // Fallback: same signature plus the cause. Degrade, do not explode.
    Stock unknownStock(String sku, Throwable cause) {
        return Stock.unknown(sku);
    }
}

// application.yml
// resilience4j.retry.instances.inventory.max-attempts: 3
// resilience4j.retry.instances.inventory.wait-duration: 200ms
// resilience4j.retry.instances.inventory.enable-exponential-backoff: true
// resilience4j.circuitbreaker.instances.inventory.sliding-window-size: 50
// resilience4j.circuitbreaker.instances.inventory.failure-rate-threshold: 50
// resilience4j.circuitbreaker.instances.inventory.wait-duration-in-open-state: 10s
// resilience4j.bulkhead.instances.inventory.max-concurrent-calls: 20

Fail fast, fail small, and always have a degraded answer ready.