Backpressure: what to do when you cannot keep up

Key points

• A system given more work than it can do has exactly four options: buffer it, drop it, slow the producer, or fall over. Only the last one is not a design decision.
• Unbounded buffering is the default failure. A queue that grows without limit converts a throughput problem into a latency problem and then a memory problem, and the work it eventually does may already be worthless.
• Bounded queues force the decision to be explicit. When the queue is full you must choose: block the producer, reject the request, or drop the oldest item.
• Load shedding - rejecting excess early - keeps latency sane for the requests you do accept. Accepting everything under overload means everyone gets a slow response and many time out anyway.
• Shed by value where you can. Rejecting a health check or a background refresh is far better than rejecting a checkout, and priority-aware shedding is what makes degradation graceful.
• Backpressure must propagate. If a service slows down but its callers keep accepting at full rate, the pressure simply moves upstream - which is how one slow component saturates a whole system.

Example

/*
 * THE FOUR OPTIONS AT THE LIMIT:
 *
 *   BUFFER    absorb the burst          bounded, or it becomes the failure
 *   DROP      reject the excess         fast, honest, keeps latency sane
 *   BLOCK     slow the producer         only works if the producer can wait
 *   COLLAPSE  do nothing                the default if you do not choose
 */

// BOUNDED, with an explicit policy. Never Executors.newFixedThreadPool
// with its unbounded queue - that is option four wearing a disguise.
@Bean
ThreadPoolTaskExecutor workExecutor() {
    ThreadPoolTaskExecutor ex = new ThreadPoolTaskExecutor();
    ex.setCorePoolSize(20);
    ex.setMaxPoolSize(50);
    ex.setQueueCapacity(200);                       // BOUNDED
    // Full queue -> run on the caller thread. The caller is slowed down,
    // which propagates backpressure upstream instead of hiding it.
    ex.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
    return ex;
}

// SHED BY VALUE - not all requests are worth the same.
@Component
class PriorityLoadShedder {

    private final AtomicInteger inFlight = new AtomicInteger();

    boolean admit(RequestPriority p) {
        int current = inFlight.get();
        // Cheap work is shed first, so capacity is kept for what matters.
        return switch (p) {
            case CRITICAL -> current < 1000;   // checkout, payment
            case NORMAL   -> current < 700;    // browsing
            case LOW      -> current < 400;    // prefetch, analytics
        };
    }
}

/*
 * WHY SHEDDING BEATS QUEUEING UNDER OVERLOAD:
 *
 *   accept everything:  10,000 in flight, every response takes 30s,
 *                       clients time out at 5s -> 100% failure, and the
 *                       server did all the work anyway
 *
 *   shed to 1,000:      1,000 served in 200ms, 9,000 get a fast 429
 *                       -> 10% succeed properly instead of 0%
 *
 * And a fast 429 is actionable: the client can back off. A 30-second
 * timeout tells it nothing.
 */

Bound every queue and shed by priority - accepting everything under overload means serving no one while doing all the work.