Capacity planning: threads, pools and Little law

Key points

• Little law is the whole of it: concurrency equals arrival rate multiplied by latency. At 500 requests per second with 200ms average latency, you have 100 requests in flight at any moment.
• That number, not a guess, is what sizes the thread pool. And it must account for the tail: sizing for average latency leaves nothing for the p99 that arrives during an incident.
• Connection pools multiply by replica count, and this is the mistake that takes databases down. Fifty pods with a pool of twenty is a thousand connections, far beyond what most engines handle well.
• Database pools should be small. Beyond the parallelism the engine can genuinely use, more connections add contention rather than throughput - queueing in the pool is cheaper than thrashing inside the database.
• Autoscaling on CPU alone hides this. Scaling out multiplies connections, so a scale-up event triggered by load can be the thing that finally exhausts the database.
• When pods must scale beyond what the database tolerates, put a proxy such as PgBouncer in front so many application connections multiplex onto few database ones.

Example

/*
 * LITTLE LAW:  L = lambda x W
 *   L      = concurrent requests in the system
 *   lambda = arrival rate (req/s)
 *   W      = average time in the system (s)
 *
 * Worked example - 500 req/s, 200ms average:
 *   L = 500 x 0.2 = 100 concurrent requests
 *
 * Now size for the TAIL, not the average. At p99 = 800ms:
 *   L = 500 x 0.8 = 400 in flight during a bad window
 *
 * A 100-thread pool sized on the average queues badly exactly when it
 * matters most. This is why headroom is not waste.
 */

/*
 * CONNECTION MATHS - where databases actually fall over:
 *
 *   pods x pool size = total connections
 *    50  x    20     = 1,000   <- far past what most engines handle well
 *
 * Postgres default max_connections is 100. MySQL 151. Each connection
 * also costs server memory, so "just raise the limit" trades one failure
 * for another.
 *
 * Work backwards from the database instead:
 *   database can serve   ~100 connections
 *   expected max pods     50
 *   -> pool size           2 per pod
 *
 * A pool of 2 sounds wrong and is usually right. The database has finite
 * cores; queueing in the pool beats thrashing in the engine.
 */

spring:
  datasource:
    hikari:
      maximum-pool-size: 5        # NOT the default 10, at this replica count
      minimum-idle: 2
      connection-timeout: 2000    # fail fast rather than queue invisibly
      leak-detection-threshold: 30000

Concurrency is arrival rate times latency, and connection count is pool size times replicas - size both from the tail, not the average.