Sizing a thread pool

Key points

• A pool fills the core threads first, then the queue, and only adds threads up to max once the queue is full.
• That order surprises people: with an unbounded queue the max size is never reached at all.
• CPU-bound work wants roughly one thread per core. I/O-bound work wants cores times one plus the wait/compute ratio.
• Always bound the queue. An unbounded queue hides overload until the heap runs out.
• CallerRunsPolicy pushes rejected work back onto the caller, which slows producers down instead of dropping tasks.
• Name your threads and watch queue depth. A queue that is always full means the pool is too small or the tasks are too slow.

Example

import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;

public class Main {
    public static void main(String[] args) throws InterruptedException {
        int cores = Runtime.getRuntime().availableProcessors();
        System.out.println("cores        : " + cores);
        System.out.println("cpu-bound    : about " + cores + " threads");
        System.out.println("io-bound     : about " + (cores * 4) + " threads (wait/compute = 3)");

        ThreadPoolExecutor pool = new ThreadPoolExecutor(
                2, 4, 30, TimeUnit.SECONDS,
                new ArrayBlockingQueue<>(2),                  // bounded queue
                new ThreadPoolExecutor.CallerRunsPolicy());   // back-pressure, never silent loss

        for (int i = 1; i <= 10; i++) {
            int job = i;
            pool.execute(() -> {
                sleep(60);
                System.out.println("job " + job + " on " + Thread.currentThread().getName());
            });
        }
        pool.shutdown();
        pool.awaitTermination(5, TimeUnit.SECONDS);

        System.out.println("largest pool : " + pool.getLargestPoolSize() + " threads (max was 4)");
        System.out.println("ran in pool  : " + pool.getCompletedTaskCount() + " of 10, the rest ran on main");
    }

    static void sleep(long ms) {
        try { Thread.sleep(ms); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
    }
}

Core, queue, max, rejection: choose all four, or the defaults choose badly for you.