Sizing Thread Pools for Optimal Throughput

Thread pool sizing comes down to whether the work is CPU-bound or I/O-bound — the right formula is different for each, and most Java backend services are I/O-bound.

CPU-bound work

• CPU-bound work uses the CPU continuously — computing a hash, parsing a large document, running a machine learning inference; the thread can't go faster than the CPU allows
• For CPU-bound pools: pool size = number of CPU cores + 1; the extra thread covers the brief moments a CPU thread is paused (GC, OS scheduling); more threads than this causes context-switching overhead that reduces throughput

I/O-bound work — Little's Law

• I/O-bound work spends most of its time waiting — waiting for a database query, an HTTP call, a file read; the thread is blocked and the CPU is free
• Little's Law: pool size = throughput x average latency; if the target is 200 requests/second and each request blocks for 100ms on I/O, you need 200 x 0.1 = 20 threads to keep all 20 blocked threads covered
• In practice for a Tomcat thread pool: start with max-threads = 200 for a typical Spring Boot service; measure thread pool saturation (active threads approaching max) under load and adjust

Spring Boot thread pool configuration

• server.tomcat.threads.max (default 200): the maximum number of request-handling threads; increase if threads are consistently saturated during load tests
• server.tomcat.accept-count (default 100): the accept queue size; when all threads are busy, new connections queue here; if the queue fills, connections are refused
• spring.task.execution.pool.max-size for @Async method pools; spring.task.scheduling.pool.size for @Scheduled tasks — these are separate from Tomcat's request thread pool

Measuring saturation

• Micrometer exposes tomcat.threads.busy and tomcat.threads.config.max; busy / max > 0.8 consistently means the thread pool is saturated; increase max-threads or reduce per-request latency (faster DB queries, connection pooling)