Latency vs Throughput: Definitions and Optimization Strategies

Latency and throughput measure different things and are optimized differently — which means a change that reduces latency may increase throughput, reduce it, or have no effect, depending on where the bottleneck is.

The definitions

• Latency is how long one request takes — the user's wait from sending the request to receiving the response; measured in milliseconds per request; P99 latency is the experience of the worst-affected 1% of users
• Throughput is how many requests the system handles per unit of time — requests per second; the ceiling is determined by the bottleneck resource (CPU, I/O, threads, connection pool)

Latency optimization — reduce per-request work

• Eliminate unnecessary operations on the critical path: a database query that joins five tables when only two columns are needed; a synchronous HTTP call to a dependency that could be async; JSON deserialization of a large payload when only one field is used
• Caching: serve responses from Redis in microseconds instead of database queries in milliseconds; the first request is slow, every subsequent one is fast
• Reduce serialization overhead: Jackson ObjectMapper is expensive to construct; singleton ObjectMapper with specific features disabled; faster serializers like Jackson Afterburner or custom serializers for hot paths

Throughput optimization — remove the bottleneck

• Throughput is limited by the bottleneck resource; increasing throughput means identifying and removing that bottleneck
• If the bottleneck is the thread pool (threads saturated): add more threads (if I/O-bound) or reduce per-request CPU work; or switch to reactive/non-blocking I/O (WebFlux) which eliminates thread-per-request
• If the bottleneck is the database: add read replicas, add indexes, implement caching; the database often becomes the bottleneck before the application does

The trade-offs

• Batching increases throughput at the cost of latency — batching 100 writes in one database transaction reduces I/O but each write waits until the batch is full
• Parallelism reduces latency (fan-out multiple I/O calls) at the cost of more threads — CompletableFuture.allOf() runs multiple downstream calls in parallel; each call uses a thread