Kafka Producer Throughput — Batching and Compression
Fast recall
Quick revision
Core answer
Awareness of producer throughput tuning at the configuration level
Setting linger.ms=0 (default) for low-latency use cases while also needing high throughput — the two goals are in tension
Explain that linger.ms and batch.size are independent limits — whichever triggers first sends the batch
Interview-ready guide
Interview answer
Kafka producers batch messages instead of sending them individually. send() adds the message to a per-partition buffer. The buffer flushes when the batch is full (batch.size, default 16KB) or the linger time expires (linger.ms, default 0ms).
With linger.ms=0, messages send almost immediately — low latency, lower throughput. Setting linger.ms=20 lets messages accumulate, producing larger batches with fewer network round-trips.
Compression (lz4 is the best default) compresses each batch before sending. Larger batches compress more efficiently — another reason to increase linger.ms.
For high throughput: linger.ms=20, batch.size=65536, compression.type=lz4. Network requests drop 10-50x.
buffer.memory (default 32MB) is the total producer buffer. When exhausted, send() blocks and eventually throws TimeoutException.
In practice, I always set compression.type=lz4 and tune linger.ms based on latency requirements — 5ms for near-real-time, 20ms for analytics.
Kafka Producer Throughput — Batching and Compression
Direct Answer
Kafka producers batch messages before sending. Tune linger.ms (wait time to fill batch) and batch.size (max bytes). Add snappy or lz4 compression for 3-5x payload reduction. These are the highest-impact producer throughput settings.
Code Example
Producer throughput on Kafka is a function of batching (how many records per request) and compression (how small each batch is on the wire). Defaults are conservative; production-grade pipelines push 10× more.
spring:
kafka:
producer:
acks: all
properties:
# ----- Batching -----
linger.ms: 20 # wait up to 20ms to fill a batch (default 0 = no batching)
batch.size: 65536 # 64 KB per batch (default 16 KB)
buffer.memory: 67108864 # 64 MB total in-flight buffer (default 32 MB)
# ----- Compression -----
compression.type: lz4 # lz4 = 2-3x compression at near-zero CPU cost
# alternatives: snappy (similar), zstd (higher ratio, slightly more CPU), gzip (highest, slowest)
# ----- Pipelining + safety -----
enable.idempotence: true
max.in.flight.requests.per.connection: 5
retries: 2147483647
delivery.timeout.ms: 120000
@Service
@RequiredArgsConstructor
public class HighThroughputPublisher {
private final KafkaTemplate<String, OrderEvent> template;
private final MeterRegistry meter;
public void publish(OrderEvent ev) {
// Hot loop: do NOT block on .get() — use the async future
Timer.Sample sample = Timer.start(meter);
template.send("orders", ev.orderId().toString(), ev)
.whenComplete((res, ex) -> {
sample.stop(meter.timer("kafka.send", "topic", "orders"));
if (ex != null) meter.counter("kafka.send.fail").increment();
});
}
}
| Setting | Default | Tuned for throughput | Effect |
|---|---|---|---|
linger.ms | 0 | 10-50 | tiny latency for huge batching gain |
batch.size | 16 KB | 64-256 KB | larger requests, fewer round-trips |
buffer.memory | 32 MB | 64-256 MB | more room for in-flight batches |
compression.type | none | lz4 (or zstd) | 2-5× smaller wire payload |
max.in.flight.requests.per.connection | 5 | 5 (keep ≤5 with idempotence) | parallel pipelining without OOO |
- Compression is almost always free. JSON / Avro typically compress 3-5×.
lz4is near-zero CPU cost;zstdgives the best ratio at modest cost. The broker stores compressed batches as-is — saves disk, network, and consumer download. - Linger.ms vs latency:
linger.ms=20adds at most 20 ms tail latency but can improve throughput 5-10× on bursty workloads. Tune it to your latency budget, not to zero. - Batch size > 64 KB has diminishing returns and bumps p99 latency. Watch
record-send-rate,batch-size-avg, andcompression-rate-avgin Micrometer — these tell you whether batching is actually happening. - Always pair these with
acks=all+enable.idempotence=trueso the throughput gain doesn't come at the cost of duplicates or silent loss.
Common Mistakes & Pitfalls
Setting linger.ms=0 (default) for low-latency use cases while also needing high throughput — the two goals are in tension
Check your understanding
Answer these without looking back at the article.
- What is the relationship between linger.ms and latency vs throughput?
- How do you choose between snappy, lz4, gzip, and zstd compression?
- How do you monitor producer batch fill rate and compression ratio?
Related interview prep categories
2Want a guided curriculum order?
These modules are also sequenced in our flagship backend interview track — open it when you prefer an ordered path instead of browsing by topic.