Batching, Compression and Throughput

Kafka · lesson 20 of 34 · 4 min read

Turn the few knobs that genuinely move throughput, and know what each one costs.

Open this lesson in the learning hub

Key points

  • Throughput comes from batching. linger.ms=10 lets a batch fill; batch.size caps how big it gets.
  • Compression is per batch, so bigger batches compress better. lz4 is fast, zstd is smaller. Both beat none.
  • On the consumer, fetch.min.bytes and fetch.max.wait.ms trade a little latency for far fewer fetches.
  • Measure your handler before touching any of this. A 40ms database call per record is the bottleneck, not the client config.
  • Beyond that, parallelism is capped by partition count. More partitions cost memory, open file handles and slower leader elections.
  • Every one of these is a latency-for-throughput trade. On a low-volume topic, leave them alone.

Example

# ---- producer: bigger batches beat more threads ----
linger.ms=10
batch.size=65536
compression.type=lz4
buffer.memory=67108864
max.in.flight.requests.per.connection=5

# ---- consumer: fewer, fatter fetches ----
fetch.min.bytes=65536
fetch.max.wait.ms=100
max.partition.fetch.bytes=2097152
max.poll.records=500

# Rule of thumb: raise linger.ms before you raise partition count.

Batch harder before you scale wider. linger.ms is the cheapest throughput you will ever buy.

Offsets, Seeking and Replay The Outbox Pattern

This is a reading copy. The full lesson — with the visual explainer, the interactive lab and a Run button for the code — lives in the Kafka course, and every lesson in it is listed on the Kafka contents page.