Quotas and noisy neighbours

Key points

• A shared cluster has no natural protection: one badly behaved producer can consume all the broker network bandwidth and degrade every other application on it.
• Kafka enforces quotas per client id, per user, or per combination - a byte rate for produce and fetch, and a request percentage for CPU time.
• Enforcement is by throttling, not rejection. The broker delays its response to bring the client back under its limit, so the client slows down rather than failing.
• That delay is invisible unless you look for it. A client hitting a quota simply appears slow, so produce-throttle-time-avg and fetch-throttle-time-avg are the metrics that turn a mysterious slowdown into an obvious one.
• The request quota is the subtle one. A client sending huge numbers of tiny requests can exhaust broker request-handler threads while using very little bandwidth, and only a request percentage quota catches that.
• Quotas are per broker, not cluster-wide. A 10 MB/s quota across six brokers permits up to 60 MB/s in total, which is a common sizing mistake.

Example

# Byte-rate quota for one application.
$ kafka-configs.sh --bootstrap-server localhost:9092 --alter \
    --add-config 'producer_byte_rate=10485760,consumer_byte_rate=20971520' \
    --entity-type clients --entity-name reporting-service

# Request-rate quota: percentage of ONE request handler thread.
# 200 means the client may use two full threads.
$ kafka-configs.sh --bootstrap-server localhost:9092 --alter \
    --add-config 'request_percentage=200' \
    --entity-type clients --entity-name reporting-service

# A default for everything not named explicitly - the safety net that
# matters most, because the client that hurts you is the one nobody knew about.
$ kafka-configs.sh --bootstrap-server localhost:9092 --alter \
    --add-config 'producer_byte_rate=5242880' \
    --entity-type clients --entity-default

# Confirm a client is being throttled rather than genuinely slow:
#
#   kafka.producer:type=producer-metrics,client-id=reporting-service
#     produce-throttle-time-avg   0     -> not throttled, it really is slow
#     produce-throttle-time-avg  85     -> throttled 85ms per request
#
# Remember quotas are PER BROKER:
#   10 MB/s quota x 6 brokers = up to 60 MB/s cluster-wide from one client.

Quotas throttle rather than reject, so a limited client just looks slow - check throttle-time before chasing a phantom performance problem.