The handful of broker metrics worth alerting on

Key points

• Under-replicated partitions is the single most important number. Anything above zero for a sustained period means the cluster tolerates fewer failures than its replication factor promises.
• Offline partitions above zero means data is unavailable right now. That is a page, not a warning.
• Request latency is broken into phases, and the phase tells you the cause. Time in the request queue means the broker is saturated before it starts work - a thread-pool or overall capacity problem. Time in local means disk. Time in remote means waiting for replicas, which usually means acks=all and a slow follower.
• An ISR shrink and expand cycle that repeats is the clearest early warning there is: replicas keep falling behind and catching up, so the cluster is running at its limit even though nothing has failed yet.
• Watch the JVM too, but modestly. A broker should run a small heap - most of the machine belongs to the page cache - so a large heap and long GC pauses are usually a misconfiguration rather than load.
• On the client side, consumer lag alone is not enough. A consumer that catches an exception, logs it and commits anyway shows perfect lag while dropping every record - so processing rate and error rate belong beside it.

Example

# The four that justify waking someone up.
#
kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions
  # > 0 sustained -> durability is degraded. Alert.

kafka.controller:type=KafkaController,name=OfflinePartitionsCount
  # > 0 -> data is unavailable NOW. Page.

kafka.server:type=ReplicaManager,name=IsrShrinksPerSec
  # repeating shrink/expand -> the cluster is at its limit. Investigate early.

kafka.controller:type=KafkaController,name=ActiveControllerCount
  # must be exactly 1 across the cluster. 0 = no controller, 2 = split brain.

# Latency, broken down - the breakdown is the diagnosis.
kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce
  #   RequestQueueTimeMs   waiting to be picked up   -> saturated, size threads
  #   LocalTimeMs          leader writing to disk    -> disk or fsync pressure
  #   RemoteTimeMs         waiting for followers     -> acks=all + slow replica
  #   ResponseQueueTimeMs  waiting to be sent
  #   ResponseSendTimeMs   writing to the socket     -> network saturation

# Consumer side - lag is necessary but NOT sufficient.
kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*
  #   records-lag-max        how far behind
  #   records-consumed-rate  is it actually processing, or silently failing?
  #   fetch-latency-avg      is the broker or the network the constraint?

Under-replicated and offline partitions are the two that matter most, and the request-latency breakdown names the bottleneck for you.