Flow control, memory alarms and the queue that eats the broker

Key points

• Unlike a log-based broker, RabbitMQ is a queue: messages are held until consumed. A queue with no consumer keeps growing, and growth is bounded by broker memory rather than by disk.
• When memory passes the high watermark - 40% of system RAM by default - the broker raises an alarm and blocks publishers. Producers simply stop, often with no error, which looks like a hung application.
• That blocking is the design working correctly. It is backpressure: the broker refuses to accept work it cannot hold rather than crashing.
• A disk alarm does the same thing when free space falls below the limit, and it is more common than people expect because persistent messages and the index both consume it.
• Lazy queues - now the default behaviour for classic queues in recent versions - keep messages on disk instead of memory, trading throughput for the ability to hold a large backlog safely.
• The real fix is bounding the queue: set x-max-length or a TTL so an unconsumed queue sheds messages to a dead-letter exchange rather than taking the broker down with it.

Example

# Is the broker blocking publishers right now?
$ rabbitmqctl status | grep -A3 alarms
#   {alarms,[memory]}        <- publishers are BLOCKED

$ rabbitmq-diagnostics memory_breakdown
#   queue_procs: 2.1 GB      <- messages held in memory
#   binary: 1.4 GB
#   connection_readers: ...

# The watermarks:
vm_memory_high_watermark.relative = 0.4      # 40% of system RAM
disk_free_limit.absolute = 2GB

---
# BOUND THE QUEUE. An unbounded queue is an outage waiting for a slow
# consumer.
# 
#   x-max-length          max messages; oldest are dropped or dead-lettered
#   x-max-length-bytes    same, by size
#   x-message-ttl         per-message expiry
#   x-overflow            drop-head | reject-publish | reject-publish-dlx

@Bean
Queue ordersQueue() {
    return QueueBuilder.durable("orders")
            .withArgument("x-max-length", 100_000)
            .withArgument("x-overflow", "reject-publish-dlx")  // shed to DLX
            .withArgument("x-message-ttl", 3_600_000)          // 1 hour
            .withArgument("x-dead-letter-exchange", "orders.dlx")
            .build();
}

---
# PREFETCH is the other half. Unlimited prefetch means one consumer pulls
# the whole queue into its own memory and the others sit idle.
spring:
  rabbitmq:
    listener:
      simple:
        prefetch: 20          # per consumer; NOT unlimited
        concurrency: 4
        max-concurrency: 16

#   prefetch too high -> uneven distribution, and memory in the consumer
#   prefetch = 1      -> perfectly even, and a round trip per message
#   20-50             -> a sensible default for most workloads

---
# QUORUM QUEUES for anything that matters: replicated, and they do not
# lose messages on a node failure the way a classic mirrored queue could.
#   .quorum()  on QueueBuilder, or x-queue-type: quorum

A memory alarm blocks publishers silently - bound every queue with a max length and a dead-letter exchange before that happens.