Queues and async work

Key points

• If the caller does not need the result, do not make them wait. Accept the request, publish an event, return 202.
• A queue absorbs spikes: producers keep running at full speed while consumers drain at whatever rate they can manage.
• Kafka keeps order only within a partition. Key by entity id so every event for one order lands on one partition, in sequence.
• Delivery is at-least-once, so duplicates happen. Consumers must be idempotent - that is the price of not losing messages.
• A message that always fails will block or loop forever. Cap the retries and send it to a dead-letter topic a human can inspect.
• A database write and a publish are two systems. Use the outbox pattern: store the event in the same transaction, publish it after.

Example

@Component
class OrderEvents {

    private final KafkaTemplate<String, OrderPlaced> kafka;

    OrderEvents(KafkaTemplate<String, OrderPlaced> kafka) { this.kafka = kafka; }

    // Key on orderId: all events for one order share a partition, so they stay ordered.
    void publish(OrderPlaced event) {
        kafka.send("orders.placed", event.orderId(), event);
    }

    @KafkaListener(topics = "orders.placed", groupId = "billing")
    void onOrderPlaced(OrderPlaced event) {
        // At-least-once delivery: this may run twice for the same event.
        billing.chargeOnce(event.orderId(), event.amount());
    }
}

// application.yml
// spring.kafka.consumer.group-id: billing
// spring.kafka.consumer.auto-offset-reset: earliest
// spring.kafka.listener.ack-mode: record

Queues trade instant answers for durability and elasticity. Consumer lag is the number you watch.