Testing Kafka Code

Key points

• Unit test the handler, not the broker. Keep business logic in a plain method so most of your tests never touch Kafka at all.
• For integration, Testcontainers starts a throwaway broker per test class. It behaves like production because it is the real thing.
• Spring also ships @EmbeddedKafka, which starts faster but runs an in-JVM broker with its own configuration quirks.
• Kafka is asynchronous, so never assert immediately after sending. Use Awaitility with a timeout instead of Thread.sleep.
• Use a unique group.id per test with auto.offset.reset=earliest, or the consumer misses earlier records.
• For Kafka Streams, TopologyTestDriver runs a whole topology in memory in milliseconds, with no broker at all.

Example

@SpringBootTest
@Testcontainers
class OrderListenerIT {

    @Container
    static final KafkaContainer KAFKA =
            new KafkaContainer(DockerImageName.parse("apache/kafka:4.0.0"));

    @DynamicPropertySource
    static void kafkaProps(DynamicPropertyRegistry registry) {
        registry.add("spring.kafka.bootstrap-servers", KAFKA::getBootstrapServers);
    }

    @Autowired KafkaTemplate<String, OrderEvent> template;
    @Autowired OrderRepository repository;

    @Test
    void listener_persists_the_order() {
        template.send("orders", "order-42", new OrderEvent("order-42", "PAID"));

        // the listener runs on another thread -- wait, do not sleep
        await().atMost(Duration.ofSeconds(10)).untilAsserted(() ->
                assertThat(repository.findById("order-42")).isPresent());
    }
}

A real broker in a container plus Awaitility. Sleeps and mocks both lie to you.