Idempotent and Transactional Producers

Key points

• An idempotent producer stamps each batch with a producer id and sequence number, so the broker discards a batch it has already written.
• It is on by default since Kafka 3.0 (enable.idempotence=true) and it also keeps ordering with up to 5 in-flight requests.
• It only dedupes retries inside one producer session. Restart your app and resend the same event and you get a duplicate.
• Transactions add a stable transactional.id. Writes across many partitions plus the consumer offsets then commit or abort together.
• Consumers must set isolation.level=read_committed, otherwise they will read records from aborted transactions.
• The cost is extra round trips and consumers waiting for commits. Pay it only for genuine Kafka-to-Kafka pipelines.

Example

// Producer: stable transactional.id, one per instance
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "enrich-orders-1");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
// Consumer: MUST disable auto commit and read only committed data
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");

producer.initTransactions();

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
    if (records.isEmpty()) continue;

    producer.beginTransaction();
    try {
        Map<TopicPartition, OffsetAndMetadata> offsets = new HashMap<>();
        for (ConsumerRecord<String, String> r : records) {
            producer.send(new ProducerRecord<>("orders-enriched", r.key(), enrich(r.value())));
            offsets.put(new TopicPartition(r.topic(), r.partition()),
                        new OffsetAndMetadata(r.offset() + 1));
        }
        // Offsets are committed inside the same transaction
        producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
        producer.commitTransaction();
    } catch (KafkaException e) {
        producer.abortTransaction();
    }
}

Idempotence is free and already on. Transactions are not free — reserve them for read-process-write.