Serializers and Poison Pills

Kafka · lesson 18 of 34 · 4 min read

Pick a payload format, and stop one unparseable record from spinning a partition forever.

Key points

Kafka stores bytes and nothing else. A serializer turns your object into bytes; the consumer must have the matching deserializer.
JSON is readable and needs no infrastructure. Avro and Protobuf are compact and enforce a contract. Choose one per topic and stay there.
A record that will not deserialize throws before your listener runs. The offset never moves, so it is retried forever.
That is a poison pill. Wrap the real one in ErrorHandlingDeserializer so the error handler can route the failure.
Spring needs to know the target type: set spring.json.value.default.type, or let the producer send type headers.
Keep spring.json.trusted.packages tight. Deserializing whatever class a header names is a genuine remote code execution path.

Example

spring:
  kafka:
    producer:
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
    consumer:
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      # never the real deserializer directly -- wrap it
      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      properties:
        spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer
        spring.json.value.default.type: com.example.orders.OrderEvent
        spring.json.trusted.packages: com.example.orders
        spring.json.use.type.headers: false

Bytes in, bytes out. Guard the boundary or one bad record stops the partition.

This is a reading copy. The full lesson — with the visual explainer, the interactive lab and a Run button for the code — lives in the Kafka course, and every lesson in it is listed on the Kafka contents page.