Schemas and Safe Evolution

Key points

• A topic is a contract. Free-form JSON means every consumer guesses, and one renamed field breaks production quietly.
• A Schema Registry stores the schema; each record carries only a small schema id. Avro, Protobuf and JSON Schema are all supported.
• BACKWARD compatibility (the default) means new consumers can read old data. Safe moves: add a field with a default, remove a field.
• FORWARD means old consumers can read new data. Choose it when producers deploy before consumers.
• Never rename or change the type of a field. Add the new one, dual-write, migrate consumers, then remove the old one.
• Compatibility is checked when the schema is registered, so a breaking change is rejected before it ever reaches a topic.

Example

spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
      properties:
        schema.registry.url: http://localhost:8081
        # Register schemas from CI, never from running production code
        auto.register.schemas: false
        use.latest.version: true
    consumer:
      group-id: billing-service
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer
      properties:
        schema.registry.url: http://localhost:8081
        specific.avro.reader: true

Add fields with defaults. Never rename. Let the registry say no on your behalf.