Kafka Streams in One Lesson

Key points

• Kafka Streams is a library, not a cluster. It runs inside your JVM and scales by starting more instances of your app.
• A KStream is an unbounded sequence of events. A KTable is the latest value per key - the same log read as state.
• Stateful operators (count, joins, windows) keep a local RocksDB store backed by a compacted changelog topic.
• Lose an instance and its replacement rebuilds the store from that changelog. That is why local state is safe to rely on.
• Changing the key forces a repartition through an internal topic. It is automatic, and it doubles the write volume for that step.
• Use Streams for joins, aggregation and windowing. For a listener that just calls a service, a plain consumer is simpler and clearer.

Example

StreamsBuilder builder = new StreamsBuilder();

KStream<String, OrderEvent> orders =
        builder.stream("orders", Consumed.with(Serdes.String(), orderSerde));

KTable<String, Long> paidPerCustomer = orders
        .filter((key, order) -> "PAID".equals(order.status()))
        // re-keying by customer forces a repartition topic behind the scenes
        .groupBy((key, order) -> order.customerId(),
                 Grouped.with(Serdes.String(), orderSerde))
        .count(Materialized.as("paid-per-customer"));

paidPerCustomer.toStream()
        .to("paid-counts", Produced.with(Serdes.String(), Serdes.Long()));

// application.id is the consumer group id AND the internal topic prefix.
// Change it and you start from scratch with brand new state.
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "order-stats");

KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();

KStream is events, KTable is state, and every store has a changelog topic behind it.