Kafka

34 lessons · free to read

Topics, partitions, consumer groups and delivery guarantees, with real config.

Take this course in the learning hub

Lessons

• What Kafka Actually IsKnow what Kafka is, how it differs from a queue, and when reaching for it is the right call.
• Topics, Partitions, OffsetsUnderstand the three units Kafka is built from and why partition count is your main scaling dial.
• Producers and Message KeysSend records, choose keys on purpose, and know exactly which partition a record will land on.
• Consumers and Consumer GroupsLearn how a group splits partitions between members and why groups are how you scale reads.
• Rebalancing Without the PainKnow what triggers a rebalance, what it costs you, and the settings that stop it from hurting.
• What Kafka Guarantees About OrderGet precise about where ordering holds, where it does not, and how to design keys around it.
• acks, Retries and DurabilityConfigure a producer that will not quietly lose messages, and see what each ack level really costs.
• Replication and the ISRUnderstand leaders, followers and the in-sync replica set that decides whether a write survives.
• At-Most-Once vs At-Least-OncePick a delivery guarantee on purpose, and see that the choice is really about when you commit.
• Idempotent and Transactional ProducersSee how Kafka removes producer duplicates and how transactions make read-process-write atomic.
• Retention and Log CompactionControl how long data lives and use compaction to turn a topic into a keyed snapshot of state.
• Consumer Lag, The Metric That MattersMeasure how far behind your consumers are and read the number correctly when things go wrong.
• Spring Kafka in 30 LinesProduce and consume with Spring Boot, and know which auto-configured beans you actually control.
• Retries and Dead Letter TopicsStop one poison record from blocking a partition and route failures somewhere you can inspect them.
• Schemas and Safe EvolutionChange the shape of your events without breaking every consumer that already reads the topic.
• Brokers, Controllers and KRaftSee what a cluster is actually made of, and how KRaft replaced ZooKeeper for metadata.
• Run Kafka Locally in Five MinutesStart a single-node KRaft broker and prove it works with the console producer and consumer.
• Serializers and Poison PillsPick a payload format, and stop one unparseable record from spinning a partition forever.
• Offsets, Seeking and ReplayControl where a consumer starts, and rewind a group on purpose without guessing.
• Batching, Compression and ThroughputTurn the few knobs that genuinely move throughput, and know what each one costs.
• The Outbox PatternFix the dual-write problem: never treat your database and Kafka as two independent writes.
• Kafka Streams in One LessonBuild a stateful topology from KStream and KTable, and see where the state really lives.
• Kafka Connect and CDCMove data in and out of Kafka with configuration instead of yet another bespoke consumer.
• TLS, SASL and ACLsLock a cluster down: encrypt the wire, authenticate the client, then authorise every topic.
• Testing Kafka CodeTest producers and listeners against a real broker, without a shared environment or a mock.
• Multi-Cluster and Disaster RecoveryReplicate topics to a second cluster, and be honest about what a failover actually loses.
• How a partition is stored on diskSegments, indexes, page cache and zero-copy - the reasons Kafka is fast.
• The consumer threading modelWhy slow processing causes rebalance storms, and the pause/resume pattern that fixes it.
• Eager, cooperative and static membershipThree answers to "what happens to the group when one member restarts".
• What a transaction actually writes to the logMarkers, the last stable offset, and why an open transaction stalls consumers.
• Quotas and noisy neighboursStop one client saturating a cluster everyone else depends on.
• Partition reassignment without hurting the clusterMoving data between brokers is a bulk copy - throttle it or it becomes the outage.
• The handful of broker metrics worth alerting onWhere a broker is actually spending its time, and which numbers predict trouble.
• Tiered storage and long retentionKeeping months of history without sizing brokers for months of disk.