Kafka Connect and CDC

Key points

• Connect is a worker cluster that runs connectors. Source connectors pull data into Kafka, sink connectors push it out.
• A connector is JSON posted to the REST API. No poll loop, no offset handling, no service of your own to deploy and monitor.
• CDC with Debezium tails the database log, so every insert, update and delete becomes an event - including the row as it was before.
• SMTs (single message transforms) rename fields, route topics or mask values inline, without a stream processor in the middle.
• Run Connect in distributed mode. It stores config, offsets and status in internal topics and rebalances tasks across workers.
• Tasks are the unit of parallelism, and on the sink side they are still capped by the partition count.

Example

// POST /connectors  -- source: Postgres row changes become events
{
  "name": "orders-cdc",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres",
    "database.dbname": "shop",
    "topic.prefix": "db",
    "table.include.list": "public.orders",
    "transforms": "unwrap",
    "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState"
  }
}

// POST /connectors  -- sink: the same events land in a search index
{
  "name": "orders-search",
  "config": {
    "connector.class": "io.aiven.kafka.connect.opensearch.OpensearchSinkConnector",
    "topics": "db.public.orders",
    "connection.url": "http://opensearch:9200",
    "tasks.max": "3",
    "key.ignore": "false"
  }
}

If a connector already exists, writing a consumer to do the same job is a choice you have to justify.