Being able to configure which columns of a table should go into the key of corresponding Kafka messages has been a long-standing feature request (DBZ-1015). To recap, by default the message key of Debezium’s data change events will contain the primary key column(s) of the represented table. That’s a sensible default, but sometimes more flexibility is desirable:

The second case is especially of interest when building Kafka Streams applications that join multiple CDC topics: in general, topic joins can only be done if the message key is the same on both sides of the join.

For instance let’s assume we have two tables, Customer and CustomerDetails with different primary keys and a foreign key relationship from CustomerDetails to Customer. By choosing that foreign key column as the message key for customer detail change events, the two table streams could be joined without the need for re-keying the customer details topic. To do so, the new message.key.columns option can be used like so:

Customizable message keys are supported for all the relational Debezium connectors.

From previous announcements you might remember that Debezium’s Cassandra connector is a bit different from the other ones, as it’s not based on the Kafka Connect framework. As such, until now it didn’t support the notion of configurable message serializers; Avro was the only supported message format.

As part of the ongoing efforts to align the Cassandra connector more closely with the other ones, it now allows to configure different serializers, so you also could use JSON, ProtoBuf and other formats (DBZ-1405). The serializer framework from Kafka Connect is reused for that, so you can leverage all the existing serializers and configure them exactly the same way as done for any other connector.

TOAST columns are a mechanism in Postgres for dealing with column values that exceed the page size limit (typically 8 KB). While the usage of TOAST is transparent when interacting with the database itself, this is not the case when obtaining change events via logical decoding. As TOASTed values are not stored within the physical data row itself, logical decoding does not expose the value of unchanged TOAST columns (unless the column is part of the table’s replica identity).

This situation used to be handled in different ways by the the logical decoding plug-ins supported by Debezium (pgoutput, decoderbufs and wal2json), one approach being the retrieval of such column "out of bands".

Unfortunately, there’s no way for savely doing this when considering concurrent writes to such record. So we reworked how TOAST columns are handled: if a TOAST column’s value hasn’t changed and that column isn’t part of the table’s replica identity, its value will not be contained in UPDATE or DELETE events. Instead, a configurable marker value will be exported in this case (defaulting to __debezium_unavailable_value).

This avoids the race conditions that were possible before, but of course raises the question how consumers should deal with this marker value. There are multiple possible answers to that:

Thinking about it, the last approach might be an interesting topic for a future blog post :-)

Besides these feature implementations, this release contains a number of bugfixes, too:

Testing for MongoDB has been expanded to also cover version 4.2 (DBZ-1389), and the Postgres driver has been updated to the latest and greatest version 42.2.7 (DBZ-1462). We’re also happy to report that going forward, the Debezium container images are also available in the quay.io container registry (DBZ-1178).

Overall, not less than 30 issues were fixed in the 0.10 CR2 release. Please refer to the release notes for the complete list of addressed issues and the upgrading procedure.

This release wouldn’t be possible without all the fantastic people from the Debezium community who contributed: Bingqin Zhou, Ching Tsai, Guillaume Rosauro, Javier Holguera Jaromir Hamala, Josh Arenberg and Taylor Rolison.

Many thanks to all of you!