Debezium 1.4.0.Alpha2 Released

PostgreSQL Snapshotter completion hook

The PostgreSQL Snapshotter API is a contract that allows for the customization of the snapshot process. This API was introduced in 0.9.3.Final and has continued to evolve in the releases since.

A new backward compatible completion hook has been added:

void snapshotCompleted()

This new hook is called by the snapshot process when the snapshot has concluded, allowing implementations to clean-up any resources it may have allocated prior streaming changes.

ExtractNewRecordState SMT field renaming support

One of the features of the ExtractNewRecordState SMT is that the transformation can retain parts of the original message in the transformed message’s header or payload. This release extends this feature to allow specifying a new name to be used for the field when added to the message header or payload.

For example, to add the source database’s event timestamp to the message header using the new renaming feature, the SMT configuration would be:

transforms=unwrap
transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
transforms.unwrap.add.headers=source.ts_ms:timestamp

The format of the add.headers and add.fields configuration options have been improved to support a comma-separated list of fields with the syntax <OLD_FIELD>[:NEW_FIELD]. The above emitted message’s headers would now contain __timestamp rather than the default __source.ts_ms field.

This syntax improvement remains backward compatible.

Bugfixes

Also a number of bugs were fixed, e.g.:

Oracle throw "no snapshot found based on specified time" when running flashback query DBZ-1446
Exception when PK definition precedes column definition DBZ-2580
Patroni can’t stop PostgreSQL when Debezium is streaming DBZ-2617
ChangeRecord informations don’t connect with the TableSchema DBZ-2679
MySQL connector fails on a zero date DBZ-2682
Oracle LogMiner doesn’t support partition tables DBZ-2683
DB2 doesn’t start reliably in OCP DBZ-2693
Dropped columns cause NPE in SqlServerConnector DBZ-2716
Timestamp default value in 'yyyy-mm-dd' format fails MySQL connector DBZ-2726
Connection timeout on write should retry DBZ-2727
No viable alternative at input error on "min" column DBZ-2738
SQLServer CI error in SqlServerConnectorIT.whenCaptureInstanceExcludesColumnsAndColumnsRenamedExpectNoErrors:1473 DBZ-2747
debezium-connector-db2: DB2 SQL Error: SQLCODE=-206 on DB2 for z/OS DBZ-2755
no viable alternative at input 'alter table order drop CONSTRAINT' DBZ-2760
Tests are failing on macos DBZ-2762

A big thank you to all the contributors from the community who worked on this release: Alexander Iskuskov, Alisa Houskova, Anisha Mohanty, Andrey Ignatenko, Arik Cohen, Babur Duisenov, Grant Cooksey, Ganesh Ramasubramanian, Hoa Le, Kewei Shang, Mans Singh, Mohamed Pudukulathan, Peter Urbanetz, Ramesh Reddy, Sergei Morozov, and Vadzim Ramanenka.

Chris Cranford

Chris is a software engineer at Red Hat. He previously was a member of the Hibernate ORM team and now works on Debezium. He lives in North Carolina just a few hours from Red Hat towers.

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.