It’s my pleasure to announce the release of Debezium 0.7.2!
Amongst the new features there’s support for geo-spatial types, a new snapshotting mode for recovering a lost DB history topic for the MySQL connector, and a message transformation for converting MongoDB change events into a structure which can be consumed by many more sink connectors. And of course we fixed a whole lot of bugs, too.
Debezium 0.7.2 is a drop-in replacement for previous 0.7.x versions. When upgrading from versions earlier than 0.7.0, please check out the release notes of all 0.7.x releases to learn about any steps potentially required for upgrading.
A big thank you goes out to our fantastic community members for their hard work on this release: Andrey Pustovetov, Denis Mikhaylov, Peter Goransson, Robert Coup, Sairam Polavarapu and Tom Bentley.
Now let’s take a closer look at some of new features.
MySQL Connector
The biggest change of the MySQL connector is support for geo-spatial column types such as GEOMETRY
, POLYGON
, MULTIPOINT
etc.
There are two new logical field types — io.debezium.data.geometry.Geometry
and io.debezium.data.geometry.Geography
— for representing geo-spatial columns in change data messages.
These types represent geo-spatial data via WKB ("well-known binary") and SRID (coordinate reference system identifier),
allowing downstream consumers to interpret the change events using any existing library with support for parsing WKB.
A blog post with more details on this will follow soon.
The new snapshotting mode schema_only_recovery
comes in handy
when for some reason you lost (parts of) the DB history topic used by the MySQL connector.
It’s also useful if you’d like to compact that topic by re-creating it.
Please refer to the connector documentation for the details of this mode,
esp. when it’s safe (and when not) to make use of it.
Another new feature related to managing the size of the DB history topic is the option to control whether to include all DDL events or only those pertaining to tables captured as per the whitelist/blacklist configuration. Again, check out the connector docs to learn more about the specifics of that setting.
PostgreSQL Connector
Similar to the MySQL connector, there’s largely improved support for geo-spatial columns in Postgres now. More specifically, PostGIS column types can be represented in change data events now. Thanks a lot for Robert Coup who contributed this feature!
Also the support for Postgres array columns has been expanded,
e.g. we now support to track changes to VARCHAR
and DATE
array columns.
Note that the connector doesn’t yet work with geo-spatial array columns (should you ever have those),
but this should be added soon, too.
If you’d like to include just a subset of the rows of a captured table in snapshots, you may like the ability to specify dedicated SELECT statements to do so. For instance this can be used to exclude any logically deleted records — which you can recognize based on some flag in that table — from the snapshot.
A few bugs in this connector where reported and fixed by community members, too, e.g. the connector can be correctly paused now (thanks, Andrey Pustovetov), and we fixed an issue which could potentially have committed an incorrect offset to Kafka Connect (thanks, Thon Mekathikom).
MongoDB Connector
If you’ve ever compared the structures of change events emitted by the Debezium RDBMS connectors (MySQL, Postgres) and the MongoDB connector, you’ll know that the message structure of the latter is a bit different than the others. Due to the schemaless nature of MongoDB, the change events essentially contain a String with a JSON representation of the applied insert or patch. This structure cannot be consumed by existing sink connectors, such as the Confluent connectors for JDBC or Elasticsearch.
This gets possible now by means of a newly added single message transformation (SMT), which parses these JSON strings and creates a structured Kafka Connect record from it (thanks, Sairam Polavarapu!). When applying this SMT to the JDBC sink connector, you can now stream data changes from MongoDB to any supported relational database.
Note that this SMT is work-in-progress, details of its emitted message structure may still change. Also there are some inherent limitations to what can be achieved with it, if you e.g. have arrays in your MongoDB documents, the record created by this SMT will be structured accordingly, but many sink connectors cannot process such structure.
We have some ideas for further development here, e.g. there could be an option for flattening out (non-array) nested structures,
so that e.g. { "address" { "street" : "..." } }
would be represented as address_street
,
which then could be consumed by sink connectors expecting a flat structure.
The new SMT is described in detail in our docs.
What’s next?
Please see the full change log for more details and the complete list of issues fixed in Debezium 0.7.2.
The 0.7.3 release is scheduled for February 14th.
We’ll focus on some more bug fixes, also we’re working on having Debezium regulary emit heartbeat messages to a dedicated topic. This will be practical for diagnostic purposes but also help to regularly trigger commits of the offset in Kafka Connect. That’s beneficial in certain situations when capturing tables which only very infrequently change.
We’ve also worked out a roadmap describing our ideas for future work on Debezium, going beyond the next bugfix releases. While nothing is cast in stone, this is our idea of the features to add in the coming months. If you miss anything important on this roadmap, please tell us either in the comments below or send a message to our Google group. Looking forward to your feedback!
About Debezium
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
Get involved
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.