Debezium introduced the notion of skipped.operations some time ago. This feature allows connectors to control what change events are emitted during the streaming phase, optionally omitting certain change types such as updates or deletes.

The skipped.operations feature was originally implemented to act more like an SMT where it was a filter that was applied toward the end of the CDC pipeline just before an event was emitted. While this worked well, there was an optimization that could be achieved here because SQL Server records the change type in the capture table along side the change data. Using this knowledge, the query to fetch changes from the capture table was adjusted so that the changes returned from SQL Server are pre-filtered based on these "skipped operations".

Overall, this optimization should improve both the time spent during iterations of capturing changes when the connector is configured to use skipped.operations as well as reduce the overall garbage collection that the connector may do over its lifetime due to generating fewer objects that are later discarded.

Many thanks to Hossein Torabi for identifying this optimization and contributing it!

Debezium supports heartbeats across its other connectors, this feature is now available to Oracle!

A heartbeat is a critical part of insuring that a connector’s offsets remain synchronized with Kafka. Normally, the offsets are automatically maintained by Kafka Connect when we emit an event to a topic. But there are circumstances where event emission can be infrequent enough that stale offsets become a concern.

Regardless which adapter the Debezium Oracle connector is deployed to use, there are situations where events may be seen by the connector and in-memory offset state is updated but because we don’t emit an event to Kafka, the offsets aren’t synchronized to Kafka Connect. A great example is when the connector could be capturing changes for a table that changes very infrequently compared to other tables in the data source.

In order to guarantee that offsets remain synchronized with Kafka Connect, specifically during periods of low capture activity, the heartbeat.interval.ms configuration option can be set to periodically keep offsets in sync.

While we added support for BLOB and CLOB data types to the Debezium connector for Oracle in an earlier 1.6 pre-release, we believe that the work towards fully supporting these types is still an ongoing one.

There is a certain amount of overhead the connector has when handling BLOB and CLOB fields. Transactions which contain these data types require a reconciliation step at commit-time to merge certain events into a single logical emitted event. Additionally, LOB data types require a certain amount of memory footprint for their values to be kept in memory while the event to be emitted is constructed. Furthermore, users may not want to have LOB data emitted at all due to the size of the data.

So with that, the Oracle connector’s support for LOB data types is now an opt-in only feature. This means that OOTB the connector won’t attempt to capture LOB column data. If LOB columns need to be captured by the connector, the connector option, lob.enabled, must be set to true so that the connector will pickup and process those column types.

Lastly, we also felt that since there is still some ongoing effort for BLOB and CLOB support, it made sense to denote LOB support as incubating for the short-term until we believe the solution has matured. We strongly encourage users who want to capture LOB-based columns to enable LOB support and give it test drive and provide any and all feedback, both good or bad, so that we can continue to improve support for LOB columns.

The Debezium connector for Oracle also saw quite a number of small improvements (DBZ-3612, DBZ-3616, DBZ-3619, DBZ-3631). These improvements focused on improving the DDL parser as well as logging.

With the recent release of Quarkus 2.0.0.Final, the Quarkus Outbox extension is now based on Quarkus 2.0 (DBZ-3602).

Overall, 27 issues have been addressed in Debezium 1.6.0.CR1. We’re deeply grateful to all the community members contributing to this release:

Bob Roldan, Camile Sing, Chris Cranford, Derek Moore, Gunnar Morling, Hossein Torabi, Hyunjin Oh, Jakub Cechacek, JapuDCret, Jiri Pechanec, and Sergei Morozov!

With CR1 out, we’re nearing the end of the stablization phase for the 1.6 release cycle. You can expect possibly one more CR (candidate release), before the final release, which is planned for the end of the month, barring any unforeseen complications of cause. Besides some more bug fixes and documentation improvements we’re also intending to upgrade to Apache Kafka 2.8, which will allow you to take a sneak peak at using Debezium with ZooKeeper-less Kafka!

In parallel, we’re going to work on the roadmap for Debezium 1.7 (due by the end of September). Please get in touch via the mailing list if you have specific feature requests for this release!