1. Releases
  2. 1.1
  3. Release Notes for Debezium 1.1

Release Notes for Debezium 1.1

All notable changes for Debezium releases are documented in this file. Release numbers follow Semantic Versioning.

Release 1.1.2.Final (June 2nd, 2020)

See the complete list of issues.

Kafka compatibility

This release has been built against Kafka Connect 2.5.0 and has been tested with version 2.5.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading the MySQL, MongoDB, PostgreSQL, SQL Server, Cassandra or DB2 connectors, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.1.2.Final from any of the earlier 1.1.x, 1.0.x, 0.10.x, 0.9.x, 0.8.x, 0.7.x, 0.6.x, 0.5.x, 0.4.x, 0.3.x, 0.2.x, or 0.1.x versions, first check the upgrading notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.1.2.Final plugin files, and restart the connector using the same configuration. Upon restart, the 1.1.2.Final connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our docker images then do not forget to pull them fresh from Docker registry.

Breaking changes

Db2 and Oracle connectors were using database name which is fixed to configure table filters. This is now simplified so only <schema_name>.<table_name> naming is used. Db2, Oracle and SQL Server connectors were using database name which is fixed to configure column/key filters and mappers. This is now simplified so only <schema_name>.<table_name>.<column_name> naming is used (DBZ-1312).

Db2 connector was providing initial_schema_only snapshot mode. This name was not consistent with other connectors so the mode was renamed to schema_only (DBZ-2051).

New Features

There are no new features in this release.

Fixes

This release includes the following fixes:

  • Outbox Quarkus Extension throws NPE in quarkus:dev mode DBZ-1966

  • [Doc] Debezium User Guide should provide example of DB connector yaml and deployment instructions DBZ-2011

  • MySQL alias FLUSH TABLE not handled DBZ-2047

  • DDL statement throws error if compression keyword contains backticks (``) DBZ-2062

  • Error and connector stops when DDL contains algorithm=instant DBZ-2067

  • Unable to parse MySQL ALTER statement with named primary key DBZ-2080

Other changes

This release includes also other changes:

  • Adding tests and doc updates around column masking and truncating DBZ-775

  • Align snapshot/streaming semantics in MongoDB documentation DBZ-1901

  • Only top-level menu items shown for MySQL connector 1.1 docs DBZ-1980

  • Upgrade to Apache Kafka 2.5.0 and Confluent Platform 5.5.0 DBZ-1981

  • Fix broken link DBZ-1983

  • Avoid broken cross-book references in downstream docs DBZ-1999

  • Fix wrong attribute name in MongoDB connector DBZ-2006

  • Remove additional Jackson dependencies as of AK 2.5 DBZ-2076

  • Tar packages must use posix format DBZ-2088

  • Upgrade to Quarkus 1.5.0.Final DBZ-2119

Release 1.1.1.Final (April 17th, 2020)

See the complete list of issues.

Kafka compatibility

This release has been built against Kafka Connect 2.4.1 and has been tested with version 2.4.1 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading the MySQL, MongoDB, PostgreSQL, SQL Server, Cassandra or DB2 connectors, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.1.1.Final from any of the earlier 1.1.x, 1.0.x, 0.10.x, 0.9.x, 0.8.x, 0.7.x, 0.6.x, 0.5.x, 0.4.x, 0.3.x, 0.2.x, or 0.1.x versions, first check the upgrading notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.1.1.Final plugin files, and restart the connector using the same configuration. Upon restart, the 1.1.1.Final connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our docker images then do not forget to pull them fresh from Docker registry.

Breaking changes

There are no breaking changes in this release.

New Features

  • Improve documentation on WAL disk space usage for Postgres connector DBZ-1732

  • Add docs for mask column and truncate column features DBZ-1954

  • Snapshot metrics shows TotalNumberOfEventsSeen as zero DBZ-1932

Fixes

This release includes the following fixes:

  • Snapshot lock timeout setting is not documented DBZ-1914

  • AvroRuntimeException when publishing transaction metadata DBZ-1915

  • Connector restart logic throttles for the first 2 seconds DBZ-1918

  • Wal2json empty change event could cause NPE above version 1.0.3.final DBZ-1922

  • Broken internal links and anchors in documentation DBZ-1935

  • Dokumentation files in modules create separate pages, should be partials instead DBZ-1944

  • Validation of binlog_row_image is not compatible with MySQL 5.5 DBZ-1950

  • High CPU usage when idle DBZ-1960

Other changes

This release includes also other changes:

  • Fix typo in Quarkus Outbox extension documentation DBZ-1902

  • Documentation should link to Apache Kafka upstream docs DBZ-1906

  • Restore documentation of MySQL event structures DBZ-1919

  • Update snapshot.mode options in SQL Server documentation DBZ-1924

  • Remove obsolete metrics from downstream docs DBZ-1947

Release 1.1.0.Final (March 23rd, 2020)

See the complete list of issues.

Kafka compatibility

This release has been built against Kafka Connect 2.4.0 and has been tested with version 2.4.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading the MySQL, MongoDB, PostgreSQL, SQL Server, Cassandra or DB2 connectors, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.1.0.Final from any of the earlier 1.1.x, 1.0.x, 0.10.x, 0.9.x, 0.8.x, 0.7.x, 0.6.x, 0.5.x, 0.4.x, 0.3.x, 0.2.x, or 0.1.x versions, first check the upgrading notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.1.0.Final plugin files, and restart the connector using the same configuration. Upon restart, the 1.1.0.Final connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our docker images then do not forget to pull them fresh from Docker registry.

Breaking changes

There are no breaking changes in this release.

New Features

  • The Postgres connector heartbeat should optionally write back a heartbeat change to the DB DBZ-1815

Fixes

This release includes the following fixes:

  • Postgres Connector ignoring confirmed_flush_lsn and skipping ahead to latest txn DBZ-1730

  • Postgresql money error handling DBZ-1755

  • MongoDB tests not working correctly DBZ-1867

  • MongoDB transaction metadata topic generates extra events DBZ-1874

  • NullPointerException on delete in ExtractNewRecordState class DBZ-1876

  • MongoDB connector unrecoverable exception DBZ-1880

  • High log volume from: "Awaiting end of restart backoff period" logs DBZ-1889

  • Kafka records from one Cassandra table get published to the kafka queue of another Cassandra table DBZ-1892

Other changes

This release includes also other changes:

  • Use snapshot versions in master branch documentation DBZ-1793

  • Misc docs issues DBZ-1798

  • Outbox Quarkus Extension: Clarify default column types when using defaults. DBZ-1804

  • Create CI job to run OpenShift test DBZ-1817

  • Failing test jobs for Mongo and SQL Server due to insecure maven registry DBZ-1837

  • Support retriable exceptions with embedded engine DBZ-1857

  • Modularize Debezium logging doc DBZ-1861

  • Centralize closing of coordinator DBZ-1863

  • Assert format of commit messages DBZ-1868

  • Bump MongoDB java driver to the latest version 3.12.2 DBZ-1869

  • Add Travis CI task for MongoDB 3.2 DBZ-1871

  • Unstable tests for PostgreSQL DBZ-1875

  • Add MongoDB JMX integration tests DBZ-1879

Release 1.1.0.CR1 (March 11th, 2020)

See the complete list of issues.

Kafka compatibility

This release has been built against Kafka Connect 2.4.0 and has been tested with version 2.4.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading the MySQL, MongoDB, PostgreSQL, SQL Server, Cassandra or DB2 connectors, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.1.0.CR1 from any of the earlier 1.1.x, 1.0.x, 0.10.x, 0.9.x, 0.8.x, 0.7.x, 0.6.x, 0.5.x, 0.4.x, 0.3.x, 0.2.x, or 0.1.x versions, first check the upgrading notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.1.0.CR1 plugin files, and restart the connector using the same configuration. Upon restart, the 1.1.0.CR1 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our docker images then do not forget to pull them fresh from Docker registry.

Breaking changes

The default value of MySQL config option gtid.new.channel.position was originally set to latest that should never be used in a production. The default value was thus set to earliest and the config option is scheduled for removal (DBZ-1705). The MySQL config option event.deserialization.failure.handling.mode was renamed to event.processing.failure.handling.mode to make the naming consistent with other connectors (DBZ-1826). MongoDB config option field.renames adds renamed field (wtih null value) even when the source field is missing. This was identified as a defect and the non-present fields are no longer added (DBZ-1848).

New Features

  • Distinguish between public (API) and internal parts of Debezium DBZ-234

  • Default gtid.new.channel.position to earliest DBZ-1705

  • Add option to skip unprocesseable event DBZ-1760

  • ExtractNewRecordState - add.source.fields should strip spaces from comma-separated list of fields DBZ-1772

  • Add support for update events for sharded MongoDB collections DBZ-1781

  • Useless/meaningless parameter in function DBZ-1805

  • Replace BlockEventQueue with Debezium ChangeEventQueue DBZ-1820

  • Option to configure column.propagate.source.type on a per-type basis, not per column-name basis DBZ-1830

  • Support MongoDB Oplog operations as config DBZ-1831

  • Add app metrics for mongodb connector to jmx DBZ-845

  • Provide SPI to override schema and value conversion for specific columns DBZ-1134

  • Retry polling on configured exceptions DBZ-1723

Fixes

This release includes the following fixes:

  • CDC Event Schema Doesn’t Change After 2 Fields Switch Names and Places DBZ-1694

  • TINYINT(1) value range restricted on snapshot. DBZ-1773

  • MySQL source connector fails while parsing new AWS RDS internal event DBZ-1775

  • Connector fails when performing a Hot Schema Update in SQLServer (Data row is smaller than a column index). DBZ-1778

  • Incosistency in MySQL TINYINT mapping definition DBZ-1800

  • Debezium skips messages after restart DBZ-1824

  • Supply of message.key.columns disables primary keys. DBZ-1825

  • MySql connector fails after CREATE TABLE IF NOT EXISTS table_A, given table_A does exist already DBZ-1833

  • Unable to listen to binlogs for tables with a period in the table names DBZ-1834

  • Mongodb field.renames will add renamed field even when source field is missing DBZ-1848

  • Redundant calls to refresh schema when using user defined types in PostgreSQL DBZ-1849

  • postgres oid is too large to cast to integer DBZ-1850

Other changes

This release includes also other changes:

  • Verify correctness of JMX metrics DBZ-1664

  • Document that server name option must not use hyphen in name DBZ-1704

  • Move MongoDB connector to base framework DBZ-1726

  • hstore.handling.mode docs seem inaccurate (and map shows null values) DBZ-1758

  • Document transaction metadata topic name DBZ-1779

  • Remove Microsoft references in Db2 connector comments DBZ-1794

  • Fix link to CONTRIBUTE.md in debezium-incubator repository README.md DBZ-1795

  • Invalid dependency definition in Quarkus ITs DBZ-1799

  • Document MySQL boolean handling DBZ-1801

  • Jackson dependency shouldn’t be optional in Testcontainers module DBZ-1803

  • Change Db2 configuration for faster test execution DBZ-1809

  • MySQL: Rename event.deserialization.failure.handling.mode to event.processing.failure.handling.mode DBZ-1826

  • Misleading warning message about uncommitted offsets DBZ-1840

  • Missing info on DB2 connector in incubator README file DBZ-1842

  • Only replace log levels if LOG_LEVEL var is set DBZ-1843

  • Modularize tutorial DBZ-1845

  • Modularize the monitoring doc DBZ-1851

  • Remove deprecated methods from SnapshotProgressListener DBZ-1856

  • Document PostgreSQL connector metrics DBZ-1858

Release 1.1.0.Beta2 (February 13th, 2020)

See the complete list of issues.

Kafka compatibility

This release has been built against Kafka Connect 2.4.0 and has been tested with version 2.4.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading the MySQL, MongoDB, PostgreSQL or SQL Server connectors, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.1.0.Beta2 from any of the earlier 1.1.x, 1.0.x, 0.10.x, 0.9.x, 0.8.x, 0.7.x, 0.6.x, 0.5.x, 0.4.x, 0.3.x, 0.2.x, or 0.1.x versions, first check the upgrading notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.1.0.Beta2 plugin files, and restart the connector using the same configuration. Upon restart, the 1.1.0.Beta2 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our docker images then do not forget to pull them fresh from Docker registry.

Breaking changes

There are no breaking changes in this release.

New Features

  • Add ability to insert fields from op field in ExtractNewRecordState SMT DBZ-1452

  • Integrates with TestContainers project DBZ-1722

Fixes

This release includes the following fixes:

  • Postgres Connector losing data on restart due to commit() being called before events produced to Kafka DBZ-1766

  • DBREF fields causes SchemaParseException using New Record State Extraction SMT and Avro converter DBZ-1767

Other changes

This release includes also other changes:

  • Superfluous whitespace in intra-level ToC sidebar DBZ-1668

  • Outbox Quarkus Extension follow-up tasks DBZ-1711

  • DB2 connector follow-up tasks DBZ-1752

  • Unwrap SMT demo not compatible with ES 6.1+ DBZ-1756

  • Instable SQL Server test DBZ-1764

  • Remove Db2 JDBC driver from assembly package DBZ-1776

  • Fix PostgresConnectorIT.shouldOutputRecordsInCloudEventsFormat test DBZ-1783

  • Use "application/avro" as data content type in CloudEvents DBZ-1784

  • Update Standard Tutorials/Examples with DB2 DBZ-1558

Release 1.1.0.Beta1 (February 5th, 2020)

See the complete list of issues.

Kafka compatibility

This release has been built against Kafka Connect 2.4.0 and has been tested with version 2.4.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading the MySQL, MongoDB, PostgreSQL or SQL Server connectors, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.1.0.Beta1 from any of the earlier 1.1.x, 1.0.x, 0.10.x, 0.9.x, 0.8.x, 0.7.x, 0.6.x, 0.5.x, 0.4.x, 0.3.x, 0.2.x, or 0.1.x versions, first check the upgrading notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.1.0.Beta1 plugin files, and restart the connector using the same configuration. Upon restart, the 1.1.0.Beta1 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our docker images then do not forget to pull them fresh from Docker registry.

Breaking changes

Before updating the DecoderBufs logical decoding plug-in in your Postgres database to this new version (or when pulling the debezium/postgres container image for that new version), it is neccessary to upgrade the Debezium Postgres connector to 1.0.1.Final or 1.1.0.Alpha2 or later (DBZ-1052).

The ExtractNewDocumentState SMT to be used with the Debezium MongoDB connector will convert Date and Timestamp fields now into the org.apache.kafka.connect.data.Timestam`p logical type, clarifying its semantics. The schema type itself remains unchanged as `int64. Please note that the resolution of Timestamp is seconds as per the semantics of that type in MongoDB. (DBZ-1717).

New Features

  • Create a plug-in for DB2 streaming DBZ-695

  • Add topic routing by field option for New Record State Extraction DBZ-1715

  • Generate date(time) field types in the Kafka Connect data structure DBZ-1717

  • Publish TX boundary markers on a TX metadata topic DBZ-1052

  • Replace connectorName with kafkaTopicPrefix in kafka key/value schema DBZ-1763

Fixes

This release includes the following fixes:

  • Connector error after adding a new not null column to table in Postgres DBZ-1698

  • MySQL connector doesn’t use default value of connector.port DBZ-1712

  • Fix broken images in Antora and brush up AsciiDoc DBZ-1725

  • ANTLR parser cannot parse MariaDB Table DDL with TRANSACTIONAL attribute DBZ-1733

  • Postgres connector does not support proxied connections DBZ-1738

  • GET DIAGNOSTICS statement not parseable DBZ-1740

  • Examples use http access to Maven repos which is no longer available DBZ-1741

  • MySql password logged out in debug log level DBZ-1748

  • Cannot shutdown PostgreSQL if there is an active Debezium connector DBZ-1727

Other changes

This release includes also other changes:

  • Add tests for using fallback values with default REPLICA IDENTITY DBZ-1158

  • Migrate all attribute name/value pairs to Antora component descriptors DBZ-1687

  • Upgrade to Awestruct 0.6.0 DBZ-1719

  • Run CI tests for delivered non-connector modules (like Quarkus) DBZ-1724

  • Remove overlap of different documentation config files DBZ-1729

  • Don’t fail upon receiving unkown operation events DBZ-1747

  • Provide a method to identify an envelope schema DBZ-1751

  • Upgrade to Mongo Java Driver version 3.12.1 DBZ-1761

  • Create initial Proposal for DB2 Source Connector DBZ-1509

  • Review Pull Request for DB2 Connector DBZ-1527

  • Test Set up of the DB2 Test Instance DBZ-1556

  • Create Documentation for the DB2 Connector DBZ-1557

  • Verify support of all DB2 types DBZ-1708

Release 1.1.0.Alpha1 (January 16th, 2020)

See the complete list of issues.

Kafka compatibility

This release has been built against Kafka Connect 2.4.0 and has been tested with version 2.4.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading the MySQL, MongoDB, PostgreSQL or SQL Server connectors, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.1.0.Alpha1 from any of the earlier 1.1.x, 1.0.x, 0.10.x, 0.9.x, 0.8.x, 0.7.x, 0.6.x, 0.5.x, 0.4.x, 0.3.x, 0.2.x, or 0.1.x versions, first check the upgrading notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.1.0.Alpha1 plugin files, and restart the connector using the same configuration. Upon restart, the 1.1.0.Alpha1 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our docker images then do not forget to pull them fresh from Docker registry.

Breaking changes

When using the outbox event routing SMT and configuring a column from which to obtain the Kafka record timestamp from (table.field.event.timestamp option), then that value could have been exported as milliseconds, microseconds or nanoseconds, based on the source column’s definition. As of this release, the timestamp always be exported as milliseconds (DBZ-1707).

The deprecated Postgres connector option slot.drop_on_stop has been removed; use slot.drop.on.stop instead (DBZ-1600).

New Features

  • MongoDB authentication against non-admin authsource DBZ-1168

  • Oracle: Add support for different representations of "NUMBER" Data Type DBZ-1552

  • Update Mongo Java driver to version 3.12.0 DBZ-1690

  • Support exporting change events in "CloudEvents" format DBZ-1292

  • Build Quarkus extension facilitating implementations of the outbox pattern DBZ-1478

  • Support column masking option for Postgres DBZ-1685

Fixes

This release includes the following fixes:

  • Make slot creation in PostgreSQL more resilient DBZ-1684

  • SQLserver type time(4)…​time(7) lost nanoseconds DBZ-1688

  • Support boolean as default for INT(1) column in MySQL DBZ-1689

  • SIGNAL statement is not recognized by DDL parser DBZ-1691

  • When using in embedded mode MYSQL connector fails DBZ-1693

  • MySQL connector fails to parse trigger DDL DBZ-1699

Other changes

This release includes also other changes:

  • Update outbox routing example DBZ-1673

  • Add option to JSON change event SerDe for ignoring unknown properties DBZ-1703

  • Update debezium/awestruct image to use Antora 2.3 alpha 2 DBZ-1713