Debezium Logging

Each log message produced by the application will be sent to a specific logger. Every logger has a name such as io.debezium.connector.mysql, and the names form a hierarchy of loggers. For example, the logger named io.debezium.connector.mysql has a parent logger named io.debezium.connector, which has a parent logger named io.debezium, all the way to the root logger at the very top of this hierarchy.

Every log message produced by the application will also have a specific log level:

Log4J also allows you to define one or more appenders, which are essentially destinations where log messages will be written. Each appender controls the format of its log messages, giving you even more control over what the log messages look like.

To configure logging, you specify the desired level for each logger and the appender(s) where those log messages should be written. Since loggers are hierarchical, the configuration for the root logger serves as a default for all of the loggers below it, although you can override any child (or descendant) logger.

If you’re running Debezium connectors in a Kafka Connect process, then Kafka Connect will use the Log4J configuration file (e.g., config/connect-log4j.properties) in the Kafka installation. The following are snippets from this file:

Unless you configure other loggers, all of the loggers used by Debezium will inherit the rootLogger configuration.

For the most part, the Debezium code sends its log messages to loggers with names that match the fully-qualified name of the Java class that is generating the log message. This normally works well, since we use packages to organize code with similar or related functions.

This means that you can easily control all of the log messages for a specific class or for all of the classes within or under a specific package. For example, to turn on debug logging for the entire MySQL connector (and the database history implementation used by the connector) might be as simple as adding the following line(s) to your log4j.properties file:

Configuring logging is a tradeoff: provide too little and it’s not clear what is going on; provide too much and it’s too difficult to understand what’s going on. Our goal is that INFO provides enough to see that the connector is healthy (though logging is not a substitute for monitoring), and that DEBUG is provides more detailed information to help you figure out what is going on. But sometimes DEBUG is not enough, and that’s where TRACE comes in. The TRACE level is literally every log message produced by or under that named logger, and it can be voluminous.

You can also configure logging for a specific subset of the classes within the connector, simply by adding more lines like those above except for more specific logger names. For example, maybe you’re not sure why the MySQL connector is skipping some events when it is processing the binlog. Rather than turn on DEBUG or TRACE logging for whole connector, you can set the connector’s logging to INFO and then configure DEBUG or TRACE on just the class that’s reading the binlog. For example:

This requires a bit more knowledge about the code, but here’s a useful tip. First turn on DEBUG or TRACE logging for the whole connector (or even the io.debezium package), find the messages that are most useful, and look at the end of the log message to see the name of the Java class that produced the message. Then, turn the connector’s logging back to INFO and add a logger configuration for each of those "interesting" classes or packages. If there are a few classes you don’t want to see as detailed log messages, add another logger configuration for those classes and set them to INFO.

We’ve only just touched on the power of Log4J, so if you want even more control look for some tutorials about how to set up and use other appenders to send different log messages to specific destinations.

Most Debezium connectors use multiple threads to perform different activities, and the Kafka Connect workers also use multiple threads. This can make it challenging to look at a log file and find only those log messages for one of these logical activities. To make this easier, Debezium uses mapped diagnostic contexts, or MDC to provide additional information for each thread via several properties that you can embed in your appender patterns:

You can use these properties within the appender’s pattern defined in the log4j.properties file. For example, the following is a modification of the stdout appender’s layout to use these MDC properties:

This will produce messages in the log similar to these:

Notice how each line includes the connector type (e.g., MySQL), the name of the connector (e.g., dbserver1), and the activity of the thread (e.g., snapshot). And here you can see at the end of the line the name of the class that generated the message.

The Debezium Docker images for Zookeeper, Kafka, and Kafka Connect all set up their log4j.properties file to configure the Debezium-related loggers and to ensure all log messages go to the Docker containers console (and thus the Docker logs) and are written to files under the /kafka/logs directory, which you can mount to easily get access to those files.

The containers use a LOG_LEVEL environment variable to set the log level for the root logger. So, when starting a container simply set this environment variable to one of the log levels (e.g., -e LOG_LEVEL=DEBUG), and all of the code within the container will start using that log level.

If you need more control, create a new image that is based on ours, except in your Dockerfile copy your own log4j.properties file into the image: