The Yelp Engineering Blog recently began a series describing their real-time streaming data infrastructure. The first post provides a good introduction and explains how moving from a monolith to a service-oriented architecture increased productivity, but also made it more challenging to work with data spread across the 100 services that own it. It’s totally worth your time to read it right now.

As Justin writes in the post, several reasons prompted them to create their own real time streaming data pipeline:

Yelp’s Real-Time Data Pipeline records changes to data on totally ordered distributed logs so that downstream consumers can receive and process the same changes in exactly the same order. Services can consume changes made by other services, and can therefore stay in sync without explicit interservice communication. This system uses among other things Kafka for event logs, a homegrown system named MySQLStreamer to capture committed changes to MySQL tables, Avro for message format and schemas, and a custom Schematizer service that tracks consumers and enforces the Avro schemas used for messages on every Kafka topic.

Perhaps most interesting for Debezium is how Yelp captures the committed changes in their MySQL databases and write them to Kafka topics. Their second post in the series goes into a lot more detail about their MySQLStreamer process that reads the MySQL binary log and continously processes the DDL statements and DML operations that appear in the log, generating the corresponding insert, update, delete, and refresh events, and writing these event messages to a separate Kafka topic for each MySQL table. We’ve mentioned before that MySQL’s row-level binlog events that result from the DML operation don’t include the full definition of the columns, so knowing what the columns mean in each event requires process the DDL statements that also appear in the binlog. Yelp uses a separate MySQL instance it calls the schema tracker database, which behaves like a MySQL slave to which are applied only the DDL statements they read from the binlog. This technique lets Yelp’s MySQLStreamer system know the state of the database schema and the structure of its tables at the point in the binlog where they are processing events. This is pretty interesting, because it uses the MySQL engine to handle the DDL parsing.

Yelp’s MySQLStreamer process uses another MySQL database to track internal state describing its position in the binlog, what events have been successfully published to Kafka, and, because the binlog position varies on each replica, replica-independent information about each transaction. This latter information is similar to MySQL GTIDs, although Yelp is using earlier versions of MySQL that do not support GTIDs.

Of course, special consideration has to be taken for databases that have been around for a long time. The MySQL binlogs are capped and will not contain the entire history of the databases, so Yelp’s MySQLStreamer process bootstraps the change data capture process of old databases by starting another clean MySQL replica, which will use the built-in MySQL replication mechanism with the MySQL blackhole database engine to obtain a consistent snapshot of the master and so that all activity is logged in the replica’s binlog while the replica actually stores no data.

Yelp’s MySQLStreamer mechanism is quite ingenious in its use of MySQL and multiple extra databases to capture changes from MySQL databases and write them to Kafka topics. The downside, of course, is that doing so does increase the operational complexity of the system.

Debezium is an open source project that is building a change data capture for a variety of DBMSes. Like Yelp’s MySQLStreamer, Debezium’s MySQL Connector can continously capture the committed changes to MySQL database rows and record these events in a separate Kafka topic for each table. When first started, Debezium’s MySQL Connector can perform an initial consistent snapshot and then begin reading the MySQL binlog. It uses both DDL and DML operations that appear in the binlog, directly parsing and using the DDL statements to learn the changes to each table’s structure and the mapping of each insert, update, and delete binlog event. And each resulting change event written to Kafka includes information about the originating MySQL server and its binlog position, as well as the before and/or after states of the affected row.

However, unlike Yelp’s MySQLStreamer, the Debezium MySQL connector doesn’t need or use extra MySQL databases to parse DDL or to store the connector’s state. Instead, Debezium is built on top of Kafka Connect, which is a new Kafka library that provides much of the generic functionality of reliably pulling data from external systems, pushing it into Kafka topics, and tracking what data has already been processed. Kafka Connect stores this state inside Kafka itself, simplifying the operational footprint. Debezium’s MySQL connector can then focus on the details of performing a consistent snapshot when required, reading the binlog, and converting the binlog events into useful change events.

Yelp’s real time data pipeline makes use of a custom Avro schema registry, and uses those Avro schemas to encode each event into a compact binary representation while keeping the metadata about the structure of the event. It’s possible to do this with Debezium, too: simply run Confluent’s Schema Registry as a service and then configure the Kafka Connect worker to use the Avro Converter. As the converter serializes each event, it looks at the structure defined by the connector and, when that structure changes, generates an updated Avro Schema and registers it with the Schema Registry. That new Avro schema is then used to encode the event (and others with an identical structure) into a compact binary form written to Kafka. And of course, consumers then also use the same Avro converter so that as events are deserialized, the converter coordinates with the Schema Registry whenever it needs an Avro schema it doesn’t know about. As a result, the events are stored in a compact manner while the events' content and metadata remain available, while Schema Registry captures and maintains the history of the Avro schema for each table as it evolves over time.

If you’re interested in change data capture with MySQL (or any other DBMSes), give Debezium a try by going through our tutorial that walks you through starting Kafka, Kafka Connect, and Debezium’s MySQL Connector to see exactly what change data events look like and how they can be used. Best of all, it’s open source with a growing community of developers that has had the benefit of building on top of recently-created Kafka Connect framework. Our MySQL connector is ready now, but we’re working on connectors for other DBMSes. Specifically, our upcoming 0.3 release will include our MongoDB Connector, with 0.4 including connectors for PostgreSQL and/or Oracle.

Correction: A previous version of this post incorrectly stated that Yelp was using a MySQL version that did support GTIDs, when in fact they are using a version that does not support MySQL GTIDs. The post has been corrected, and the author regrets the mistake.