Over the last five years, Debezium has become a leading open-source solution for change data capture for a variety of databases. Users from all kinds of industries work with Debezium for use cases like replication of data from operational databases into data warehouses, updating caches and search indexes, driving streaming queries via Kafka Streams or Apache Flink, synchronizing data between microservices, and many more.
When talking to Debezium users, we generally receive very good feedback on the range of applications enabled by Debezium and its flexibility: e.g. each connector can be configured and fine-tuned in many ways, depending on your specific requirements. A large number of metrics provide deep insight into the state of running Debezium connectors, allowing to safely operate CDC pipelines also in huge installations with thousands of connectors.
All this comes at the cost of a learning curve, though: users new to Debezium need to understand the different options and settings as well as learn about best practices for running Debezium in production. We’re therefore constantly exploring how the user experience of Debezium can be further improved, allowing people to set up and operate its connectors more easily.
Today it’s my great pleasure to introduce you to a proof-of-concept for a potential future Debezium graphical user interface. The goal for this PoC is to explore how a graphical UI could facilitate the getting started and operational experience of Debezium users.
The scope of the PoC is the set-up flow for configuring and instantiating a Debezium Postgres connector. The user is guided through the required configuration steps in a wizard interface, starting from mandatory information (e.g. database credentials), over selecting the tables to be captured, up to optional settings like different data mapping options. After reviewing the final configuration, the UI will instantiate the connector in Kafka Connect.
You can see a short demo of how this looks like in this video:
We focused on some core interaction patterns, e.g. the preview functionality for the selecting the captured tables. Instead of solely taking key/value pairs of configuration parameters, the UI should guide the user through the process, provide context and help, e.g. by showing the allowed options for settings in drop-downs, validating the provided settings after each step, and more.
Now this is just the beginning, there’s many more things that could be done in such Debezium UI, e.g. in the connection configuration step, we could validate whether the given user has all the required database permissions, whether the right WAL level is set in the database, etc. There could be views for monitoring and trouble-shooting connectors. When running on Kubernetes, the UI could produce resource definitions processed by a Kafka (Connector) operator like Strimzi (instead of calling the Kafka Connect REST API), and much more.
But before further progressing with this, we’d like to gather your feedback and opinions: Do you consider a graphical UI for Debezium useful in general, and is it something you would use in your projects? What is your feedback on the functionality currently implemented in the PoC? Which other functionality besides connector configuration would you like to see in a Debezium UI? We’ve provided a short survey with these and a few other questions:
Before taking the questionnaire, please watch the video or run the PoC yourself (see below). Answering these questions should take just a few minutes; your participation would be very helpful for us in order to decide whether and how we should move forward with this effort.
Trying It Out Yourself
As everything in Debezium, the UI PoC is fully open source (Apache License Version 2.0); you can find its source code under the Debezium organization on Git Hub. The PoC is implemented as a Quarkus-based web application, using React as the frontend technology.
The Quarkus backend is configured with the URL(s) of one more Kafka Connect clusters. Note that there’s currently no means of authentication or authorization implemented in the PoC, so don’t use it with your production Connect clusters just yet. After starting the application, you can choose the cluster to work with from the drop-down to the top right. Different from what’s shown in the video recording, the "Delete" button is working in the PoC now, too ;) There’s an example Docker Compose file, which starts up all required components for getting started quickly. Alternatively, you can obtain a pre-built container image with the Debezium UI PoC from Docker Hub. Please refer to the README file for more details on building and running the Debezium UI PoC.
We’re looking forward very much to learning about your feedback on the Debezium UI PoC. Try it out yourself and let us know about your thoughts in the comments below and by participating in the quick survey linked above.
A big thank you to the team working on this PoC: Ashique Ansari, Indra Shukla, June Zhang, Mark Drilling, Na Ding, and René Kerner!
About Debezium
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
Get involved
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.
