February 18, 2021
An Overview About the Different Kafka Connect Plugins
Kafka Connect is a great framework for connecting Kafka with external systems. In the best case you can use Connect right away. But in some special cases you might have to write your own plugins to add missing functionality to the framework. In this blog post I give a short overview about the different plugin types which can be used to add new functionality to Connect.
In the last part of this blog post I give you a short introduction to my GitHub project Ready, Steady, Connect - A Kafka Connect Quickstart. This project contains example Java code you can use to extend Connect with your own plugins.
The name of the project came from the blog post (Ready, Steady, Connect. Help Your Organization to Appreciate Kafka) I wrote about the experience we had with Connect.
There are two main plugin categories that can be used to add new functionality to Kafka Connect:
- Connect Plugins are part of the Connect API and can be used to extend the functionality of Connect.
- Kafka Client Plugins are part of the Kafka Client (Consumer / Producer API). Kafka Connect is build on top of the Kafka Consumer / Producer API, so we have the possibility to write plugins which are part of these “lower API’s” and use them with Connect.
Connect Plugins (Connect API)
|Sink Connector||A ||SinkConnector, SinkTask|
|Source Connector||A ||SourceConnector, SourceTask|
|Single Message Transforms (SMTs)||With a ||Transformation|
|Predicates||A SMT can be configured with a ||Predicate|
|Config Providers||A ||ConfigProvider|
|Rest Extensions||With a ||ConnectRestExtension|
Kafka Client Plugins (Kafka Producer / Consumer API)
|Kafka Consumer Interceptor||The ||ConsumerInterceptor|
|Kafka Producer Interceptor||The ||ProducerInterceptor|
|Kafka Metrics Reporter||The ||MetricsReporter|
Create Your Own Connect Plugins
The Docker image and the source code for all plugin examples can be found in the Ready, Steady, Connect - A Kafka Connect Quickstart (rueedlinger/kafka-connect-quickstart) repository.
The first step is to clone the project.
git clone https://github.com/rueedlinger/kafka-connect-quickstart
The main components of the project are the Docker Image, Java source code and the Docker Compose file.
- The custom Connect Docker image has two parts.
- The builder part to build the example plugins from the Java source code.
- The main part to run the Kafka Connect container with all the Kafka Connect plugins.
- The Java source code contains all the Kafka Connect plugin examples (connectors, transforms, etc.).
- The Docker Compose file can be used to run the whole infrastructure (Kafka broker, zookeeper, etc).
The next step is to build all the plugins (Java) and start the containers with Docker.
docker-compose up --build
When all containers are started you can access the following services:
- Kafka Connect Rest API => http://localhost:8083/
- Kafdrop from Obsidian Dynamics (GitHub) => http://localhost:8082/
- Schema Registry from Confluent (GitHub) => http://localhost:8081/
- Kafka UI from Provectus (GitHub) => http://localhost:8080/
- Kafka Connect UI from Lenses.io (GitHub) => http://localhost:8000/
Now that everything is up and running. You can start to play around with Kafka Connect. I hope the kafka-connect-quickstart project is useful and gives you an easy start into the world of Kafka Connect plugins.