Kafka Connector Integration with Mule 4

Introduction:

Kafka is a scalable pub/sub system, primarily used to collect & analyze large volumes of data. It enables us to pass messages from one end-point to another. Kafka messages are persisted on the disk and replicated among the cluster to prevent data loss. Kafka is constructed on top of the ZooKeeper synchronization service.

Benefits:

1. Reliability
2. Scalability
3. Durability
4. Performance

Use Case:

Send and Receive message from Kafka messaging system by integrating with the Mulesoft platform.

Please follow the step by step installation procedure of Kafka Installation.

Mule Flow (Publish Message):

Step -1: Configure the HTTP Listener by giving it a hostname, port number, and path. Along with this, specify allowed methods (Optional) in the Advanced tab of the HTTP connector.

Step-2: Drag & Drop the Logger component to log the message that was consumed by Kafka.

Step-3: Kafka comes as a custom connector needed to get it from Exchange. So, here Drag & Drop the Kafka Publish Message from Exchange.

From above:

Topic : Message to send.

Key : Belongs to message that is going to be sent.

Mule Flow (Consume Message):

Step-4: Drag & Drop the Kafka Message Consumer connector as the source from the palette.

From above:

Topic : Message to receive.

Group id : Unique string that identifies consumer group that this consumer belongs to.

Step-5: Drag & Drop the Logger component to log the consumed message.

Final Config.xml:

<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:kafka="http://www.mulesoft.org/schema/mule/kafka" xmlns:http="http://www.mulesoft.org/schema/mule/http"
	xmlns="http://www.mulesoft.org/schema/mule/core"
	xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/kafka http://www.mulesoft.org/schema/mule/kafka/current/mule-kafka.xsd">
	<kafka:kafka-producer-config name="Apache_Kafka_Producer_configuration" doc:name="Apache Kafka Producer configuration" doc:id="0289a213-cdf8-49ab-a13b-59af6cb25f97" >
		<kafka:basic-kafka-producer-connection bootstrapServers="localhost:9092" />
	</kafka:kafka-producer-config>
	<http:listener-config name="HTTP_Listener_config" doc:name="HTTP Listener config" doc:id="7700a54a-38e2-4385-a195-6a539d201b37" >
		<http:listener-connection host="0.0.0.0" port="8082" />
	</http:listener-config>
	<kafka:kafka-consumer-config name="Apache_Kafka_Consumer_configuration" doc:name="Apache Kafka Consumer configuration" doc:id="6e24bf87-b86e-4319-8ec2-624f68b245b5" >
		<kafka:basic-kafka-consumer-connection groupId="1" bootstrapServers="localhost:9092" />
	</kafka:kafka-consumer-config>
	<flow name="Kafka-Publish" doc:id="47de8881-580b-41d2-af4f-4f505480d9ec" >
		<http:listener doc:name="Listener" doc:id="173de33a-4b7d-4fd1-951c-a153dde6b52a" config-ref="HTTP_Listener_config" path="/kafka" allowedMethods="POST"/>
		<logger level="INFO" doc:name="Logger" doc:id="c85d216f-b005-4663-9309-28864a818da6" message="#[message.payload]"/>
		<kafka:producer doc:name="Publish Message" doc:id="e5457b47-7dcd-42bc-b83d-d6922dec0641" config-ref="Apache_Kafka_Producer_configuration" topic="muleesb" key="#[now()]">
			<kafka:message ><![CDATA[#[message.payload]]]></kafka:message>
		</kafka:producer>
	</flow>
	<flow name="Kafka-Consumer" doc:id="c0e6fd8a-0bcd-4ccb-91b6-0e97804d5607" >
		<kafka:consumer doc:name="Message Consumer" doc:id="660a5e95-a897-4cdb-a971-10df13d9d4be" topic="muleesb" config-ref="Apache_Kafka_Consumer_configuration"/>
		<logger level="INFO" doc:name="Logger" doc:id="86114247-dce0-4626-8278-81450bc928b6" message="#[payload]"/>
	</flow>
</mule>

Sample Input & Output:

H@ppy Le@rninG…@@

Conclusion

We have successfully integrated a Kafka connector to send an automatic email notification to the customers in one of the Order Management System POC. For additional details, please submit your questions in the below form. We are happy to assist you.