Oracle Service Bus: A quickstart for the Kafka transport


As mentioned on the following blog post by Lucas Jellema, Kafka is going to play a part in several Oracle products. For some usecases it might eventually even replace JMS. In order to allow for easy integration with Kafka, you can use Oracle Service Bus to create a virtualization layer around Kafka. Ricardo Ferreira from Oracle’s A-Team has done some great work on making a custom Kafka Service Bus transport available to us. Read more about this here, here  and here. The Kafka transport is not an ‘officially supported’ transport. Quote from the A-team blog: ‘The Kafka transport is provided for free to use “AS-IS” but without any official support from Oracle. The A-Team reserves the right of help in the best-effort capacity.’. I hope it will become an officially supported part of the Service Bus product in the future.

In this blog I summarize what I have done to get the end to end sample working for SOA Suite and Kafka based on the blogs I mentioned. This allows you to quickly start developing against Apache Kafka.

Setting up Apache Kafka

  • Setting up Apache Kafka for development is easy. You follow the quickstart on: To summarize the quickstart:
  • Download Apache Kafka:
  • Unzip it: tar -xzf kafka_2.11-
  • Go to the Kafka directory: cd kafka_2.11-
  • Start ZooKeeper: bin/ config/
  • Start a new console
  • Start the Kafka broker: bin/ config/
  • Create a topic: bin/ –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic test

Setting up the Kafka transport in OSB

Copy the following files:

  • $KAFKA_HOME/libs/slf4j-api-1.7.21.jar
  • $KAFKA_HOME/libs/kafka-clients-


In my case this was /home/oracle/.jdeveloper/system12. I’m using the JDeveloper IntegratedWebLogicServer

Download the Kafka transport from here:

Extract the zip file.
Copy kafka-transport.ear and kafka-transport.jar to $MW_HOME/osb/lib/transports.

Start the domain

Execute from the kafka-transport zipfile. Use in my case from: /home/oracle/Oracle/Middleware12212/Oracle_Home/oracle_common/common/bin/

Provide the required information. It will ask for Url, username, password of your WebLogic server and deploy the kafka-transport.jar and kafka-transport.ear to the specified server (AdminServer + cluster targets). If the deployments are already there, they are first undeployed by the script.

Stop the domain

The below part I got from the following blog. This is required to be able to configure the Kafka transport from the webinterface.

Locate the following file: $MW_HOME/osb/lib/osbconsoleEar/webapp/WEB-INF/lib/adflib_osb_folder.jar.

Extract this JAR and edit /oracle/soa/osb/console/folder/l10n/

Add the following entries: Kafka transport allows you to create proxy and business services that communicate with Apache Kafka brokers. Kafka transport allows you to create proxy services that receive messages from Apache Kafka brokers. Kafka transport allows you to create business services that route messages to Apache Kafka brokers.

ZIP up the result as a new adflib_osb_folder.jar

Check the Service Bus console

After the above steps are completed, you can start the domain and use the Kafka transport from the servicebus console.




capture04 capture06


Setting up JDeveloper

Copy the JDeveloper plugin descriptor (transport-kafka.xml) to the plugins folder:
$MW_HOME/osb/config/plugins. In my case this is: /home/oracle/Oracle/Middleware12212/Oracle_Home/osb/config/plugins/. The Kafka transport, since it is a custom transport, is not visible in the regular palette. You can however do File, New, Proxy or Business service to use the Kafka transport.


Also you will not see possible options for consumer or producer settings but you can use the settings from: here and here

Running an end to end sample

Apache Kafka provides shell scripts to test producing and consuming messages:
– Producing: bin/ –broker-list localhost:9092 –topic test
– Consuming: bin/ –bootstrap-server localhost:9092 –topic test –from-beginning

It helps to add a report, log or alert action to your Service Bus pipeline so you can see messages which have passed. As a report key I have used the Kafka offset from $inbound: ./ctx:transport/ctx:request/tp:headers/kafka:offset



And now?

As you can see, several steps need to be performed to install this custom transport. It is only supported on a best-effort basis by the A-Team. I could not see options for properties in the Service Bus Console as was shown in the blog posts mentioned at the start of this post, but that is not a real issue since if a fixed set would be provided and more options would become available in a new version of Kafka, this might become limiting. It is a shame custom transports are not visible in the component palette in JDeveloper. Once you know however you can use the Kafka transport by creating Proxy and Business services from File, New this also becomes a non-issue.

There are of course other solutions to take care of the integration with Kafka such as using Kafka connectors or create a custom service to wrap Kafka, but I like the way this custom transport allows you to integrate it with Service Bus. This allows you to make Kafka only available through this channel. This offers options like easily applying policies, monitoring, alerting, etc. I do expect in Oracle’s Cloud offering interaction with Kafka products running in the Oracle Cloud such as the Event Hub, will of course be much easier. We’re looking forward to it.


About Author

Maarten is an Integration Consultant and Oracle ACE. Over the past years he has worked for numerous customers in the Netherlands in developer, analyst and architect roles on topics like software delivery, performance, security and other integration related challenges. Maarten is passionate about his job and likes to share his knowledge through publications, frequent blogging and presentations.


  1. Hi Marteen,

    Thanks for posting such a nice post.I am trying to install kafka transport in OSB
    Followed all the above steps correctly and able to see the kafka transport both under deployment section and in osb console.But while creating any service based on kafka transport i am getting the error as:-

    “ No transport provider registered with ID: kafka” while activating the session in osb console.
    Is the above error related to the “transport-kafka.xml” file you have provided in the kafka transport zip file.Please suggest.

    Note:I am running all the above process in windows OS.

    Thanks in Advance!!

  2. Thulasi Yemmela on

    Hi Maarten,

    Nice post. This worked for me like a charm in Linux environment.
    Now I am trying to test this on Windows environment and unable to get the “kafka” listed under transports in both locations, JDeveloper & SBConsole. when I did the, installation completed without any issues and I can see them under deployments. Are you aware of any success or issues in getting this worked on Windows.


Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.