Oracle Service Bus: A quickstart for the Kafka transport

7

As mentioned on the following blog post by Lucas Jellema, Kafka is going to play a part in several Oracle products. For some usecases it might eventually even replace JMS. In order to allow for easy integration with Kafka, you can use Oracle Service Bus to create a virtualization layer around Kafka. Ricardo Ferreira from Oracle’s A-Team has done some great work on making a custom Kafka Service Bus transport available to us. Read more about this here, here and here. The Kafka transport is not an ‘officially supported’ transport. Quote from the A-team blog: ‘The Kafka transport is provided for free to use “AS-IS” but without any official support from Oracle. The A-Team reserves the right of help in the best-effort capacity.’. I hope it will become an officially supported part of the Service Bus product in the future.

In this blog I summarize what I have done to get the end to end sample working for SOA Suite 12.2.1.2.0 and Kafka 0.10.1.0 based on the blogs I mentioned. This allows you to quickly start developing against Apache Kafka.

Setting up Apache Kafka

  • Setting up Apache Kafka for development is easy. You follow the quickstart on: https://kafka.apache.org/quickstart. To summarize the quickstart:
  • Download Apache Kafka: https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.1.0/kafka_2.11-0.10.1.0.tgz
  • Unzip it: tar -xzf kafka_2.11-0.10.1.0.tgz
  • Go to the Kafka directory: cd kafka_2.11-0.10.1.0
  • Start ZooKeeper: bin/zookeeper-server-start.sh config/zookeeper.properties
  • Start a new console
  • Start the Kafka broker: bin/kafka-server-start.sh config/server.properties
  • Create a topic: bin/kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic test

Setting up the Kafka transport in OSB

Copy the following files:

  • $KAFKA_HOME/libs/slf4j-api-1.7.21.jar
  • $KAFKA_HOME/libs/kafka-clients-0.10.1.0.jar

To $OSB_DOMAIN/lib.

In my case this was /home/oracle/.jdeveloper/system12.2.1.2.42.161008.1648/DefaultDomain/lib. I’m using the JDeveloper IntegratedWebLogicServer

Download the Kafka transport from here: http://www.ateam-oracle.com/wp-content/uploads/2016/10/kafka-transport-0.4.1.zip

Extract the zip file.
Copy kafka-transport.ear and kafka-transport.jar to $MW_HOME/osb/lib/transports.

Start the domain

Execute install.py from the kafka-transport zipfile. Use wlst.sh in my case from: /home/oracle/Oracle/Middleware12212/Oracle_Home/oracle_common/common/bin/wlst.sh

Provide the required information. It will ask for Url, username, password of your WebLogic server and deploy the kafka-transport.jar and kafka-transport.ear to the specified server (AdminServer + cluster targets). If the deployments are already there, they are first undeployed by the script.

Stop the domain

The below part I got from the following blog. This is required to be able to configure the Kafka transport from the webinterface.

Locate the following file: $MW_HOME/osb/lib/osbconsoleEar/webapp/WEB-INF/lib/adflib_osb_folder.jar.

Extract this JAR and edit /oracle/soa/osb/console/folder/l10n/FolderBundle.properties.

Add the following entries:

desc.res.gallery.kafka=The Kafka transport allows you to create proxy and business services that communicate with Apache Kafka brokers.
desc.res.gallery.kafka.proxy=The Kafka transport allows you to create proxy services that receive messages from Apache Kafka brokers.
desc.res.gallery.kafka.business=The Kafka transport allows you to create business services that route messages to Apache Kafka brokers.

ZIP up the result as a new adflib_osb_folder.jar

Check the Service Bus console

After the above steps are completed, you can start the domain and use the Kafka transport from the servicebus console.

capture01

capture02

capture03

capture04 capture06

capture05

Setting up JDeveloper

Copy the JDeveloper plugin descriptor (transport-kafka.xml) to the plugins folder:
$MW_HOME/osb/config/plugins. In my case this is: /home/oracle/Oracle/Middleware12212/Oracle_Home/osb/config/plugins/. The Kafka transport, since it is a custom transport, is not visible in the regular palette. You can however do File, New, Proxy or Business service to use the Kafka transport.

kafka-in-jdev

Also you will not see possible options for consumer or producer settings but you can use the settings from: here and here

Running an end to end sample

Apache Kafka provides shell scripts to test producing and consuming messages:
– Producing: bin/kafka-console-producer.sh –broker-list localhost:9092 –topic test
– Consuming: bin/kafka-console-consumer.sh –bootstrap-server localhost:9092 –topic test –from-beginning

It helps to add a report, log or alert action to your Service Bus pipeline so you can see messages which have passed. As a report key I have used the Kafka offset from $inbound: ./ctx:transport/ctx:request/tp:headers/kafka:offset

capture10

capture11

And now?

As you can see, several steps need to be performed to install this custom transport. It is only supported on a best-effort basis by the A-Team. I could not see options for properties in the Service Bus Console as was shown in the blog posts mentioned at the start of this post, but that is not a real issue since if a fixed set would be provided and more options would become available in a new version of Kafka, this might become limiting. It is a shame custom transports are not visible in the component palette in JDeveloper. Once you know however you can use the Kafka transport by creating Proxy and Business services from File, New this also becomes a non-issue.

There are of course other solutions to take care of the integration with Kafka such as using Kafka connectors or create a custom service to wrap Kafka, but I like the way this custom transport allows you to integrate it with Service Bus. This allows you to make Kafka only available through this channel. This offers options like easily applying policies, monitoring, alerting, etc. I do expect in Oracle’s Cloud offering interaction with Kafka products running in the Oracle Cloud such as the Event Hub, will of course be much easier. We’re looking forward to it.

image-25

About Author

Maarten is a software architect and Oracle ACE. Over the past years he has worked for numerous customers in the Netherlands in developer, analyst and architect roles on topics like software delivery, performance, security and other integration related challenges. Maarten is passionate about his job and likes to share his knowledge through publications, frequent blogging and presentations.

7 Comments

  1. In the plugin it indicates: Allows sync/async commits when the option ‘enable.auto.commit’ is disabled.
    how it would be done?
    thank you

  2. Niloy Chatterjee on

    Hi Team,

    We have a requirement to connect to a kerberised kafka server from OSB. The Kafka servers are secured so we will need extra level of authentication in OSB servers.

    We followed each and every step provided in this blog and also made some customization to handle the extra level of authentication.

    1. Setting up transport in OSB Console/Jdeveloper by installing transport.ear and transport.jar file in OSB_DOMAIN/lib.
    2. Placed the transport-kafka.xml under osb/config
    3. Deployed the ear and jar in weblogic console.
    4. Created a kerberos-kafka-client-jaas.conf file which contains the keytab file details.
    5. Imported kafka certificates in weblogic KSS keystore.
    6. Set up a dummy proxy having the kafka broker details and topic name alongwith Group Identifier.
    7. Created pipeline in to log the message consumed.

    After setting up all the above mentioned changes we are not seeing any success/error messages in the server logs. Even we checked with the Kafka engineers and they are not seeing any consumers at their topic.

    Can you please provide some help on this.

    Thanks and regards,
    Niloy

  3. Hi Martin,

    could you pls give an example of this part: Start the domain

    Execute install.py from the kafka-transport zipfile. Use wlst.sh in my case from: /home/oracle/Oracle/Middleware12212/Oracle_Home/oracle_common/common/bin/wlst.sh
    ?

    Thank you a lot.

  4. Hi Maarten,

    I am working this configurations in 12.2.1.2.0 version, after all the steps I haven’t found kafka in OSB console and I came to this blog for resolution but still I am not able to see the kafka transport in OSB.

    Note: I am able to get this kafka transport in JDEV, and I am running all the process in windows os.

    Please suggest!!

    Thanks in Advance!!

  5. Hi Marteen,

    Thanks for posting such a nice post.I am trying to install kafka transport in OSB 12.2.1.0.0.
    Followed all the above steps correctly and able to see the kafka transport both under deployment section and in osb console.But while creating any service based on kafka transport i am getting the error as:-

    “com.bea.wli.sb.transports.TransportException: No transport provider registered with ID: kafka” while activating the session in osb console.
    Is the above error related to the “transport-kafka.xml” file you have provided in the kafka transport zip file.Please suggest.

    Note:I am running all the above process in windows OS.

    Thanks in Advance!!

  6. Thulasi Yemmela on

    Hi Maarten,

    Nice post. This worked for me like a charm in Linux environment.
    Now I am trying to test this on Windows environment and unable to get the “kafka” listed under transports in both locations, JDeveloper & SBConsole. when I did the install.py, installation completed without any issues and I can see them under deployments. Are you aware of any success or issues in getting this worked on Windows.

    Thanks
    Thulasi

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.