Setting up Oracle Event Hub (Apache Kafka) Cloud Service and Pub & Sub from local Node Kafka client

Lucas Jellema

Oracle offers an Event Bus Cloud Service – an enterprise grade Apache Kafka instance – with large numbers of partitions and topics, (retained) messages and distributed nodes. Setting up this cloud service is simple enough – especially once you know what to do, as I will demonstrate in this article. In order to communicate with this Event Bus from a local client – in this case created with Node – we need to open up some ports for access from the public internet.

The steps I went through:

  • create Oracle Event Hub Cloud Service – Platform instance (the Kafka & Zookeeper instance)
  • create Oracle Event Hub Cloud Service – Service instance (a Topic)
  • create two network rules to allow access from the public internet to ports 2181 – Zookeeper – and 6667 – Kafka Server)
  • create two Node applications using the kafka-node package – one to produce and one to consume (based on the work done by Kunal Rupani of Oracle (GitHub: https://github.com/kunalrupani/OracleEventHubConsumer)

Note: the Event Hub Cloud Service is a metered service, billed per hour. The cost for the smallest shape is around $0,70 per hour (or $200/month for non metered).

    Create Oracle Event Hub Cloud Service – Platform instance (the Kafka & Zookeeper instance)

    image

    image

    image

    imageimage

    SNAGHTML23dbd55a


    Create Oracle Event Hub Cloud Service – Service instance (a Topic)

    Switch to the Big Data Compute CS and select the service category Oracle Event Hub Cloud Service –Topics.

    SNAGHTML23dcc70d

    image

    image

    image

    image

    Create two network rules to allow access from the public internet to ports 2181 – Zookeeper – and 6667 – Kafka Server)

    SNAGHTML23de6174

    image

    SNAGHTML23ded01c

    image

    Create two Node applications using the kafka-node package

    one to produce and one to consume (based on the work done by Kunal Rupani of Oracle (GitHub: https://github.com/kunalrupani/OracleEventHubConsumer)

      Note: the topic was created as microEventBus. The actual name under which it is accessed is partnercloud17-microEventBus (which is identity domain as prefix to the topic name)

      The package.json contains the dependency on kafka-node:

      image

      The straightforward code for producing to the Event Hub Kafka Topic:

      var EVENT_HUB_PUBLIC_IP = '129.xxxxxxxx';
      var TOPIC_NAME = 'partnercloud17-microEventBus';
      var ZOOKEEPER_PORT = 2181;
      
      var kafka = require('kafka-node');
      var Producer = kafka.Producer;
      var client = new kafka.Client(EVENT_HUB_PUBLIC_IP + ':'+ZOOKEEPER_PORT);
      var producer = new Producer(client);
      
      let payloads = [
        { topic: TOPIC_NAME, messages: '*', partition: 0 }
      ];
      
      console.log(payloads[0].messages);
      setInterval(function () {
        console.log('called about every 1 second');
        producer.send(payloads, function (err, data) {
          if (err) {
            console.error(err);
          }
          console.log(data);
        });
        if (payloads[0].messages.length < 10) {
          payloads[0].messages = payloads[0].messages + "*";
        }
        else {
          payloads[0].messages = "*";
        }
      }, 1000);
      
      

      And the similar code for consuming. Note that the producer talks to Zookeeper (port 2181) and the consumer to the Kafka Server itself (port 6667):

      var EVENT_HUB_PUBLIC_IP = '129xxxxxxx';
      var TOPIC_NAME = 'partnercloud17-microEventBus';
      var KAFKA_SERVER_PORT = 6667;
      
      var kafka = require('kafka-node'),
          Consumer = kafka.Consumer,
          client = new kafka.KafkaClient({ "kafkaHost": EVENT_HUB_PUBLIC_IP + ':' + KAFKA_SERVER_PORT }),
          consumer = new Consumer(
              client,
              [
                  { topic: TOPIC_NAME, offset: 1 }
              ],
              {
                  autoCommit: false,
                  fromOffset: true
              }
          );
      
      consumer.on('message', function (message) {
          console.log(message.value)
      });
      

      Here is the output for producing:

      image

      and here for consuming:

      image

      Leave a Reply

      This site uses Akismet to reduce spam. Learn how your comment data is processed.

      Next Post

      Oracle SOA Suite and WebLogic: Overview of key and keystore configuration

      Facebook0TwitterLinkedinKeystores and the keys within can be used for security on the transport layer and application layer in Oracle SOA Suite and WebLogic Server. Keystores hold private keys (identity) but also public certificates (trust). This is important when WebLogic / SOA Suite acts as the server but also when it […]