Setting up Oracle Event Hub (Apache Kafka) Cloud Service and Pub & Sub from local Node Kafka client

0
Share this on .. Tweet about this on TwitterShare on LinkedIn38Share on Facebook0Share on Google+0Email this to someoneShare on Tumblr0Buffer this page

Oracle offers an Event Bus Cloud Service – an enterprise grade Apache Kafka instance – with large numbers of partitions and topics, (retained) messages and distributed nodes. Setting up this cloud service is simple enough – especially once you know what to do, as I will demonstrate in this article. In order to communicate with this Event Bus from a local client – in this case created with Node – we need to open up some ports for access from the public internet.

The steps I went through:

  • create Oracle Event Hub Cloud Service – Platform instance (the Kafka & Zookeeper instance)
  • create Oracle Event Hub Cloud Service – Service instance (a Topic)
  • create two network rules to allow access from the public internet to ports 2181 – Zookeeper – and 6667 – Kafka Server)
  • create two Node applications using the kafka-node package – one to produce and one to consume (based on the work done by Kunal Rupani of Oracle (GitHub: https://github.com/kunalrupani/OracleEventHubConsumer)

Note: the Event Hub Cloud Service is a metered service, billed per hour. The cost for the smallest shape is around $0,70 per hour (or $200/month for non metered).

    Create Oracle Event Hub Cloud Service – Platform instance (the Kafka & Zookeeper instance)

    image

    image

    image

    imageimage

    SNAGHTML23dbd55a


    Create Oracle Event Hub Cloud Service – Service instance (a Topic)

    Switch to the Big Data Compute CS and select the service category Oracle Event Hub Cloud Service –Topics.

    SNAGHTML23dcc70d

    image

    image

    image

    image

    Create two network rules to allow access from the public internet to ports 2181 – Zookeeper – and 6667 – Kafka Server)

    SNAGHTML23de6174

    image

    SNAGHTML23ded01c

    image

    Create two Node applications using the kafka-node package

    one to produce and one to consume (based on the work done by Kunal Rupani of Oracle (GitHub: https://github.com/kunalrupani/OracleEventHubConsumer)

      Note: the topic was created as microEventBus. The actual name under which it is accessed is partnercloud17-microEventBus (which is identity domain as prefix to the topic name)

      The package.json contains the dependency on kafka-node:

      image

      The straightforward code for producing to the Event Hub Kafka Topic:

      var EVENT_HUB_PUBLIC_IP = '129.xxxxxxxx';
      var TOPIC_NAME = 'partnercloud17-microEventBus';
      var ZOOKEEPER_PORT = 2181;
      
      var kafka = require('kafka-node');
      var Producer = kafka.Producer;
      var client = new kafka.Client(EVENT_HUB_PUBLIC_IP + ':'+ZOOKEEPER_PORT);
      var producer = new Producer(client);
      
      let payloads = [
        { topic: TOPIC_NAME, messages: '*', partition: 0 }
      ];
      
      console.log(payloads[0].messages);
      setInterval(function () {
        console.log('called about every 1 second');
        producer.send(payloads, function (err, data) {
          if (err) {
            console.error(err);
          }
          console.log(data);
        });
        if (payloads[0].messages.length < 10) {
          payloads[0].messages = payloads[0].messages + "*";
        }
        else {
          payloads[0].messages = "*";
        }
      }, 1000);
      
      

      And the similar code for consuming. Note that the producer talks to Zookeeper (port 2181) and the consumer to the Kafka Server itself (port 6667):

      var EVENT_HUB_PUBLIC_IP = '129xxxxxxx';
      var TOPIC_NAME = 'partnercloud17-microEventBus';
      var KAFKA_SERVER_PORT = 6667;
      
      var kafka = require('kafka-node'),
          Consumer = kafka.Consumer,
          client = new kafka.KafkaClient({ "kafkaHost": EVENT_HUB_PUBLIC_IP + ':' + KAFKA_SERVER_PORT }),
          consumer = new Consumer(
              client,
              [
                  { topic: TOPIC_NAME, offset: 1 }
              ],
              {
                  autoCommit: false,
                  fromOffset: true
              }
          );
      
      consumer.on('message', function (message) {
          console.log(message.value)
      });
      

      Here is the output for producing:

      image

      and here for consuming:

      image

      Share this on .. Tweet about this on TwitterShare on LinkedIn38Share on Facebook0Share on Google+0Email this to someoneShare on Tumblr0Buffer this page

      About Author

      Lucas Jellema, active in IT (and with Oracle) since 1994. Oracle ACE Director and Oracle Developer Champion. Solution architect and developer on diverse areas including SQL, JavaScript, Docker, Machine Learning, Java, SOA and microservices, events in various shapes and forms and many other things. Author of the Oracle Press books: Oracle SOA Suite 11g Handbook and Oracle SOA Suite 12c Handbook. Frequent presenter on community events and conferences such as JavaOne, Oracle Code and Oracle OpenWorld.

      Leave a Reply