Apache Kafka on the Oracle Cloud: My First experiences with Oracle Event Hub Cloud Service

Oracle recently made their ‘Kafka on Cloud’ service available: the Event Hub cloud service – offered as part of the Big Data Compute Cloud Service. In this article, I will briefly show the steps I went through to get up and running with this service. To be frank, it was not entirely intuitive – so this may be handy the next time I have to do it or perhaps when you are struggling. One obvious point of confusion is the overloaded use of the ‘service’ – which is applied among others for Kafka Topics.

The steps are:

1. Initialize the Oracle BigData Compute CS | Oracle Event Hub Cloud Service – Platform – -expose REST Proxy
2. Create Network Access Rule to enable access to the Event Hub Platform instance from the public internet
3. Create a Event Hub service instance on the Event Hub Platform – corresponding to a Kafka Topic
4. Inspect Event Hub using PSM
5. Interact with Event Hub through CURL
6. Create application (e.g. NodeJS) to access the Event Hub Service (aka Kafka Topic): produce message, create consumer, consume messages (through the REST proxy) – discussed in a subsequent blog article

Initialize the Oracle BigData Compute CS | Oracle Event Hub Cloud Service – Platform – -expose REST Proxy

 

From the Dashboard, open the BigData Compute Cloud Service. Open the dropdown menu. Select option Oracle Event Hub Cloud Service – Platform.

 

 

 

Provide the name of the service – the Kafka Event Hub Cluster: soaringEventsHub. Click Next

Specify Basic deployment type (for simple explorations). Upload the SSH Public Key.

For a trial, select just a single node (I started out with 5 nodes and it cost me a huge pile of resources).

Enable REST Access – to be able to communicate with the Event Hub through the REST API. Specify the username and password for accessing the REST proxy. (in hindsight, I probably should have picked a different username than admin).

 

Click Next. The service will be created – with a generous memory an storage allocation:

When the service creation is complete – details are shown, including the public IP for the Kafka cluster and the ZooKeeper instance

And the Rest Proxy details:

Create Network Access Rule to enable access to the Event Hub Platform instance from the public internet

To be perfectly honest – I am not sure anymore that this step is really required. I had trouble accessing the Event Hub service from my laptop – and I am still not able to connect to the Kafka broker from a regular Kafka client – so I tried many different things, including exposing the public IP address on the Event Hub (Kafk)a node. Whether it helped or made any difference, I am not entirely sure. I will share the steps I took, and you figure out whether you need them. (I am still hoping that accessing the Event Hub from applications running inside the same identity domain on Oracle PaaS will be easier).

I opened the Access Rule for the Event Hub service:

I created an access rule

And enabled it. In red the new access rule. In the blue rectangle the access rule that opens up the REST Proxy for the Event Hub service:

 

Create Event Hub Service on the Event Hub Service Platform aka Create a Kafka Topic

Open the Oracle Event Hub Cloud Service console (not the Oracle Event Hub Cloud Service – Platform!)

 

Click on Create Service. This corresponds to creating a Kafka Topic. The naming is awfully confusing here. Create an instance of Event Hub Service on top of an instance of Event Hub Platform Service actually means create a Kafka message topic. Now that is much clearer.

 

 

Specify the name for the service – which will be part of the name of the Kafka Topic. Specify the Number of Partitions and the default Retention Period for messages published to the topic. Indicate on which instance of the Event Hub Cloud Service Platform – created in the first step in this article – the topic should be created.

Click Next.

Click Create.

The Topic is created with a name that is composed from the name of the identity domain and the name specified for the Event Hub Service instance: partnercloud17-SoaringEventBus.

 

The popup details for the “service”  (aka Topic) indicate the main configuration details as well as the REST endpoint for this Topic. Messages can be produced at this endpoint:

The log of all Service activity (again, Service here is at the same level of Kafka Topic)

The Service Console now lists these instances:

 

 

Inspect Event Hub using PSM

The PSM (PaaS Service Manager) command line interface for the Oracle Cloud (read here on how to install it and get going) can be used to inspect the state of the Event Hub Service. It cannot be used to produce and consume messages though.

Using PSM for Event Hub is described in the documentation:  http://docs.oracle.com/en/cloud/paas/java-cloud/pscli/oehcs-commands.html

To list all services (topics)
psm oehcs services

— details for a single service
psm oehcs service –service-name SoaringEventBus

— to create a new Topic:

psm oehcs create-service

— to update the retention time for a topic:

psm oehcs update-service

 

Interact with Event Hub through CURL

One way to produce messages to the topic and/or consume message from a topic is by using cURL.

It took me some time to figure out the correct syntax for each of these operations. It seems not abundantly well documented/explained at present. Anyways, here it goes. (note: the -k option tells cURL to accept an unknown certificate)

* To produce a message to the topic on Event Hub, use:

curl -X POST -k -u username:password -H “Content-Type: application/vnd.kafka.json.v1+json”  -d “{ \”records\”: [ { \”key\”: \”myKey\”, \”value\”: \”mySpecialValue\”}] }” https://public_ip_REST-PROXY:1080/restproxy/topics/NAME_OF_EVENT_HUB_SERVICE

* Create a consumer group: in order to consume messages, you first need to create a consumer group with a consumer instance. Subsequently, you can consume messages through the consumer:

curl -X POST -k -u username:password -H “Content-Type: application/vnd.kafka.json.v1+json”  -d “{ \”name\”:  \”soaring-eventbus-consumer\”, \”format\”: \”json\” }”  https://public_ip_REST-PROXY:1080/restproxy/consumers/soaring-eventbus-consumer-group

* To consume messages – levering the consumer group soaring-eventbus-consumer-group and the consumer instance soaring-events-consumer:
curl -X POST -k -u username:password -H “Accept: application/vnd.kafka.json.v1+json”    https://public_ip_REST-PROXY:1080/restproxy/consumers/soaring-eventbus-consumer-group/instances/soaring-eventbus-consumer/topics/partnercloud17-SoaringEventBus

 

Here three commands: create a consumer group, produce a message and consume (all) available messages – all from the same topic: