0 
0 Oracle recently made their ‘Kafka on Cloud’ service available: the Event Hub cloud service – offered as part of the Big Data Compute Cloud Service. In this article, I will briefly show the steps I went through to get up and running with this service. To be frank, it was not entirely intuitive – so this may be handy the next time I have to do it or perhaps when you are struggling. One obvious point of confusion is the overloaded use of the ‘service’ – which is applied among others for Kafka Topics.
The steps are:
- Initialize the Oracle BigData Compute CS | Oracle Event Hub Cloud Service – Platform – -expose REST Proxy
- Create Network Access Rule to enable access to the Event Hub Platform instance from the public internet
- Create a Event Hub service instance on the Event Hub Platform – corresponding to a Kafka Topic
- Inspect Event Hub using PSM
- Interact with Event Hub through CURL
- Create application (e.g. NodeJS) to access the Event Hub Service (aka Kafka Topic): produce message, create consumer, consume messages (through the REST proxy) – discussed in a subsequent blog article
Initialize the Oracle BigData Compute CS | Oracle Event Hub Cloud Service – Platform – -expose REST Proxy
From the Dashboard, open the BigData Compute Cloud Service. Open the dropdown menu. Select option Oracle Event Hub Cloud Service – Platform.
Provide the name of the service – the Kafka Event Hub Cluster: soaringEventsHub. Click Next
Specify Basic deployment type (for simple explorations). Upload the SSH Public Key.
For a trial, select just a single node (I started out with 5 nodes and it cost me a huge pile of resources).
Enable REST Access – to be able to communicate with the Event Hub through the REST API. Specify the username and password for accessing the REST proxy. (in hindsight, I probably should have picked a different username than admin).
Click Next. The service will be created – with a generous memory an storage allocation:
When the service creation is complete – details are shown, including the public IP for the Kafka cluster and the ZooKeeper instance
And the Rest Proxy details:
Create Network Access Rule to enable access to the Event Hub Platform instance from the public internet
To be perfectly honest – I am not sure anymore that this step is really required. I had trouble accessing the Event Hub service from my laptop – and I am still not able to connect to the Kafka broker from a regular Kafka client – so I tried many different things, including exposing the public IP address on the Event Hub (Kafk)a node. Whether it helped or made any difference, I am not entirely sure. I will share the steps I took, and you figure out whether you need them. (I am still hoping that accessing the Event Hub from applications running inside the same identity domain on Oracle PaaS will be easier).
I opened the Access Rule for the Event Hub service:
I created an access rule
And enabled it. In red the new access rule. In the blue rectangle the access rule that opens up the REST Proxy for the Event Hub service:
Create Event Hub Service on the Event Hub Service Platform aka Create a Kafka Topic
Open the Oracle Event Hub Cloud Service console (not the Oracle Event Hub Cloud Service – Platform!)
Click on Create Service. This corresponds to creating a Kafka Topic. The naming is awfully confusing here. Create an instance of Event Hub Service on top of an instance of Event Hub Platform Service actually means create a Kafka message topic. Now that is much clearer.
Specify the name for the service – which will be part of the name of the Kafka Topic. Specify the Number of Partitions and the default Retention Period for messages published to the topic. Indicate on which instance of the Event Hub Cloud Service Platform – created in the first step in this article – the topic should be created.
Click Next.
Click Create.
The Topic is created with a name that is composed from the name of the identity domain and the name specified for the Event Hub Service instance: partnercloud17-SoaringEventBus.
The popup details for the “service” (aka Topic) indicate the main configuration details as well as the REST endpoint for this Topic. Messages can be produced at this endpoint:
The log of all Service activity (again, Service here is at the same level of Kafka Topic)
The Service Console now lists these instances:
Inspect Event Hub using PSM
The PSM (PaaS Service Manager) command line interface for the Oracle Cloud (read here on how to install it and get going) can be used to inspect the state of the Event Hub Service. It cannot be used to produce and consume messages though.
Using PSM for Event Hub is described in the documentation: http://docs.oracle.com/en/cloud/paas/java-cloud/pscli/oehcs-commands.html
To list all services (topics)
psm oehcs services
— details for a single service
psm oehcs service –service-name SoaringEventBus
— to create a new Topic:
psm oehcs create-service
— to update the retention time for a topic:
psm oehcs update-service
Interact with Event Hub through CURL
One way to produce messages to the topic and/or consume message from a topic is by using cURL.
It took me some time to figure out the correct syntax for each of these operations. It seems not abundantly well documented/explained at present. Anyways, here it goes. (note: the -k option tells cURL to accept an unknown certificate)
* To produce a message to the topic on Event Hub, use:
curl -X POST -k -u username:password -H “Content-Type: application/vnd.kafka.json.v1+json” -d “{ \”records\”: [ { \”key\”: \”myKey\”, \”value\”: \”mySpecialValue\”}] }” https://public_ip_REST-PROXY:1080/restproxy/topics/NAME_OF_EVENT_HUB_SERVICE
* Create a consumer group: in order to consume messages, you first need to create a consumer group with a consumer instance. Subsequently, you can consume messages through the consumer:
curl -X POST -k -u username:password -H “Content-Type: application/vnd.kafka.json.v1+json” -d “{ \”name\”: \”soaring-eventbus-consumer\”, \”format\”: \”json\” }” https://public_ip_REST-PROXY:1080/restproxy/consumers/soaring-eventbus-consumer-group
* To consume messages – levering the consumer group soaring-eventbus-consumer-group and the consumer instance soaring-events-consumer:
curl -X POST -k -u username:password -H “Accept: application/vnd.kafka.json.v1+json” https://public_ip_REST-PROXY:1080/restproxy/consumers/soaring-eventbus-consumer-group/instances/soaring-eventbus-consumer/topics/partnercloud17-SoaringEventBus
Here three commands: create a consumer group, produce a message and consume (all) available messages – all from the same topic:
Related posts:
Oracle Public Cloud and Kafka – Events powering the cloud – The Oracle PaaS Cloud Event BusHub 
Setting up Oracle Event Hub (Apache Kafka) Cloud Service and Pub & Sub from local Node Kafka client 
Getting started with Oracle PaaS Service Manager Command Line Interface (PSM) 
Prepare and link bind Oracle Database Cloud, Application Container Cloud, Application Container Cache and Event Hub 
Microservices and Updating Data Bound Context on Oracle Cloud with Application Container and Event Hub (plus DBaaS and MongoDB)–Part One
I can’t find any reference to using KSQL on the Oracle Cloud. Is that feature available?
Hi Jonathan,
KSQL does not run on the Kafka Server but as a Java Library from any Java application. You can run the Java Application with KSQL library on Oracle Application Container Cloud.
kind regards
Lucas
Hi Lucas, this lab works great – not Node but Java. (Without the REST proxy) http://www.oracle.com/webfolder/technetwork/tutorials/obe/cloud/oehcs/topicusingnativeapis/topicusingnativeapis.html
Just make sure using the code from the images as the snippets contain small errors that are fixed in the images 🙂
This requires adjusting the security rules as advised in the lab.
Hi, tested this on my mac, many thanks. In the consume there’s a small error using -X POST, should be omitted like in your actual example call (or use -X GET) in cUrl. I didn’t have to create any access rules, worked out of the box. Many thanks again, Mika
Hi Mika,
Thanks for your reaction and the correction.
Were you able to publish to/consume from a topic from a native Kafka client – so not through the REST Proxy? For me it is either the REST proxy (and now additional access rules are required) or using a native Kafka client (such as npm kafka-node) and then the REST Proxy should be disabled and the network access rules have to be created and enabled.
Lucas
Thanks Lucas. Only the REST proxy so far, should try the npm kafka-node also the soonest! 🙂