Oracle Cloud Streaming Service – Scalable, Reliable, Kafka-like Event service on OCI

0

Oracle Cloud Infrastructure offers the Streaming service. This service is easily described as a Kafka-like fully managed event platform for high volume streams of data that can be processed in real or delayed time in a durable, reliable way. And from the documentation: “Streaming can be used for messaging, ingesting high-volume data such as application logs, operational telemetry, web click-stream data, or other use cases in which data is produced and processed continually and sequentially in a publish-subscribe messaging model.”

The picture shows how a Stream can be published to by any Publisher (on prem, in OCI or in third party cloud) working through the REST API or the Java SDK) and can be consumed by any consumer, wherever it lives. Note that consumers have to actively come to the Streaming service and collect any messages waiting for them. There is no push mechanism that will trigger listeners whenever new messages have been published to the Stream.

image

Streaming uses streams (similar to Kafka Topics) as the log of messages that somehow belong together. Messages can be published to streams via HTTP calls to a REST API or via a Java based SDK (as well as the OCI CLI). Consuming messages works through these same channels. The Streaming service does not push messages to listeners (unfortunately). Messages are retained for up to 7 days. There are limits on how much data a Stream can handle per second, but these limits are fairly high (1MB per partition per second, 1MB maximum message size, five consume calls per second).

Streams can be managed through the CLI, the REST API and the Console. Test messages can be produced and consumed from the Console, which makes it easy to get started.

Quick Start OCI Streaming – through the Console

Creating a Stream in the Console is dead easy. Producing a Test Message too and Consuming the Test Messages as well. Getting going this way takes all of three minutes, if not less.

Open Streams in the OCI Console:

image

Click on Create Stream:

image

Provide some details on the stream – its name, the number of partitions, the retention time:

image

and press Create Stream. Now the Stream will be created.

image

Which takes a few seconds.

image

When the Stream has been created and is active, click on Produce Test Message to …

image

Type a message and press Produce. The console will indicate that the message was produced successfully.

To check on the message, close the popup window and press Load Messages. All recently published messages on the stream are displayed – [only] including the one that was just published:

image

Consume Messages through the REST API

Like all OCI services, Streaming provides a REST API through which admin tasks (such as creating streams) can be performed and which are used for publishing and consuming messages. In addition to the REST API, a Java SDK is available for a more native experience.

Working with the REST API for Streaming is very similar to all other OCI REST API activity. The main challenge being the signing of the HTTP requests. The REST API is documented here: https://docs.cloud.oracle.com/iaas/api/#/en/streaming/20180418/Stream/. Here a snippet from the Node code that consumes messages from Streams through the REST API:

image

Node JavaScript code discussed in this article is available in GitHub: https://github.com/lucasjellema/oci-cloud-native-explorations/tree/master/streams-pubsub.

SNAGHTML4ce929c8

Publish Messages through the REST API

The REST API also supports publishing messages.

Publishing messages is easily done with a signed HTTP POST request to an endpoint that has the stream identifier as path parameter. The request body contains a JSON string that defines a message property which is a collection of message details that each have a key and a value. The value is the base64 encoded string that represents the actual message.

image

image

After running the code, the messages have been published. Checking in the console for the newly published messages:

image

And using the Node based Message Consumer introduced in the previous section:

SNAGHTML4cf84efa

Resources

Node JavaScript code discussed in this article is available in GitHub: https://github.com/lucasjellema/oci-cloud-native-explorations/tree/master/streams-pubsub.

Documentation on OCI Streaming Service: https://docs.cloud.oracle.com/iaas/Content/Streaming/Concepts/streamingoverview.htm

REST API for Streams service: https://docs.cloud.oracle.com/iaas/api/#/en/streaming/20180418/Stream/

Blog article Todd Sharp: Getting your feet wet with Oracle Streaming: https://blogs.oracle.com/developers/getting-your-feet-wet-with-oci-streams

About Author

Lucas Jellema, active in IT (and with Oracle) since 1994. Oracle ACE Director and Oracle Developer Champion. Solution architect and developer on diverse areas including SQL, JavaScript, Kubernetes & Docker, Machine Learning, Java, SOA and microservices, events in various shapes and forms and many other things. Author of the Oracle Press book Oracle SOA Suite 12c Handbook. Frequent presenter on user groups and community events and conferences such as JavaOne, Oracle Code, CodeOne, NLJUG JFall and Oracle OpenWorld.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.