Oracle Cloud Streaming Service – Scalable, Reliable, Kafka-like Event service on OCI

Oracle Cloud Infrastructure offers the Streaming service. This service is easily described as a Kafka-like fully managed event platform for high volume streams of data that can be processed in real or delayed time in a durable, reliable way. And from the documentation: “Streaming can be used for messaging, ingesting high-volume data such as application logs, operational telemetry, web click-stream data, or other use cases in which data is produced and processed continually and sequentially in a publish-subscribe messaging model.”

The picture shows how a Stream can be published to by any Publisher (on prem, in OCI or in third party cloud) working through the REST API or the Java SDK) and can be consumed by any consumer, wherever it lives. Note that consumers have to actively come to the Streaming service and collect any messages waiting for them. There is no push mechanism that will trigger listeners whenever new messages have been published to the Stream.

Streaming uses streams (similar to Kafka Topics) as the log of messages that somehow belong together. Messages can be published to streams via HTTP calls to a REST API or via a Java based SDK (as well as the OCI CLI). Consuming messages works through these same channels. The Streaming service does not push messages to listeners (unfortunately). Messages are retained for up to 7 days. There are limits on how much data a Stream can handle per second, but these limits are fairly high (1MB per partition per second, 1MB maximum message size, five consume calls per second).

Streams can be managed through the CLI, the REST API and the Console. Test messages can be produced and consumed from the Console, which makes it easy to get started.

Quick Start OCI Streaming – through the Console

Creating a Stream in the Console is dead easy. Producing a Test Message too and Consuming the Test Messages as well. Getting going this way takes all of three minutes, if not less.

Open Streams in the OCI Console:

Click on Create Stream:

Provide some details on the stream – its name, the number of partitions, the retention time:

and press Create Stream. Now the Stream will be created.

Which takes a few seconds.

When the Stream has been created and is active, click on Produce Test Message to …

Type a message and press Produce. The console will indicate that the message was produced successfully.

To check on the message, close the popup window and press Load Messages. All recently published messages on the stream are displayed – [only] including the one that was just published:

Consume Messages through the REST API

Like all OCI services, Streaming provides a REST API through which admin tasks (such as creating streams) can be performed and which are used for publishing and consuming messages. In addition to the REST API, a Java SDK is available for a more native experience.

Working with the REST API for Streaming is very similar to all other OCI REST API activity. The main challenge being the signing of the HTTP requests. The REST API is documented here: https://docs.cloud.oracle.com/iaas/api/#/en/streaming/20180418/Stream/. Here a snippet from the Node code that consumes messages from Streams through the REST API:

Node JavaScript code discussed in this article is available in GitHub: https://github.com/lucasjellema/oci-cloud-native-explorations/tree/master/streams-pubsub.

Publish Messages through the REST API

The REST API also supports publishing messages.

Publishing messages is easily done with a signed HTTP POST request to an endpoint that has the stream identifier as path parameter. The request body contains a JSON string that defines a message property which is a collection of message details that each have a key and a value. The value is the base64 encoded string that represents the actual message.

After running the code, the messages have been published. Checking in the console for the newly published messages:

And using the Node based Message Consumer introduced in the previous section:

Resources

Node JavaScript code discussed in this article is available in GitHub: https://github.com/lucasjellema/oci-cloud-native-explorations/tree/master/streams-pubsub.

Documentation on OCI Streaming Service: https://docs.cloud.oracle.com/iaas/Content/Streaming/Concepts/streamingoverview.htm

REST API for Streams service: https://docs.cloud.oracle.com/iaas/api/#/en/streaming/20180418/Stream/

Blog article Todd Sharp: Getting your feet wet with Oracle Streaming: https://blogs.oracle.com/developers/getting-your-feet-wet-with-oci-streams