OCI Queue– simple, powerful, heavy duty reliable decoupling image 31

OCI Queue– simple, powerful, heavy duty reliable decoupling

The new OCI Queue Service provides a straightforward offering: publish messages into a persisted store and allow consumers to retrieve these messages from that store – the oldest message first (FIFO). image

Interaction with OCI Queue can be done through REST API calls or through the STOMP-protocol (the Simple (or Streaming) Text Orientated Messaging Protocol. A STOMP client can connect and interact “natively” with OCI Queue. Note that the OCI SDKs for languages such as Java, Python and Go has support for Queue, as has the OCI CLI.

Multiple producers can produce messages to the same queue. Multiple consumers can poll the queue for messages – and compete with each for getting hold of a message – but only one consumer will actually get the message (no fan-out). Each message is consumed successfully only once (if a consumer does not report success, the message becomes available again for another consumer). There is no notion of subscriptions.

image

Messages have a string payload (not byte[]); the string can of course be in any format, including XML, JSON, CSV, and a binary payload in Base64-encoded message and even compressed formats such as gzip.

This service appears robust. It is not super fast: it is not targeted at streaming processing, near-real-time sub-second latency (for that use the Apache Kafka like OCI Streaming service); think in the at least multi-second message processing times. The services does not yet guarantee FIFO – even though its behavior will in reality be very close to FIFO.  Note: the queue is passive, like a file system or database. Messages are sent to it – and it will dutifully persist them – and can be requested from it – and it will return them as requested. The queue does not itself push anything or take a form of initiative. Only when the retention time ends for a message does the queue spring into action: it removes that message.

Queues as well as individual Messages can have a retention time and a maximum number of delivery attempts. Messages will not be retained longer than the retention time – and will immediately be removed when a consumer indicates success processing (by deleting the messages it has consumed).

When a consumer consumes a message, it specifies the visibility time out (default 30 seconds, maximum is 12 hours). During this window, the message is hidden on the queue (but still retained). The consumer can extend the visibility timeout if it needs more time to process the message. When a consumer consumes the message from the queue, the delivery attempt counter is increased by one. If the consumer does not signal success within the “visibility timeout (window)”, the delivery attempt is assumed to have failed and the message becomes again available for consumption.

Messages that are not successfully processed within the maximum number of delivery attempts are moved to a Dead Letter Queue and will be kept there until either the retention time ends or a form of “manual” processing handles the message.

A consumer can't process a message from the queue, so the message is moved to the dead letter queue.

Unlike the similar Azure Storage Queue offering, OCI Queue does not have delayed delivery: the ability to have message become available for consumption at a specific moment. I like that feature, so I hope it will be added to OCI Queue. (by including an intended delivery timestamp in messages and having a generic consumer that is executed immediately after publishing messages to the queue and that grabs all messages it can, releasing the ones with a delivery timestamp in the past and extending the visibility timeout window to the intended delivery timestamp – we can emulate such behavior)

OCI Queues are fully managed. Once a message is successfully published, it is guaranteed to be persisted. Scalability, redundancy, fail over, back up & recovery, internal routing, storage etc are all taken care of. The Queue is used as a (regional) platform service – readily available at all times. Messages are encrypted – while in transit and when stored.

Access to queues is governed using IAM policies. The resource type queue-push and queue-pull can be granted to dynamic groups representing applications or compute instances (Functions, Container Instance, VM).

Queues are not partitioned – each queue is just one big pipe of messages. When consuming messages, there is no support for consumer aware subscriptions, (content based) filtering, changing the read offset, the concept of consumer group etc. It is simple and straightforward. Suitable for some use cases – and not for others.

Some numbers: A maximum of 10 queues per tenancy per region. Each queue can hold up to 100K in-flight messages of up to 128KB per message. The total storage for all queues combined is capped at 20GB – with a maximum of 2GB per queue. Messages can be retained for up to 7 days. Messages can be published in batches – up to 20 messages of in total up to 512KB. Messages can also be consumed in batches – up to 20 messages with a combined size of up to 2MB. Interactions with a queue take place through API requests (PUT, GET, DELETE). The queues can handle up to 1000 API requests per second. The service is priced on the number of API calls. The first million (1M) calls each month is free.

The total number of queues allowed by default is fairly small. It would seem that this type of queue is positioned for example for straight through (but not near real time high speed) processing, of fairly large messages. 

Resources

OCI Documentation on Queue Service – https://docs.oracle.com/en-us/iaas/Content/queue/overview.htm#overview

Oracle Blog: OCI Queue is now available in all commercial regions – Phil Wilkins and Abhishek Bhaumikhttps://blogs.oracle.com/cloud-infrastructure/post/oci-queue-is-now-available-in-all-commercial-regions

OCI Queue Product Page – https://www.oracle.com/cloud/queue/?source=:so:li:or:dg:odv::: 

Phil Wilkins Blog on Oracle Cloud Messaging Service – https://www.oracle.com/technical-resources/articles/cloud/wilkins-ocms.html (2017 – history repeating itself?!)

OCI Architecture Center – Learn About Selecting an Appropriate Messaging Solution on OCI – selecting a messaging solution (comparing Notifications, Integration, Queue, Streaming, TEQ in Database)

OCI Queue Demo repository on GitHub – a sample Java application that includes code to create a queue, publish messages to the queue, read messages from the queue and delete these messages from the queue, delete a queue and inspect/list queues.

STOMP – the Simple (or Streaming) Text Orientated Messaging Protocol. – https://stomp.github.io/

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.