Delayed Delivery in OCI Queue Service–time based message release

Lucas Jellema
0 0
Read Time:2 Minute, 57 Second

Messages published to an Oracle Cloud Infrastructure (OCI) Queue may not immediately be intended for consumption. It is a not uncommon requirement to make messages available for consumption only at a specific moment in time, rather than right away. OCI Queue does not currently have built in support for a “delayed delivery timestamp” when publishing a message. However, with its visibility timeout mechanism, it is very straightforward to implement functionality that comes quite close.

1. We start with a list of messages – one or more. For each message, we know the intended delivery time.

image

The steps:

2. publish messages

the result will be a list of message identifiers assigned by the queue service, in the same order as the list of published messages

3. poll/get messages – set a visibility timeout of a few seconds

the result will be a list of messages – each with an id and a “receipt” key; also an expiry time and a current visibility timeout

4. loop over the retrieved messages; each message whose id corresponds to an id value in the result of step 1 is one of ours; this message can be updated with a call to the update-message(s) operation in the API; we have to pass the message receipt key and the new visibility window; to reduce the number of API calls, we combine as many messages in this update call as we can

in case we have not yet found all the messages published in step 1, we have to repeat steps 2 and 3

5. regular consumers – poll for messages and only receive messages that are eligible for consumption. It is possible that a regular consumer preempts the “delayed delivery controller consumer” and gets a message it should not yet get. This may not be a problem or we can add logic to the consumer to not process such a message (in which case the intended delivery timestamp should be part of the message itself)

Note: this method only works for deliveries delayed by no more than 12 hours (the longest duration of the invisibility timeout).

Additionally or alternatively, we can explicitly include the intended delivery time in a property in the message itself. We have to include a generic step in all potential consumers that poll the queue that evaluates this delivery timestamp and updates the message with the appropriate visibility timeout – without of course processing it. When all consumers behave like this, we can even support deliveries delayed by several days, up to the retention time of the message (and provided the maximum delivery attempts defined for the message allow for that many poll & update actions).

Note

The method described here requires at least two additional API calls for every (batch of) message(s): a poll/read and an update,. The Queue service is charged by API call (with 1M calls free each month). These added calls may be an inhibiting factor – as well the additional processing involved (no green IT).

The best way forward obviously would be for Oracle to add this feature in an upcoming release of OCI Queue. Given the existing support for visibility timeout as part of Get and Update of messages, it would be not very complex to allow us to define the visibility time out off the bat – when we publish the message. Or would it, dear Oracle?

Resources

My earlier blog article: First Steps with OCI Queue Service–fully managed massively scalable FIFO queue

OCI CLI Documentation on Queue Message manipulation: https://docs.oracle.com/en-us/iaas/tools/oci-cli/3.22.0/oci_cli_docs/cmdref/queue.html

About Post Author

Lucas Jellema

Lucas Jellema, active in IT (and with Oracle) since 1994. Oracle ACE Director and Oracle Developer Champion. Solution architect and developer on diverse areas including SQL, JavaScript, Kubernetes & Docker, Machine Learning, Java, SOA and microservices, events in various shapes and forms and many other things. Author of the Oracle Press book Oracle SOA Suite 12c Handbook. Frequent presenter on user groups and community events and conferences such as JavaOne, Oracle Code, CodeOne, NLJUG JFall and Oracle OpenWorld.
Happy
Happy
0 %
Sad
Sad
0 %
Excited
Excited
0 %
Sleepy
Sleepy
0 %
Angry
Angry
0 %
Surprise
Surprise
0 %

Average Rating

5 Star
0%
4 Star
0%
3 Star
0%
2 Star
0%
1 Star
0%

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Next Post

Apache NiFi: Let through the latest event in a timeframe

In the IoT world, some devices generate large volumes of events that can be difficult for back-end systems to process in real time. Of course you can use NiFi to throttle messages, however this will not be sufficient if the flow of events is consistently higher than what can be […]
%d bloggers like this: