Messages published to an Oracle Cloud Infrastructure (OCI) Queue may not immediately be intended for consumption. It is a not uncommon requirement to make messages available for consumption only at a specific moment in time, rather than right away. OCI Queue does not currently have built in support for a “delayed delivery timestamp” when publishing a message. However, with its visibility timeout mechanism, it is very straightforward to implement functionality that comes quite close.
1. We start with a list of messages – one or more. For each message, we know the intended delivery time.
The steps:
2. publish messages
the result will be a list of message identifiers assigned by the queue service, in the same order as the list of published messages
3. poll/get messages – set a visibility timeout of a few seconds
the result will be a list of messages – each with an id and a “receipt” key; also an expiry time and a current visibility timeout
4. loop over the retrieved messages; each message whose id corresponds to an id value in the result of step 1 is one of ours; this message can be updated with a call to the update-message(s) operation in the API; we have to pass the message receipt key and the new visibility window; to reduce the number of API calls, we combine as many messages in this update call as we can
in case we have not yet found all the messages published in step 1, we have to repeat steps 2 and 3
5. regular consumers – poll for messages and only receive messages that are eligible for consumption. It is possible that a regular consumer preempts the “delayed delivery controller consumer” and gets a message it should not yet get. This may not be a problem or we can add logic to the consumer to not process such a message (in which case the intended delivery timestamp should be part of the message itself)
Note: this method only works for deliveries delayed by no more than 12 hours (the longest duration of the invisibility timeout).
Additionally or alternatively, we can explicitly include the intended delivery time in a property in the message itself. We have to include a generic step in all potential consumers that poll the queue that evaluates this delivery timestamp and updates the message with the appropriate visibility timeout – without of course processing it. When all consumers behave like this, we can even support deliveries delayed by several days, up to the retention time of the message (and provided the maximum delivery attempts defined for the message allow for that many poll & update actions).
Note
The method described here requires at least two additional API calls for every (batch of) message(s): a poll/read and an update,. The Queue service is charged by API call (with 1M calls free each month). These added calls may be an inhibiting factor – as well the additional processing involved (no green IT).
The best way forward obviously would be for Oracle to add this feature in an upcoming release of OCI Queue. Given the existing support for visibility timeout as part of Get and Update of messages, it would be not very complex to allow us to define the visibility time out off the bat – when we publish the message. Or would it, dear Oracle?
Resources
My earlier blog article: First Steps with OCI Queue Service–fully managed massively scalable FIFO queue
OCI CLI Documentation on Queue Message manipulation: https://docs.oracle.com/en-us/iaas/tools/oci-cli/3.22.0/oci_cli_docs/cmdref/queue.html
