In an earlier article, I have explored the promise of OCI Queue – a managed queue service launched in December 2022. In this article I will show my first real activities with OCI Queue. Small steps, granted. But surely helpful to give you some idea of what this service is about.
The steps I will describe are simple and straightforward:
- create a queue – define queue configuration (default retention time, visibility timeout)
- publish a message to the queue (from the console)
- verify the arrival of the message on the queue
- poll for / peek into the message from the console (this is not proper consumption)
- poll four more times; at this point we hit the maximum delivery attempts without successful consumption; the message is moved to the Dead Letter Queue
- check the DLQ for the message that should now have been moved here
- try to publish and consume through the CLI?
Create a Queue through the OCI Console
Navigate to Queue service – for example through search “queue”
In the Queue overview page
click on Create Queue.
Type the name of the new queue – here first-q.
Accept the default settings for visibility timeout (how long is a message hidden from message consumers that come polling after a message has been retrieved by a consumer), maximum retention period (after how much time after publication will a message be removed if no successful consumption has taken place), maximum delivery attempts (the number of times the message can be consumed – and not subsequently be deleted by the consumer to signal a successful consumption – before being moved to the Dead Letter Queue) and encryption (accept the default key offered or define a custom key for message encryption).
In this case I accept the defaults. Press Create to have the managed queue provisioned according to my specifications.
The queues overview page is shown. The status of first-q is shown:
After a little while the queue will be active:
I can start interacting with the queue. And the interesting thing is: it is a fully managed black box,. I do not know where it lives, how it is managed nor even which technology is used to implement it. I interact through APIs and what’s underneath is hidden from view. As it should be! (I am curious of course, but perhaps I do not know the actual implementation, lest I start relying on it somehow).
Click on the queue and drill down to the details page:
Click on Action | Send Message to publish a message on the queue.
Type some text in the popup window and press Send message. This will invoke the REST API that publishes the message on the queue. It is persisted at this point, and will be retained for 24 hours (or until it is consumed successfully).
After closing this window, the overview page for the queue looks like this:
There is 1 visible message (and 0 invisible messages) on the queue.
I click on Poll for messages. The console now behaves a little like a message consumer: it polls for messages. The console cannot truly consume messages (it cannot delete the messages from the queue). Every time however the console polls a specific message, the delivery attempt count for the message is increased. The popup that appears after I press the poll button says as much:
The messages it finds are displayed:
Note how the delivery attempts is already at 1 because of this poll operation.
The visibility timeout is 30 seconds. That means that for 30 seconds, this message is not visible to other consumers (or to the console itself if you poll again).
After polling four more times – each time pausing for at least 30 seconds between poll actions to allow the message to become available again to be polled – the message hits five delivery attempts – and is removed from the queue:
and moved to the Dead Letter Queue (for the remainder of the retention time or until the message is manually handled from the DLQ. Here is the UI that presents the DLQ:
I can purge the entire DLQ or delete individual messages from the Console. Using the API or the OCI CLI I can read the message and publish it afresh on the queue (where it will be just a new message at the FI end of the FIFO queue) before removing it from the DLQ.
Working with the OCI Queue through OCI CLI
I need the OCID of the Queue – easily obtained from the Queue details page. Note: the DLQ has its own OCID.
Open Cloud Shell.
List Queues in Compartment:
oci queue queue-admin queue list –c <compartmentID>
Show details for a queue:
oci queue queue-admin queue get –-queue-id <Queue Id>
To publish two messages:
oci queue messages put-messages –-queue-id <Queue Id> –messages file://messages.json
File messages.json is created with this command: oci queue messages put-messages –generate-param-json-input messages > messages.json
The required file is extremely simple. Let;’s edit it:
And let;’s now publish these messages to queue first-q:
oci queue messages put-messages –from-json file://put-messages-on-queue.json –queue-id $queue_id
My request for publishing messages fails with an HTTP-404 – suggesting the resource cannot be found or is not accessible somehow. I cannot understand why this error occurs,. The queue exists – I have verified the queue OCID several times and by the way – I can inspect the queue just fine using this OCID. I am connected to Cloud Shell as the admin user of my tenancy. It should have full privileges – no questions asked.
Note: this alternative approach gives me the same result: Using: oci queue messages put-messages –generate-full-command-json-input I can have the CLI generate a JSON file that has the proper structure required for putting messages on the queue.
I can then edit a JSON file with the proper values in it and finally execute the operation using:
oci queue messages put-messages -–from-json file://put-messages-on-queue.json
I have also added a property compartmentId with the correct value, but that does not change anything.
I will try to get help from Oracle (product management) on this particular issue. Edit: I did get help (thanks Phil) and it turns out I ran into a bug. It will be fixed.
OCI CLI Raw Request
Let’s try the OCI CLI raw request – see if that helps any. Use the messaging endpoint found in the Queue Details page in the target URI.
oci raw-request –http-method POST –target-uri https://cell-1.queue.messaging.us-ashburn-1.oci.oraclecloud.com/20210201/queues/ocid1.queue.oc1.iad.amaaaaaa6sde7caak6kpobll3d7cjnd4ltmeeehmxa7w7puqhkyjzm3zuooq/messages –request-body file://put-messages-on-queue.json
And this works!
The contents of file put-messages-on-queue.json:
Two messages published on the queue. As can be confirmed in the console:
Let’s see if a raw request can also be used to poll for them:
First this command to create the json file to use for the raw request payload
oci queue messages get-messages –generate-full-command-json-input > get-messages.json
Then edit the resulting file get-messages.json
And execute the command:
oci raw-request –http-method GET –target-uri https://cell-1.queue.messaging.us-ashburn-1.oci.oraclecloud.com/20210201/queues/ocid1.queue.oc1.iad.amaaaaaa6sde7caak6kpobll3d7cjnd4ltmeeehmxa7w7puqhkyjzm3zuooq/messages –request-body file://get-messages.json
with this result:
It seems I am getting only a single message, despite specifying a limit of 5 in the request body.
After publishing three messages to the queue and using this raw request four times in a row, I receive the first, second and third message and then again the first. Despite specifying a long visibility timeout.
When I add a request parameter ?visibilityInSeconds=3599&limit=5 then this does influence the result as expected.
oci raw-request –http-method GET –target-uri https://cell-1.queue.messaging.us-ashburn-1.oci.oraclecloud.com/20210201/queues/ocid1.queue.oc1.iad.amaaaaaa6sde7caak6kpobll3d7cjnd4ltmeeehmxa7w7puqhkyjzm3zuooq/messages?visibilityInSeconds=3599&limit=5
Using Sample Java Client for Interaction with Queue
The GitHub Repo https://github.com/oracle-devrel/oci-arch-queue-demo provides sources of a small Java application that interacts with an OCI Queue. I will try to use that application for publishing and later on consuming as well.
Clone Repository
Open Cloud Editor in OCI Console and edit file standalone-queue.sh: set queue ocid and compartment ocid and region and reference to OCI config file (/etc/oci/config on Cloud Shell).
Save changes.
Add execution privilege to shell script.
chmod +x standalone-queue.sh
The Java code needs to be compiled. The OCI Java SDK is preinstalled on Cloud Shell, so we do not need to download it first.
javac -cp .:$OCI_JAVA_SDK_FULL_JAR_LOCATION:$OCI_JAVA_SDK_LOCATION/third-party/lib/* SoloOCIQueueDemoTool.java
This fails with compilation errors – presumably because the OCI cloud shell does not have the latest Java SDK with support for the Queue Service.
I am a bit stuck.
Resources
My introductory article on OCI Queue – OCI Queue– simple, powerful, heavy duty reliable decoupling
OCI CLI Command reference for Queue – https://docs.oracle.com/en-us/iaas/tools/oci-cli/3.22.0/oci_cli_docs/cmdref/queue.html
OCI CLI in Cloud Shell – Interactive Mode – https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliusing_topic-Using_Interactive_Mode.htm
OCI CLI raw request https://docs.oracle.com/en-us/iaas/tools/oci-cli/3.6.2/oci_cli_docs/cmdref/raw-request.html
