It is the end of the year: time to look back, time to look forward. In the past year, I had a lot of extra time because of the Corona measures. I spent most of that time creating blog articles about AWS, starting with a series about the AWS shop example [1]. The series describes the implementation of the AWS part of a (virtual) caching machine that sends data to AWS. AWS accepts the data, decrypts it and puts it in a DynamoDB table. I put SNS topics in between to decouple these steps and make it easier to add new functionality later.
Should I have added an SQS queue after the SNS topic?
The end of the year is also re:Invent time [2], [3]. This year, I was able to join re:Invent because it was a virtual event. I learned a lot. And I got a little bit scared when I saw this sheet in the presentation about “Application Integration patterns”: should I have added queues after my SNS topics?
There are advantages and disadvantages. The advantage is that you can see which messages are waiting to be processed. On the other hand: when a message is delivered on the SNS topic and the SNS topic confirms that the message has been received, then the sender can be sure that the message will be stored somewhere until the message is delivered. And when the SNS topic delivers the message to Lambda, Lambda has its own internal queue where messages are stored until a Lambda function is able to process it.
I was still thinking about this, when I looked at the presentation about “Scalable serverless event-driven architecture with SNS, SQS & Lambda”. In that presentation, the following decision tree was shown:
The idea is that when there is a rate limit, the time that messages have to wait before they are processed can take (too) long. In that case, it is wise to add an SQS queue. When there is no rate limit, the SQS queue is not needed. In my case I didn’t use concurrency limitations to my Lambda function, my shop example therefore didn’t need an SQS queue.
Message ID
In the same presentation “Scalable serverless event-driven architecture with SNS, SQS & Lambda”, I saw this sheet:
When the FIFO SQS queue sends a message to Lambda, there is a MessageDeduplicationId in it, which should be used to prevent the double processing of a single message. This sounded familiar, I wrote about this in the blog about SNS duplicate messages [4]. In that blog, I described a solution where the message ID is created in the client.
On the moment I saw this sheet, I thought “creating a message ID in the client can cause issues when the message ID is the same. Though the chance of having two clients within one shop creating a message on exactly the same microsecond are not that high, it still would be better to prevent these potential problems by using some kind of unique ID that is given by the API gateway”. So I deployed my shop example again and looked into the data that is sent by the API gateway to the Accept Lambda function. There is no MessageDeduplicationId in this event, but I think that the requestId field in the requestContext part of the event would be a good candidate to use instead of the message_id I created myself in the client:
Next year…
The end of the year is also a good time to look forward: I wish you a happy 2021. Stay safe and let’s enjoy life. I’m looking forward to write more articles about nice features in AWS. See you in 2021!
Links
[1] Example application in AWS using Lambda: https://technology.amis.nl/2020/04/26/example-application-in-aws-using-lambda/
[2] re:Invent: https://reinvent.awsevents.com/
[3] YouTube channel from AWS: https://www.youtube.com/user/AmazonWebServices
[4] AWS Shop example: SNS duplicate messages: https://technology.amis.nl/2020/06/03/aws-shop-example-sns-duplicate-messages/
really nice article thanks