Writing your own Lambda function for secret rotation

Introduction

A few weeks ago I published a blog about secret rotation [1], I used the default Lambda functions for single user and multi users that are maintained by AWS. In this blog, I will go one step further and change one of those Lambda functions for my own way of working.

I want to change the Multi User Lambda function in a way that not just the password is rotated, but that I also change the username. A part of the username is fixed (f.e. webuser-) and then some randomized characters are added. Even though the complexity of the password is already high with 32 characters, by adding some randomized characters to the username it is more difficult to do brute force attacks on usernames that you don’t know. This is also true when you might know or guess the prefix of the username.

My Lambda function has the following environment variables:

Based on the defaults, you will get usernames like ‘user-gW7nm’. The maximum length of a username is 32 characters. In the standard AWS Lambda functions this is 16 characters.

The secret has the same format as the AWS Multi User Lambda function:

{
   "engine": "mysql",
   "host":  "<instance host name/resolvable DNS name>",
   "username": "<username>",
   "password": "<password>",
   "dbname": "<database name. If not specified, defaults to None>",
   "port": "<TCP port number, if not specified, defaults to 3306>"
   "masterarn": "arn of a secret of a database user that has CRUD (Create, Read, Update, Delete) permissions on the database"
}

I’m using the standard AWS Single-User Key rotation for the main database user with Create, Read, Update and Delete permissions.

How to start

AWS did a great job by making the default software rotation open source [2], so you can take the software from AWS and add your own code to that. I used the same coding style as the original Lambda functions.

I had some issues to get this working: when you use

pip install -r .\Lambdas\requirements.txt --target .\Lambdas 

in Windows and then deploy the software you will get the error “AttributeError: module ‘cryptography.hazmat.bindings._rust.openssl’ has no attribute ‘hashes’”.

When I read that I needed Windows Subsystem for Linux (WSL) because the Lambda function is using Linux, I tried this command again from WSL. I then got the error “RuntimeError: ‘cryptography’ package is required for sha256_password or caching_sha2_password auth methods“, even though the cryptography library was part of the zip file. Changing the environment variables of the Lambda function to search for the Python packages in the right place (/var/task) didn’t solve this issue.

In the end it turned out that I had to give extra parameters to the pip command to get Python packages that are compatible with Lambda [3]. In my case (see the repository [4], standing in the secretsmanager directory) the following command did the trick:

pip install \
 --platform manylinux2014_x86_64 \
 --target=./Lambdas \
 --implementation cp \
 --python-version 3.10 \
 --only-binary=:all: \
 --upgrade \
 --requirement ./Lambdas/requirements.txt

After downloading the packages, the Lambda zip file can be build and put in an S3 bucket with:

aws cloudformation package \
   --template-file Database-CustomRotationLambda.yml \
   --output-template-file Database-CustomRotationLambda-Deploy.yml \
   --s3-bucket <name of your bucket>

The command for the deployment is:

aws cloudformation deploy \
  --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND \
  --template-file Database-CustomRotationLambda-Deploy.yml \
  --stack-name Database-CustomRotationLambda \
  --s3-bucket <name of your bucket>

These commands are (because of the –platform parameter) also possible from Windows without using WSL.

You can use test the Lambda function with aws secretsmanager get-secret-value and likewise with the commands rotate-secret, update-secret-version-stage (and even delete-secret) in the EC2 that is deployed by CloudFormation. I will talk more about these two last commands later in this blog.

How AWS rotates a secret

In the AWS code, you will find several steps in the rotation of the secret. AWS Secrets Manager will call the Lambda function with Step as parameter. There are four steps: createSecret, setSecret, testSecret and finishSecret.

createSecret

A new version of the secret is created in this step. You will have to give this new version of the secret the version-stage AWSPENDING. When software asks for the value of a secret, it will get the AWSCURRENT version unless otherwise specified.

I changed this step so that it didn’t just generate a new password, but also a new username.

After putting the result in the AWSPENDING version of the secret, the Lambda function stops.

setSecret

When the previous step was successful, AWS Secretsmanager will then call the Lambda again, but now with step name setSecret. In this step, the database is called to accept the new values for username and password. There are also some security measures, for example to test if the current version of the secret is able to log on to the database. When this isn’t the case, this step will fail: someone might try to set a new password without having a valid password for the database in the first place.

After the checks, all permissions and roles of the current user are copied to the new user. When I tested this code, it turned out that the AWS code will not set the default role of the new user, even though the original user had a default role. I added some code to fix this. The code will use the last role it finds in the list of permissions of the original user to create the default role for the new user.

testSecret

The next step is to test the new secret. The default Lambda function from AWS just asks for the time (SELECT NOW()), but it is better to add a test that will get some data from the tables in the database.

I choose to count the number of rows in my prices table. This also verifies if the new user has the read permissions that I expect it to have.

finishSecret

In this last step, the secret is activated: the AWSPENDING version will become AWSCURRENT. Though it is not visible in the code, the AWSCURRENT version before the rotation started is still available via version AWSPREVIOUS.

In this step I also added the removal of non-current users from the database. The implicit assumption here is that any user that starts with my prefix is maintained by this rotation Lambda function.

Testing the code

When you change the Lambda function, you might fill the AWSPENDING version of the secret with an incorrect value by accident. You can look at this value by using the command

aws secretsmanager get-secret-value --secret-id WebsiteUser --version-stage AWSPENDING

When you see the issue and then want to delete the AWSPENDING version of the secret to stop the retries on the testSecret phase of your Lambda function, then don’t do this with the delete-secret action of secretsmanager. Even though it might look that it is possible to delete just the AWSPENDING and not the AWSCURRENT version-stage of the secret by giving –secret-id the ARN of the AWSPENDING secret, you will delete the complete secret. This includes the AWSCURRENT and AWSPREVIOUS versions: the ARN of the secret is the same for all these versions.

The correct way to delete only the AWSPENDING value of the secret is to use the update-secret-version-stage command with parameters –secret-id WebsiteUser –version-stage AWSPENDING –remove-from-version-id <VersionId>. The VersionId in the secret is different for the three versions AWSCURRENT, AWSPENDING and AWSPREVIOUS. You can check this with the get-secret-value command.

When you replace the Lambda function with a new version by using aws cloudformation package and aws cloudformation deploy, you can then use rotate-secret to call the new version of your Lambda function and try again.

The Lambda function itself will send all its output to CloudWatch.

Conclusion

It isn’t that hard to add a new secret rotation Lambda function when you use the github code from AWS. The hardest part was to use the correct pip statement. It also took me some time to figure out how to debug the Lambda function, I hope this blog will save you that time when you face similar issues.

Links

[1] Previous blog: https://technology.amis.nl/aws/first-steps-in-rotating-secrets-in-aws-secrets-manager/

[2] Open source repository with secret rotation Lambda functions: https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas

[3] AWS Documentation on how to package Python Lambda functions: https://docs.aws.amazon.com/lambda/latest/dg/python-package.html

[4] Github repository with example code: https://github.com/FrederiqueRetsema/Blogs-2023

Photo keys: Keys free photo by Henkster on freeimages.com (url: https://www.freeimages.com/photo/keys-1196217 )