First steps in rotating secrets in AWS Secrets Manager

Introduction

AWS Secrets Manager was released in 2018. It is a nice replacement for secret strings in the SSM Parameter Store. With Secrets Manager it is possible to automate the rotation of secrets. AWS helps us with out-of-the-box Lambda functions to rotate the passwords for MariaDB, MySQL, Oracle, PostgreSQL and Microsoft SQL Server users. You can also write your own Lambda function to rotate the password [1]. In this blog I will show you the two variants used by the default Lambda functions in AWS: rotating single users and rotating alternating users.

How to follow along for single users

I wrote a CloudFormation template to deploy this solution. You can find it in my Github repository [2]. The deployment takes about 15 minutes, it will deploy a VPC with two public subnets. A serverless Aurora MySQL database server is deployed in these subnets. It is not possible to reach this database from the public internet, I therefore also deployed a test EC2 instance to demonstrate a connection to the database by using the user-id and password from AWS Secrets Manager. You can log on to this test instance via SSM Session Manager.

The template will also deploy the Lambda function to rotate the main database user. This Lambda must be able to connect to the database, it has therefore a network interface card (NIC) in the VPC. NICs that are attached to Lambda functions unfortunately cannot access the public internet directly. We therefore need a VPC Endpoint to make a connection from the Lambda function to AWS Secrets Manager.

In a production environment you would probably connect the Lambda function and the database connections to a private network. For this demo this will do.

The deployment in a diagram:

To make it possible to debug the deployment, I added VPC FlowLogs to both subnets. You can find these logs in CloudWatch.

Contents of the secret

When the template is deployed we can look at the secret that is created. It is called DatabaseMainAdminUser. The rotation Lambda requires the secret to have a certain format [3], for MySQL this format is:

 { 
    "engine": "mysql", 
    "host": "<instance host name/resolvable DNS name>", 
    "username": "<username>", 
    "password": "<password>", 
    "dbname": "<database name. If not specified, defaults to None>",
    "port": "<TCP port number. If not specified, defaults to 3306>" 
} 

CloudFormation can generate the password in the secret on initialization. We need a password before the databasecluster is created, the database properties will be added later. By specifying ExcludePunctuation we can be sure that the databaseserver will accept the new password.

  DatabaseMainAdminSecret:
    Type: AWS::SecretsManager::Secret
    UpdateReplacePolicy: Delete
    DeletionPolicy: Delete
    Properties:
      Name: DatabaseMainAdminUser
      Description: "Main admin user for the database"
      GenerateSecretString:
        SecretStringTemplate: !Sub |
            { 
               "engine": "mysql",
               "username": "${DatabaseMainAdminUser}"
            }
        GenerateStringKey: "password"
        PasswordLength: 32
        ExcludePunctuation: True

SecretTargetAttachment

For secret rotation the secret has to contain the information about the database. This is done via a resource with the type SecretTargetAttachment:

  DatabaseMainAdminSecretRDSInstanceAttachment:
    Type: AWS::SecretsManager::SecretTargetAttachment
    Properties:
      SecretId: !Ref DatabaseMainAdminSecret
      TargetId: !Ref DatabaseCluster
      TargetType: AWS::RDS::DBCluster

After deployment the content of the secret looks like this in the AWS Console:

Schedule

When the secret is attached to the database (and the database is running) then the rotation schedule can be deployed. When you use the AWS managed Lambda functions [4], the CloudFormation template has to start with the line “Transform: AWS::SecretsManager-2020-07-23”, the AWS::SecretsManager::RotationSchedule will then be implemented with a nested stack.

The RotateImmediatelyOnUpdate will force the Lambda function to rotate directly after the deployment of this resource. This can also be useful to test the rotation directly after deployment, though CloudFormation will not wait for the rotation to be completed.

You can find the rotation engine types in the CloudFormation documentation [5]. The RotationLambdaName can be anything you like, as long as this name is not in use by other Lambda functions. You cannot share one AWS managed rotation Lambda function with multiple secrets.

I put all the characters that are part of ExcludePunctuation in ExcludeCharacters, except for “–“ and “_”. Unfortunately the Lambda function doesn’t understand the ExcludePunctuation parameter. The GetRandomPassword call will always put at least one character of the different groups in the new password. You will get the error “[ERROR] InvalidParameterException: An error occurred (InvalidParameterException) when calling the GetRandomPassword operation: All characters of the desired type have been excluded” when you try to put all punctuation characters in the exclusion list.

  DatabaseMainAdminSecretRotationSchedule:
    DependsOn: 
      - SecretRDSInstanceAttachment
      - DatabaseInstance
    Type: AWS::SecretsManager::RotationSchedule
    Properties:
      SecretId: !Ref DatabaseMainAdminSecret
      RotateImmediatelyOnUpdate: True
      RotationRules:
         AutomaticallyAfterDays: 1
      HostedRotationLambda:
         RotationType: MySQLSingleUser
         RotationLambdaName: SecretsManagerRotation
         VpcSecurityGroupIds: !Ref SecurityGroupAutoRotationSecretsManager
         VpcSubnetIds: !Sub "${PublicSubnetAZa},${PublicSubnetAZb}"
         ExcludeCharacters: '!"#$%&()*+,./:;<=>?@[\]^`{|}~'''

Demo single user

When the deployment is done you can log on to the Test Instance with SSM Session Manager. You can then use the mysql command to log on to the database:

You can now rotate the password via the command line interface and see that the password isn’t valid anymore after rotating the secret. This command to do this is: aws secretsmanager rotate-secret –secret-id DatabaseMainAdminUser

Alternating users

This works nice for most situations, but what if you want to have a valid user with a valid password on any given time? Let’s assume we have a website where the application first gets the password from Secrets Manager and before it can use the password Secrets Manager rotates the password. We then cannot log on to the database. It is possible to get the new password from Secrets Manager and then try again, but this might take too long for certain situations.

One of the ways to solve this issue is to use alternating users. AWS will create a second user with a new name and a new password and the same permissions as the original user. The first user is still valid and then there is more time to gradually implement the new user id and password.

The third rotation Secrets Manager will just rotate the password of the first user, show the username of the first user and the second user isn’t touched. The rotation after that, the second user will just get a new password as well. The documentation has nice images to show how this works [6], in the demo I will also show some screen prints from my deployment.

How to follow along for alternating users

Let’s see how this works: delete the stack for single users and deploy the stack for alternating users (Database-MySQLMultiUser.yml). The AWS architecture is more or less the same. The database main admin user secret is unchanged, I added an extra secret with less privileges with the name “WebsiteUser”. The secret of this user uses alternating users rotation. The name in the database is “webuser”. “websiteuser” wasn’t possible because the length of “websiteuser_clone” would be longer than the maximum of 16 characters.

In the deployment of the EC2 instance I added the creation of this user in the database. There’s a new table in the database (“prices”), the website user can only read from this table. I added some content to this table as well. By doing this in the EC2 instance, the database must be deployed before the EC2 instance can start. The deployment of the stack will take more time because of this.

Contents of the secret

In CloudFormation the contents of the secret are almost the same. The only difference is that we have to add a secret of a high privileged account to the json. This secret will be used to clone the user and to update the passwords.

Schedule

For the purpose of this demo, I disabled the RotateImmediatelyOnUpdate parameter. In this way I can show what the situation is before the first rotation. After that, we can see the effect of that first rotation. In a production environment I would recommend to leave this on. The effects of the rotation are visible directly after the deployment.

The database main admin user secret needed a policy to pass the value of the secret to the rotation Lambda function of the non-privileged user:

  AllowUseOfDatabaseMainAdminSecretAtRotationOfWebsiteUserSecret:
    Type: AWS::SecretsManager::ResourcePolicy
    UpdateReplacePolicy: Delete
    DeletionPolicy: Delete
    Properties:
      BlockPublicPolicy: False
      SecretId: !Ref DatabaseMainAdminSecret
      ResourcePolicy:
        Version: "2012-10-17"
        Statement:
        - Effect: "Allow"
          Action: "secretsmanager:GetSecretValue"
          Resource: "*"
          Principal: "*"
          Condition:
            "ArnLike": 
              "aws:PrincipalArn": !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/${AWS::StackName}*SecretsManager*"

This resource policy is needed because the policy of the Lambda function that rotates the multi user secret is:

It took me some time to figure out why this policy couldn’t get the secret value of the DatabaseMainAdminSecret. My original assumption was that the condition “secretsmanager:resource/AllowRotationLambdaArn”: <ARN of the Lambda function> would be true during the whole execution of the Lambda function, for all secrets (because of the star in “Resource”: “arn:aws:secretsmanager:eu-west-1:040909972200:secret:*”). This turned out to be incorrect: the policy is checked for every call for every individual secret. This means that when the secret of the DatabaseMainAdminUser is retrieved, the AllowRotationLambdaArn of this secret is the rotation Lambda of the DatabaseMainAdminUser. It is not the ARN of the Lambda function that is asking for the secret. Because of this, the request for the secret value of DatabaseMainAdminUser fails.

The condition in the resource policy of the DatabaseMainAdminUser might confuse you:

          [...]
          Condition:
            "ArnLike": 
              "aws:PrincipalArn": !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/${AWS::StackName}*SecretsManager*"

This part is needed, because the Lambda function has a role with a dynamic name that starts with the stack name and has the word SecretsManager in it. The name of the Lambda function itself is not part of this IAM role. It is also not clear how to make a difference between the role for the single user lambda function and the role for the multi user lambda function. This makes the condition of the main database admin user very generic: when you have multiple secrets and multiple Lambda functions for the rotation, this condition will match all the IAM roles of all the rotation Lambda functions.

Unfortunately the name of the IAM role is not passed back as a result of the RotationSchedule. I created an issue to ask to change this [7]. When this would be implemented, the resource policy can be changed into something better readable and better least privileged like:

          [...]
          Condition:
            "ArnLike": 
              "aws:PrincipalArn": 
                   - !GetAtt WebsiteUserSecretRotationSchedule.LambdaIAMRole
                   - !GetAtt DatabaseMainAdminUserSecretRotationSchedule.LambdaIAMRole

With the implementation as is, the only way to get the ARN of the role that is attached to the created Lambda function is to create a custom resource Lambda function. In this way the CloudFormation template can get the information via a GetFunctionConfiguration call from this custom resource Lambda function. I implemented this in the second version of this template ( Database-MySQLMultiUser-with-custom-resource.yml).

Demo alternating users

First, let’s look at the database users that are present in the database. Then get the password of the websiteuser and check that the website user doesn’t have the permission to change the prices table. The commands are:

aws secretsmanager get-secret-value --secret-id DatabaseMainAdminUser

mysql -u"databaseadmin" -p"<password from secrets manager>" -h"<host from secrets manager>"
SELECT user FROM mysql.user;
exit

aws secretsmanager get-secret-value --secret-id WebsiteUser

mysql -u"webuser" -p"<password from secrets manager>" -h"<host from secrets manager>"
SELECT * FROM demodb.prices;
INSERT INTO demodb.prices VALUES (3,'Errors', 0);
SELECT user FROM mysql.user;
exit

Let’s now rotate the secret of the webuser (with the command aws secretsmanager rotate-secret –secret-id WebsiteUser ) and look again:

The permissions are correct and both users can be used. Now do another rotation: it just changes the password of the original user. No new users are added to the database:

The password of the original user is changed. There are no new users added to the database server, everything works as expected.

Conclusion

I think that AWS did a great job with this service to rotate secrets in an automated way. In most cases the single user rotation will do. When you use the multi user rotation you might have to add a custom resource to get least privileged access to the high privileged database user.

Links

[1] Write your own Lambda Function: https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotate-secrets_turn-on-for-other.html

[2] Github repository for this blog: https://github.com/FrederiqueRetsema/Blogs-2023

[3] Documentation with requirements for the structure of the secrets: https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_available-rotation-templates.html

[4] Github repository for Lambda functions for AWS Secrets Manager: https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master

[5] CloudFormation documentation for rotation engine types: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-secretsmanager-rotationschedule-hostedrotationlambda.html#cfn-secretsmanager-rotationschedule-hostedrotationlambda-rotationtype

[6] Documentation on alternating users: https://docs.aws.amazon.com/secretsmanager/latest/userguide/getting-started.html#rotating-secrets-two-users

[7] Issue in the github repository to ask for a return value of the ARN of the Lambda role for the resource RotationSchedule type: https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/issues/115

Image Top Secret: Cristina Canciani at Italian Wikipedia, Public domain, via Wikimedia Commons