Introduction
I used LastPass for several years, and it helped me a lot to access my passwords and secrets on any moment, from multiple devices, in a secure way. But to be honest, I freaked out when I heard about the last hack [1]. I changed my most important passwords, but I still kept thinking “what will happen when attackers are successful in getting my passwords?”.
I discussed this with some of my colleagues, and one of them uses Bitwarden on a self-hosted environment. I thought “that’s a great idea: let’s use self-hosted Bitwarden in AWS and then use AWS Security Groups to only allow IP traffic from my own devices”.
In this blog, I will explain how you can use this in your environment as well. You can find the CloudFormation template and supporting scripts in my gitlab repository [2]. But before I go into the AWS part of this solution, let’s first talk about Bitwarden.
Bitwarden
SaaS
Bitwarden is a password manager. It can store passwords in vaults. You can use the central vault from bitwarden, this is a SaaS solution with all the benefits that SaaS solutions have. There is a free, personal, version and when you pay $10 per year you can use extra functionality like Yubikeys.
For $40 per year you can use a Family license and share passwords with at most 6 people. For organizations you pay per user ($3 per user per month for the Teams Organization and $5 per user per month for Enterprise Organizations). With Enterprise Organizations you can integrate with SSO.
Desktop applications, browser plugins and mobile phone apps
You can use desktop applications and plugins to keep track of passwords. The advantage of plugins in the browser is that Bitwarden will detect when you are trying to log in to one of the sites for which you stored your password. When you click on the password then your credentials will be entered automatically. Bitwarden also comes with apps: both Android and iPhone are supported, all these applications work more or less the same.
There are four different kind of objects within Bitwarden: passwords, cards, identities and secure notes. Cards are credit cards, Bitwarden supports all the fields that you expect for a credit card (like expiration month, security code etc). Identities are for example first name, last name, company name etc. Secure notes is free text.
Objects can be put in folders. Unfortunately, you cannot nest folders and one object can only be put in one folder.
The applications all store their data locally and all fields are encrypted. The advantage is that you don’t have to connect to the central Bitwarden vault to use a password. There is a button to synchronize with the vault to get new passwords. When you add a new password, the password is always stored in the central vault directly.
Extra functionality
You can add “custom fields” to any of these objects: with custom fields you can connect a specific field on the web page with data that should be filled in on that page. This could be a license key id for example.
Another option is to mark the object as “favorite” or enforce entering the master password before the record is shown.
It’s also possible to add two step login codes, for example the codes that Authy generates. This might be useful when either you have a lot of websites using these codes or when you share one account with a two step login code between multiple people.
Self-hosted environment
The functionality for self-hosted Bitwarden is the same as the SaaS solution. Both Windows and Linux are supported. Bitwarden uses Docker Compose to install multiple containers on the server. One of those containers is Microsoft SQL Server.
In the near future Bitwarden wants to distribute the software in one single container [3]. This container will then connect to a database outside that container. There are several advantages to this way of working. Unfortunately this solution is currently only available in beta and production passwords shouldn’t be stored in this solution (yet).
Documentation
I’m only scratching the surface in explaining what Bitwarden is able to do. The documentation of Bitwarden is pretty good, you might want to look at it yourself [4]. For some features they also have videos[5].
Deployment in AWS
Overview
I wrote a CloudFormation template to deploy all necessary resources to AWS. You need a Route 53 hosted domain to use the template and the AWS Command Line Interface should be installed. My scripts for Linux and Mac assume that jq is installed as well.
It will take about 7 minutes to deploy all resources to AWS. When all the resources are deployed, you can log on to the virtual machine (using Systems Manager) and then install Bitwarden on the Virtual Machine.
Bitwarden will (by default) request a certificate from Let’s Encrypt. You will be prompted for an email address that will be used to send you expiration reminders. The bitwarden installation script will request and install the certificate automatically. Installing Bitwarden on the virtual machine can be done within 15 minutes.
Step Functions
The main idea behind the solution is that you only allow access to the virtual machine when you need it.
The Lambda function will first delete all the current ingress ports from the two Security Groups (one for IPv4 and one for IPv6). It will then, based on the parameters that are given to the Lambda function, open the ports for the specified IP addresses. This also makes it possible to use the same Lambda function for both opening and closing the vault: the parameters for opening the vault are retrieved from the parameters of the Step Function, the parameters for closing the vault don’t contain IP addresses and are specified within the Step Function itself.
Within the step function I did a simple calculation for determining how long the Step Function should wait. This is done by the step “Determine Seconds to wait”:
{
"DetermineSecondsToWait": {
"Type": "Pass",
"Parameters": {
"secondsToWaitBeforeWarningToSNS.$": "States.MathAdd($.OpenDurationSeconds, -300)"
},
"Next": "Wait for raising alarm"
}
}
By adding dot-dollar to the name of the parameter, Step Function will use the variable from the parameters of the step function instead of using the result of the previous step. We don’t need the Lambda function to just pass a variable from input to output where this variable is irrelevant for the Lambda function itself.
States.MathAdd is used to subtract the 5 minutes from the total duration time in seconds. Unfortunately there is just an Add (and a randomizer) to do calculations with. I hope that multiplication is possible in the future versions of Step Functions as well: I’d like to give the duration in minutes instead of in seconds.
SNS is called from the step function to warn that the vault will be closed in five minutes. You might wounder why there is no call to SNS to inform the user that the vault is opened or closed. The reason for this is that you also want to be informed when the ingress ports are changed by other means, for example when someone uses the AWS Console to add ingress ports instead of the Step Function. That is done using Event Bridge.
EventBridge
There are several checks for changes in the environment:
- Check for adding or changing ingress port rules in security groups
- Check for changes in network adaptors of Virtual Machines
- Check for changes in network adaptors of Virtual Machines
- Check for failed snapshots
The first two checks register changes that are made to the inbound traffic to the Virtual Machine. By using SMS, the owner of the Bitwarden environment will be informed immediately. It is also useful to know that opening or closing the vault by the Step Function was successful.
When you don’t get an SMS, check that you are out of the SNS sandbox. By default only SMS messages to known mobile numbers are allowed. In a new account, maximum $1 per month can be spent on SMS traffic. When you request a change on this last soft limit, you have to answer some questions to show that you will not abuse SMS for sending spam messages. When you are out of the SNS sandbox you might have to change the setting in SNS > Text Messaging (SMS) > Account spend limit.
The checks on connecting to the Virtual Machine and failed snapshots are less urgent. They are sent to an SNS topic that sends the message to an email address.
Alarms
Talking about checks: the bitwarden Virtual Machine might run perfectly well – or something might go wrong. There are four checks to inform the administrator that something within the Virtual Machine is wrong:
- CPU: when more than 80% CPU is used for some time, an alarm will be raised
- Memory: when more than 80% memory is used for some time, an alarm will be raised
- Disk usage of the root disk: when more than 80% of the disk space of the root disk is used, an alarm will be raised
- Disk usage of the bitwarden disk (connected to /opt/bitwarden): when more than 80% of the disk space of the Bitwarden software and data is used, then an alarm will be raised.
All these alarms will be sent to the mail SNS topic.
Flow Logs
It’s good to know that FlowLogs are enabled, all data is sent to a log group. When you need help querying these logs, there is nice documentation with examples [6].
Updates
Updates are done within the Virtual Machine, using cron. Both Bitwarden and the yum packages are updated automatically.
Snapshots
Both the root disk and the bitwarden disk are snapshotted every day. Snapshots are (by default, see parameters of the CloudFormation template) stored for one month. There is a quite convenient way to create snapshots of EBS Volumes: it is called Data Lifecycle Management (DLM).
Based on the tags of the instance, DLM will snapshot all the volumes that are connected to those instances. Via a cron expression DLM will determine when to create a snapshot. Via Retain Rules it is determined how long snapshots will be kept.
Using a dedicated user for Bitwarden
It’s good to use Least Privileged where possible. I added two policies to help you implement Least Privileged access:
– bitwarden-cli-role-policy: use this one when you want to use the command line interface with least privileged access to start the shell scripts in this repository
– bitwarden-cli-cloud-shell-role-policy: use this one when you want to use the shell script from AWS CloudShell (when you don’t see this role then update the stack and change the UseCloudShellParameter parameter to True)
When you add the second policy you also allow AWS Console access (via the browser). When you use your AWS app to get access to the console via this user, the only thing you are allowed to do is to start the Step Function. You will not need this very often – as discussed before most passwords will be present on your mobile phone via the bitwarden app already.
Business solution
This solution works for one person, using Bitwarden individually. When you want to use Bitwarden as an organization, you can change a few parts to make it work: currently the ingress port definition contain a very global description:
You could add a tag to the JSON of the Step function and add that tag to the description of the inbound rule. When the Lambda function removes the rules, it should only remove the roles with the right description (tag). When you implement this, you can have (by default) 60 inbound rules per security group, this is a soft limit and it can be increased on request. Multiple step functions can run at the same time, each running for a different user, all controlling different ingress rules of the same Security Group.
Costs
Currently the costs for this solution might be higher than expected. Though I used a t3.medium and the Virtual Machine is hardly used, the costs are about $1.10 for EC2 per day. This is roughly $400 per year, which is much more than I expected.
The costs of SMS can also be high: when you have three IP addresses that should be whitelisted, then you will get 2 times 3 messages to open the vault, one message to warn you that the vault will be closed and then 2 times 3 messages to close the vault. At the current rate of $0.1189 per message (in the Netherlands [7]). This would be around $1 (including tax) for each time you open the vault.
When the one-container solution comes available, then it would be great to run Bitwarden on ECS Fargate, using a serverless Aurora MySQL database that can scale to zero when it isn’t used. The containers can also be stopped outside business hours to save some costs. It might even be acceptable to start the container when the vault is opened and stop the container when the vault is closed, this would save a lot of costs.
Conclusion
I can sleep better now I know that my vault is not accessible from the public internet when I’m not using it. For attackers, my vault is not that interesting – it is more likely that they will attack the SaaS solution for Bitwarden/Lastpass/… than that they would try to hack into my vault for just one person.
Currently, the costs to run Bitwarden on a virtual machine in your own environment are high. I’m looking forward to the moment that Bitwarden will have a production-ready release for their one-containter solution.
Links
[1] Information about LastPass security breaches: https://www.spiceworks.com/it-security/data-security/news/lastpass-second-data-breach/
[2] Github repository: https://github.com/FrederiqueRetsema/aws-bitwarden.git
[3] Bitwarden running on one container, in beta: https://bitwarden.com/blog/new-deployment-option-for-self-hosting-bitwarden/
[4] Bitwarden documentation: https://bitwarden.com/help/
[5] Bitwarden videos: https://bitwarden.com/help/getting-started-videos/
[6] Searching in Flow Logs: https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-cwl.html#view-flow-log-records-cwl
[7] Pricing for SMS messages per region: https://aws.amazon.com/sns/sms-pricing/
