Some of the data used by Terraform to create cloud resources is sensitive and should not be stored in plain text in source code repositories. Examples are passwords, client secrets, API tokens. Even though the Terraform configurations are (infrastructure as) code – not every element can be treated as code.
Something similar applies to context dependent settings – values that are not necessarily sensitive but specific to an environment or situation or quite dynamic and therefore also not suitable for hard coding in Terraform configurations. In both these cases, we want values used by Terraform to (only) be available to Terraform at run time (when the plan is applied) in a secure way.
Sensitive values are typically stored in a Vault or Key/Secret Store and only retrieved and used by Terraform when it needs them for creation of the OCI resource. Context dependent values can also be stored as secrets in a vault or much more relaxed in a configuration management tool or simply in a JSON document (configuration as code); at apply time, Terraform will decide based on the environment to provision which of the multiple configuration files should be used to provide the configuration data.
Configuration data external to the Terraform configuration can still be applied to the configuration. This allows users not well versed in Terraform and without access to the environment where the configuration is applied to still influence its behavior – in a way that should be access controlled and carefully audited. It is not the only and certainly not always the best way to configure Terraform for different environments. It is an additional mechanism to leverage when appropriate. Alternatively, values for variables can be defined in context dependent files that are included with the Terraform configuration and at apply time using the proper value for the command line argument –var-file we can specify which of the files should be used to get the variable values from.
Edit: 3 Jan 2023 – based on feedback from Aris van Ommeren I have made a few edits to this article – to apply good practices regarding naming and directory structure and to make clear that the technique discussed in this article is only to complement the Terraform best practices for environment specific configuration in special cases and for a probably small number of properties.
This article is intended to demonstrate how Terraform is capable of retrieving data from external sources when executed and using that data to influence management of resources. A suggested good practice “for doing this is to separate each environment into its own folder or even an entirely separate repository”. The technique demonstrated in thus article can be used for complement this practice with some special externally managed inputs.
The figure shows some options for storing sensitive and dynamic configuration data. Sensitive values are stored as (part of) secrets in a Vault in OCI. Dynamic Configuration Data can be managed in JSON files – with the name of the file aligned with the environment or situation represented by the file. These files can be stored simply in a Git repository or in a Bucket on OCI Object Storage. Ideally at least in a location that is easily accessible by Terraform when it applies a configuration.
This approach allows us to use the same Terraform configuration for multiple environments – with each time the configuration data specific to that environment. And to store the Terraform definitions as regular code – because they do not have to contain sensitive values.
Of course storing the configuration and sensitive data in JSON documents and Secrets is only useful if at the time of application we can make Terraform leverage these values. That turns out to be not very hard.
Terraform variables are static: they are assigned using their default value (which cannot use function calls or references to other variables) or by overrides at startup time (from command line parameter or environment variable). To use dynamic values as desired here – Terraform offers locals. These can be referred to using local.<name of local> in the same way as variables are used: var.<name of variable>. Locals are available only in the module in which they are defined.
Locals can easily be assigned values using functions in Terraform – functions for example for looking up values from a Map, for turning a JSON formatted string into a Map and for reading the content from a file into a string. The OCI Terraform provider has resources for reading a JSON document in a Bucket on Object Storage and for reading a secret from an OCI Vault. Conveniently, Terraform allows us to merge multiple maps into a single one – so we do not have to worry about whether a setting came from a secret, from a file on GitHub or an object on Object Storage.
The next figure extends the previous one a little. It shows how through variables we specify the url for the file to consume configuration data from and the OCID for the secret to read from an OCI Vault and to process for sensitive values. The data retrieved from these stores is stored in Terraform maps in locals. Multiple maps can be merged into one “settings” map that can be used to derive all dynamic values from in resources and datasources.
This article discusses the implementation of what is shown in this figure in two steps: preparation (creating the secret and the json files) and application. The corresponding sources are found on GitHub.
Preparation
I will now demonstrate the use of a Terraform configuration to:
– create a secret on OCI Vault with a JSON payload and some settings
– create two JSON file on OCI Object Storage with some other settings (one for dev and one for prod)
This really is preparation for the actual Terraform execution that will leverage the values in this secret and these JSON files.
The three json files contain the dynamic respectively sensitive values that we will to retrieve later on. These are two be stored in two object in a bucket on OCI and in a secret in a vault.
Here is the variables.tf content:
it defines the tenancy_ocid, compartment_ocid and bucket_name to know where to create the bucket and which name to give it. It also defines the vault_ocid, secret_name and master_key_ocid that help create the secret in the right vault using the desired name and encrypted with the specified master key.
The contents from the three files is loaded into the string locals settings1 , settings2 and sensitivesettings.
The hard work is done in resources.tf:
The most interesting resource is shown in the rectangle: the creation of the secret in the vault – its content derived from the string local.sensitivesettings.
The result of running Terraform on this configuration can be checked in the console:
and
Application
With the secret in the vault and configuration data files in the Object Storage bucket we are ready to proceed and use these externally managed, dynamic and sensitive values from a Terraform configuration.
File variables.tf defines the identifiers for tenancy, compartment, bucket name, file and secret. It uses data sources to retrieve the two files from the bucket and the secret from the vault. It converts the JSON content to maps and then merges the maps together to form a single map called settings.
The file main.tf is empty for now – no actual resources are managed by this Terraform configuration. File outputs.tf contains (debug) outputs for the settings from secret and objects/files. More useful: it shows the actual value for apiASecureToken (a sensitive value, retrieved from a secret) and the api_a_endpoint – retrieved from whichever configuration data file is the current one: dev or prod. The variable environment in variables.tf is used to determine the environment; this value can also be provided when terraform is executed using the command line parameter –var environment=dev or with an environment variable TF_VAR_environment.
As mentioned earlier: it is a recommended practice for Terraform to have environment specific directories with each their own main.tf and variables.tf. Additionally, environment specific values for variables can be passed on the command line, like so: –var-file=”development.tfvars” or –var-file=”production.tfvars”. The technique demonstrated in this article is a complement to this foundation – only used to allow manipulation of values outside the immediate Terraform context if so required.
With environment set to prod, running terraform for this configuration results in:
When I change the value for environment in file variables.tf to dev:
The result of running terraform changes as expected:
The value for the endpoint for API A is now derived from the development environment specific configuration data file.
After creating a new version of the secret:
And waiting for the new secret version to be saved:
I can run terraform again. I would expect that the updated value for the apiASecureToken is retrieved by Terraform from the new current version of the sec ret:
I believe this proves that I can have a Terraform configuration – pure code – that does not contain environment specific values or sensitive values and provision resources with sensitive properties and content dependent settings. Retrieving sensitive values and configuration data at runtime – from a JSON file in Object Storage or indeed anywhere a HTTP GET request can retrieve it and from a secret in a Vault – is fairly straightforward and allows independent management of the Terraform definition (that is stable across environments) and values that are bound to context.
Note this warning from Martin Bach’s article: “Whilst this is without a doubt a great step towards safer code management, there is still an unsolved issue with Terraform: the state file is considered sensitive information by HashiCorp. When using the local backend for storing the state file-the default-passwords and other sensitive information are stored in clear text in a JSON file, Access to the state file should therefore be very much limited.
Edit: based on feedback from Aris van Ommeren, I have made several updates to the article – and refactored the files and folder structure of the sources. I have not taken over the suggestion to create a dev/main.tf and prod/main.tf because the message of the article was to show how configuration data external to the Terraform configuration can still be applied to the configuration. This allows users not well versed in Terraform and without access to the environment where the configuration is applied to still influence its behavior – in a way that should be access controlled and carefully audited.
Resources
Sources in GitHub for this article: https://github.com/lucasjellema/oci-terraform-composites/tree/main/dynamic-and-sensitive-data
My article Terraform – injecting local and remote Configuration Data (read dynamic data from JSON documents – local or remote)
Merge Maps in Terraform – https://developer.hashicorp.com/terraform/language/functions/merge; LookUp – https://developer.hashicorp.com/terraform/language/functions/lookup;
my GitHub Repo with OCI Terraform definitions – https://github.com/lucasjellema/oci-terraform-composites
Blog article by Martin Bach – Using OCI Vault Secrets for Terraform resources – https://blogs.oracle.com/developers/post/using-oci-vault-secrets-for-terraform-resources
HashiCorp Docs – Standard Module Structure – https://developer.hashicorp.com/terraform/language/modules/develop/structure
HashiCorp Docs – Variables – https://developer.hashicorp.com/terraform/language/values/variables
Structuring HashiCorp Terraform Configuration for Production XANDER GRZYWINSKI – https://www.hashicorp.com/blog/structuring-hashicorp-terraform-configuration-for-production
