Very first steps in Oracle Cloud Infrastructure as Code with Terraform

Resources in Oracle Cloud Infrastructure can be managed through the console – the browser based UI. That works great but requires manual steps – no automation – that take time and easily go wrong (and become extremely boring over time). Additionally, it is a rather individualistic way of working that does not lend itself to a lot of collaboration. Resources can also be managed through the CLI and the REST APIs. In scripts, I can program the operations to execute. It requires programming – which can be quite time consuming – and it is imperative rather than declarative that requires me to explicitly instruct everything I want OCI to do.

A final option – and the subject of this article – is using a declarative style in which I describe the situation I would like my OCI resources to be in and leave it to Terraform to take my definition of this desired state and make it happen. Terraform compares what I want there to be with what is currently there and performs the actions required to close the gap. Existing resources that fully comply with the declared, desired state are untouched, resources that deviate in their definition are updated and declared resources that do not exist at all are created by Terraform. In fairly simple scripts – that are version controlled and can easily be collaborated on – I express my intent that are automatically turned into reality. That at least is what it says on the box.

I recently started out with Terraform and the Terraform provider for OCI. To see for myself whether it works as advertised. And how I can get it to work as advertised. If you already played with Terraform for OCI, you probably went through the same phases I experienced. Read and recognize. If you are contemplating using Terraform with OCI, read and learn and save yourself hours or more in work and frustration. I got it to work, but it was not as straightforward as I expected. Of course it did not help that when I started out with Terraform for OCI, I had zero experience with Terraform.

Some of the questions I struggled with for some time are listed here. They are all answered in this article.

• Where do I run Terraform to operate on my OCI tenancy?
• How to install the Terraform Provider for OCI?
• How to configure the provider for my tenancy?
• How to define the value of a variable?
• Why do I get warnings when trying to replay examples I find on the Internet?
• How to have the provider process multiple files?
• How to define a resource using properties of another resource?
• How can I remove the resources that were created through Terraform?

For this article I worked with Terraform 0.12 on Linux (Ubuntu 18.04).

Set Up Environment

Terraform is a command line tool that you install locally. Locally can be anywhere really, including a VM on OCI. You most certainly do not have to run Terraform inside OCI in order to manage resources on OCI.

You can go to Installing Terraform and follow the instructions. I have used a project on GitHub that provides a script for automated installation of Terraform. This makes it very easy to set up the latest version of Terraform in my local Linux environment:

git clone https://github.com/robertpeteuil/terraform-installer

Run the Terraform installer to install Terraform

./terraform-installer/terraform-install.sh

Test the currently installed version of Terraform. This will be at least 0.12.

terraform version

At this point, you can start to download or create Terraform configuration files that define resources and have Terraform manage those resources. This is currently a vanilla Terraform environment, without the provider for OCI.

So how do I install this provider for OCI, in order to make the environment OCI aware? The answer turns out to be: you do not (have to). When you refer to the oci provider in your configuration files, Terraform will recognize it and automatically install the provider for you.

Let me make this a little bit more tangible:

I have created a new directory with a file called variables.tf that contained this definition:

provider "oci" {<br>}

Then I initialized the Terraform provider from within this directory:

terraform init

At this point, Terraform checks all files in the working directory and scans them for the providers that are listed. In this case, the oci provider is encountered. Terraform out of the box does not contain that provider, so it installs it for me. At this point, the OCI Provider is part of my environment.

 

This answered one of my questions. Another one is: how does Terraform know which files it should even look at when I run this init command – or one of the other commands such as plan, apply and destroy? Well, that is simple: it looks at all the files in the current, working directory that have an extension of .tf. Whether you spread the resource definitions over multiple files in the same directory (with the extension .tf) or have them all in the same file, that does not matter at all.

There is a best practice it seems (though not enforced by Terraform) to have separate files for variable definitions – called variables.tf by convention –  for resource definitions – typically called main.tf – and for output – usually called output.tf.

Note: when making use of modules , then files from other directories can play a role as well.

Manage OCI Resources through Terraform

Now that I have a working environment with Terraform and the OCI provider set up, I would like to define an OCI resource that I would like Terraform to create for me.

There is plenty of documentation on how to define OCI resources. Let’s take a simple resource: a bucket on Object Storage. This bucket in its almost simplest form can be defined like this:

# the bucket to be managed by Terraform

# the bucket to be managed by Terraform
resource “oci_objectstorage_bucket” “lab_bucket” {
     compartment_id = “ocid…..”
     name           = “lab-bucket”
     namespace      = “my-namespace”
}

This definition lives in main.tf.

My Terraform configuration now resides in two files: variables.tf (which defines the provider oci) and main.tf (which defines the bucket resource).

I feel now ready to run Terraform and ask it to make the plan for managing this resource, using this command:

terraform plan -out config.tfplan

This command does not execute successfully. It tells me that it did not find a proper configuration for the tenancy.

That I can understand. Of course. Nowhere did I specify which OCI tenancy the Terraform provider should be working for.

So how do I steer Terraform to my OCI tenancy? I struggled with this for a little while. There are several ways. One is to configure the oci provider in variables.tf with hard coded values for properties tenancy_ocid, user_ocid, fingerprint, region and private_key_path – and provide a file with the private key. A better way is to define the values of these properties using variables, as is shown here:

The question then becomes: where do these variables get their values from? And, as I realized later on, where are these variables defined in the first place. You can not just refer to variables without first explicitly defining them. We will get back to these variables a little bit later on.

It turns out that there is a third way for the OCI provider to know how to connect to a tenancy – and this is exactly the same way as used by the OCI Command Line Interface. The configuration file and private key file that you typically set up for using the OCI CLI can be leveraged by the Terraform provider for OCI as well. The files have to be called config and oci_api_key.pem and they have to be located in the directory ~/.oci.

The private key file contains the pem formatted primary key and the config file contains key-value pairs as described in this documentation and as shown below, just like you use them with the CLI:

[DEFAULT]
user=ocid1.user.oc1..aaaaaaaa5xonod6pvh2t3kcqhivndrn5jpci7po7tna
fingerprint=eb:59:c1:56:65:80:2f:46:13:29:37:cb:5b:4e
key_file=/root/.oci/oci_api_key.pem
tenancy=ocid1.tenancy.oc1..aaaaaaaagvsodyym6we2egmlf3gg6okq
region=us-ashburn-1

Note: the Terraform provider for oci will always use the [DEFAULT] configuration.

After setting up directory ~/.oci with files config and oci_api_key.pem, I feel ready to try again the terraform plan command.

terraform plan -out config.tfplan

 

The screenshot shows the output I get from this command. It is looking good: Terraform – and more specifically the oci provider – seem to have accepted the configuration for my OCI tenancy. It has interpreted my resource definition, checked the current state of affairs in my tenancy and concluded that in order to achieve the desired state from the main.tf definition, it will have to create one resource – as shown in the output. Terraform read two files in this case: variables.tf andf main.tf. I did not instruct it to look at those files specifically: it will always process all files in the working directory that have the .tf extension.

This looks good to me, so I instruct Terraform to proceed and make my wish come true:

terraform apply

(I could have added a path to a configuration plan as was suggested in the output to terraform plan . However, I do not have to. In fact, I do not have to go through the plan step at all. This step helps me to understand the effect of the apply step ahead of time: it generated the migration plan and is particularly useful for letting someone else validate the change or to verify that there are no unwanted changes. If I do provide the migration plan in the apply step, the apply will be interrupted if differences are found between the current OCI state and state as found when the plan step was performed.)

Terraform writes output to explain what it intends to do. It then prompts me to confirm that action  by entering yes. After this confirmation, it will go ahead and perform the proposed actions.

The final output indicates what has been done. In this case: one resource was added, the new bucket called lab-bucket.

Using the terraform show command I can get an impression of the state of the resources after Terraform has weaved its magic:

The show command can be used to inspect a plan to ensure that the planned operations are expected, or to inspect the current state as Terraform sees it as I did here.

In the OCI Console, I can get the confirmation of the bucket creation in a user friendlier way:

 

Terraform offers another useful command: destroy . When you execute this command, all resources managed by Terraform are deleted. In other words: all resources defined in all .tf files in the working directory will be removed from the OCI tenancy. Rest assured: you need to explicitly confirm this command by typing yes when prompted.

terraform destroy

One resource is managed by Terraform (the bucket) and after entering yes it is created. Using terraform apply it is created again. All much faster than I could get it done in the console.

Note: The behavior of any terraform destroy command can be previewed at any time with an equivalent terraform plan -destroy command.

 

Managing Variables

So I have managed to get OCI provider running on Terraform and have it create and later destroy a resource that I defined in a configuration file. That seems a fine start.

My next question: how can I set resource properties in a dynamic way – not using hard coded values. Right now the name of the bucket is hard coded in the resource definition; that is not what I want.

In Terraform, the values of resource properties can be based on variables or on local values . The property is defined using an expression that refers to the variable. Using variable references, the resource definition for the new bucket can be written like this:

resource "oci_objectstorage_bucket" "lab_bucket" {
    # variables compartment_id, namespace and tags are defined in variables.tf
    compartment_id = var.compartment_id
    name           = var.lab_bucket_name
    namespace      = var.namespace
  }

Note: before Terraform v0.12, references to variables were written as ${var.name_of_variable}. As of v0.12, that syntax is deprecated and results in warning (that for now, can be ignored).

In order to successfully refer to variables, these variables need to have been declared – either in the same .tf file or in another file in the same directory. Typically, a variables.tf file is used – at least for definitions of all variables that are meaningfully exposed. Such variable definitions are quite simple: they consist of a name, optionally a type and optionally a default value.

variable "compartment_id" {}

variable "namespace" {}

# this variable holds the name for the bucket that will be created; the default value is tf-bucket. This could be overridden through an Environment Variable 
 variable "lab_bucket_name" {
   default = "tf-bucket"
 }

variable "tags" {
   type = map(string)
   default = {
     "created-for" = "KatacodaLab"
   }
}

When you plan or apply a directory with .tf files that contain variables,  you need to make sure that all variables have a value. If no value has been provided, you will be prompted by Terraform at the command line:

The values of variables can be provided to Terraform in several ways (in this order):

• passed on the command line through the -var ‘variable-name=value’ syntax
• defined in a .tfvars file
• set in an Environment Variable called TF_VAR_name-of-variable
• set as the default value in the variable definition
• provided by the user who is prompted on the command line when all else fails

Probably the most common way to define variable values is through environment variables. In that way, the values are not part of the Terraform files and therefore not included in the source code control system. It is especially relevant for sensitive values that they are not checked in.

For the variable definitions included previously, I have to provide values for compartment_id and namespace. The variables lab_bucket_name and tags have default values for Terraform to fall back on. To provide values for these two variable, I can basically adopt two strategies:

1. set two environment variables before running terraform; ensure that these environment variables are named like the variables in the terraform scripts with TF_VAR_ prepended:

export TF_VAR_compartment_id=<OCID of compartment>

export TF_VAR_namespace=<namespace>

2. pass values for these variables on the command line to terraform with the plan and apply and even the destroy commands:

terraform plan -out config.tfplan -var compartment_id=<OCID of compartment>  -var namespace=<namespace>

You can use environment variables in the values passed on the command line; for example:

terraform plan -out config.tfplan -var compartment_id=$TF_VAR_compartment_id -var namespace=my-namespace-$some_env_var

If I adopt both strategies, the second will overrule the first. Values passed as parameters on the command line take precedence over environment variables.

Produce Output – both for debugging and for result reporting and verification

In Terraform configuration files, we can define output values. These are used to return values from modules to parent modules and – more relevant to me at this stage – to print certain values to the CLI output after running an apply command. Through an output value, I can instruct Terraform to print the value of a variable, a resource or resource property or a local value (more on local values a little bit later).

The definition of an output is very simple:

# report on the managed bucket resource by printing its OCID
output "show-new-bucket" {
  value = oci_objectstorage_bucket.lab_bucket.bucket_id
}

The output has a name and single property: value. The value argument takes an expression whose result is to be returned. Output values are only returned in the apply stage – not for example for plan nor destroy.

The output shown here will return

If I change the output’s value expression as follows (removing .bucket_id)

# report on the managed bucket resource by printing the object
output "show-new-bucket" {
  value = oci_objectstorage_bucket.lab_bucket
}

then the output becomes the complete resource object:

This output construct is frequently used for debugging – to check on the value of variables or the created resources.

Refer to Resource Properties in Resource Properties

One more challenge I struggled with in my very first steps with Terraform is the desire to derive the value of some properties for managed resources from properties of other resources – either non-managed, pre-existing resources or resources that are also managed by Terraform.

There are a few core mechanisms you need to be aware of:

• a resource in a Terraform configuration file can refer to other resources in that same (or a different .tf) file; in its property definitions, it can refer to properties on those other resources. Each managed resource exports a number of attributes. Each of these attributes can be referred to. See for example this document on the OCI Bucket resource type. The document lists the attributes that are exported – and can be referred to. I will show an example a little bit later on
• Data Sources in Terraform are read-only queries that retrieve information from Terraform providers that can be used elsewhere in Terraform configuration files, for example in resource definitions. This document on the OCI Bucket Data Source shows how to query buckets and lists the information provided on those buckets. Results returned by data sources can be referred to in a resource property definition. An example will follow.
• local values are local variables that can be used to set values in resource property definitions. The local value can be defined using an expression that refers to data exported by resources and queried by data sources; through local values we can prepare values to assign to properties.

We will look at a simple scenario:

• Retrieve details for an existing compartment called lab-compartment
• Create a new bucket in that compartment
• Create a file object in that bucket

We use a data source to find the compartment. The filter is used to restrict the result returned from the data source to a list with only elements for which the condition name == “lab-compartment” holds true. This return a list of one [compartment].

data "oci_identity_compartments" "lab_compartments" {
     compartment_id = var.tenancy_id
     # only retain the compartment called lab-compartment
     filter {
         name   = "name"
         values  = [ "lab-compartment"]
     }
 }

We can use an output to verify that this data source provides what we expect it to:

output "lab_compartment" {
  value = data.oci_identity_compartments.lab_compartments
}

In the resource definition for the managed bucket, we can refer to the result from the data source.

# the bucket to be managed by Terraform
resource "oci_objectstorage_bucket" "lab_bucket" {
    # variables namespace and tags and data source oci_identity_compartments.lab_compartments are defined in variables.tf
    compartment_id = data.oci_identity_compartments.lab_compartments.compartments[0].id
    name           = var.lab_bucket_name
    namespace      = var.namespace
    freeform_tags  = var.tags
}

The compartment_id property is set using the expression data.oci_identity_compartments.lab_compartments.compartments[0].id. This can be read as: locate data source with identifier oci_identity_compartments.lab_compartments. This data source returns a collection called compartments. Take the first element in that collection and use the id property in that element.

An alternative approach could work through local values. Here I put the ‘complex expression’ in the local value and keep the resource definition simple. More importantly, this approach allows me to reuse the local value without having to duplicate the logic for deriving the value from the data source result:

locals {
  # store the first (and only) compartment returned from the data source in the local variable
  lab_compartment = data.oci_identity_compartments.lab_compartments.compartments[0]
}

resource "oci_objectstorage_bucket" "lab_bucket" {
     compartment_id = local.lab_compartment.id
    name           = var.lab_bucket_name
    namespace      = var.namespace
    freeform_tags  = var.tags
}

Note how the compartment_id is set using the simple expression local.lab_compartment.id that can be read as: find local value called lab_compartment and use its id property.

Local values can provide the value for an output value – again, handy for debugging:

output "lab_compartment_id" {
  value = local.lab_compartment.id
}

The definition of a resource can directly reference attributes exported by other managed resources. In this case, an object is managed in the bucket that is managed. This means that when both are new, first the bucket is created and subsequently the object is created in that bucket – for which details about the bucket must be known.

The resource definition for the object uses a reference to the bucket resource – through the expression oci_objectstorage_bucket.lab_bucket.name. This expression does two things: it produces the value of the name property of the resource called oci_objectstorage_bucket.lab_bucket which is the managed bucket and it tells Terraform about a dependency between these two resources – and therefore about the sequence in managing these resources. When applying, first the bucket then the object and when destroying, first the object and then the bucket.

resource "oci_objectstorage_object" "hello-world-object-in-bucket" {
    bucket = oci_objectstorage_bucket.lab_bucket.name
    content = "Hello World"
    namespace = var.namespace
    object = "my-new-object"
    content_type = "text/text"
}

These two output values:

# report on the managed bucket resource by printing its OCID
output "show-new-bucket" {
  value = oci_objectstorage_bucket.lab_bucket.bucket_id
}

# report on the managed object resource by printing the full object
output "show-new-object" {
  value = oci_objectstorage_object.hello-world-object-in-bucket
}

produce this output:

And this is what it looks like in the console:

A data source definition can use references to other data sources, variables, local values and managed resources for its ‘query’ definition. For example, here is a data source that reads the managed object:

data "oci_objectstorage_object" "read-hello-world-object" {
  bucket = oci_objectstorage_object.hello-world-object-in-bucket.bucket
  namespace = oci_objectstorage_object.hello-world-object-in-bucket.namespace
  object = oci_objectstorage_object.hello-world-object-in-bucket.object
}

output "show-hello-world-object" {
  value = data.oci_objectstorage_object.read-hello-world-object
}

Summary

It took some time for me to come to workable terms with Terraform and the OCI provider. Of course I only read the documentation after I tried and failed. And I got overwhelmed by the many articles and examples that all seemed to skip the humble first steps that I was still struggling with. I am now at a point where I can start to appreciate the beauty of using Terraform and the OCI Provider as opposed to (imperative) OCI CLI scripts – which I already picked over manual repetitive steps in the Console. I hope my description of the steps I took will help you get started a little bit quicker than I did.

 

Resources

OCI Documentation on Terraform Provider

OCI Documentation on Resource Manager

Terraform Documentation on provider for OCI

Blog Article on using Local Values in Terraform to read individual elements from a list returned from a Data Source

Red Expert Alliance – Online Katacoda Lab on Terraform provider for OCI as part of the Red Expert Alliance Katacoda Course on Oracle Cloud Infrastructure

Terraform installer helper on GitHub: https://github.com/robertpeteuil/terraform-installer

Terraform OCI docs on Bucket: Example OCI Terraform Provider – Bucket – https://www.terraform.io/docs/providers/oci/r/objectstorage_bucket.html 

Terraform Docs on Output Values: https://www.terraform.io/docs/configuration/outputs.html