In order to provision a Kubernetes cluster on Oracle Cloud Infrastructure, you need to prepare the OCI tenancy and create a compartment with appropriate network configuration and associated resources. All set up can be performed through the console – but this is quite tedious, error prone and outright stupid if you have to go through these steps more than once. I should know – I went through them multiple times.
In a previous article I described how easy it is to get going with the OCI CLI (using a Docker container) and in this article I will share the OCI CLI statements that should be executed in order to fully prepare the OCI tenancy for subsequently provisioning the OKE Cluster. (OKE = Oracle Kubernetes Engine). The article “Create OKE Kubernetes Cluster on Oracle Cloud Infrastructure” will subsequently take you through the steps of creating an K8S instance leveraging the artifacts created in this article.
Assuming you have the OCI CLI configured with a user with the right privileges connecting to the target OCI tenancy, below you will find the steps to perform through the OCI to create all artifacts that are required in order before an OKE Cluster can be created. After providing with the step by step commands, I will also show you the beginnings of a Shell script that will execute all individual steps automatically,saving quite a bit of manual work.
Step by Steps Commands
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Export the OCID of the root compartment: | |
| export ROOT_COMPARTMENT_ID=ocid1.tenancy.oc1..aaaaaaaaot | |
| Create a new OKE policy in the root compartment of your tenancy: | |
| oci iam policy create --name oke-service --compartment-id $ROOT_COMPARTMENT_ID --statements '[ "allow service OKE to manage all-resources in tenancy"]' --description 'policy for granting rights on OKE to manage cluster resources' | |
| List all policies, to verify the success of the creation: | |
| oci iam policy list --compartment-id $ROOT_COMPARTMENT_ID --all | |
| Now create a special compartment for all OKE resources | |
| oci iam compartment create --compartment-id $ROOT_COMPARTMENT_ID --name oke-compartment --description "Compartment for OCI resources created for OKE Cluster" | |
| From here on, work in the oke-compartment , set the COMPARTMENT_ID with the identifier returned by the previous command: | |
| export COMPARTMENT_ID=ocid1.compartment.oc1..aaaaaaaaheaww5zcb | |
| Create the Virtual Cloud Netwerk: | |
| oci network vcn create --compartment-id $COMPARTMENT_ID --cidr-block '10.0.0.0/16' --display-name oke-vcn --dns-label oke1 | |
| Use the OCID returned from the previous command to set the value for environment variable VCN_ID | |
| export VCN_ID=ocid1.vcn.oc1.iad.aaaaaaaadzp2obutldz2 | |
| The VCN in which you want to create and deploy clusters must have an internet gateway. The internet gateway must be specified as the target for the destination CIDR block 0.0.0.0/0 in a route rule in a route table. | |
| oci network internet-gateway create --compartment-id $COMPARTMENT_ID --vcn-id $VCN_ID --is-enabled true --display-name internet-gateway-oke | |
| Also export the OCID returned by this statement as INTERNET_GATEWAY_ID | |
| export INTERNET_GATEWAY_ID=ocid1.internetgateway.oc1.iad.aaaaaaaavm | |
| Create the route table in the VCN: | |
| oci network route-table create --compartment-id $COMPARTMENT_ID --vcn-id $VCN_ID --display-name routing-table-oke --route-rules '[{"cidrBlock":"0.0.0.0/0","networkEntityId":'$INTERNET_GATEWAY_ID'}]' | |
| Export the OCID of the route table like this: | |
| export ROUTE_TABLE_OCID=ocid1.routetable.oc1.iad.aaaaaaaaqpm7i2kgps4 | |
| Create the DHCP Options | |
| oci network dhcp-options create --compartment-id $COMPARTMENT_ID --vcn-id $VCN_ID --options '[{"type": "DomainNameServer", "customDnsServers": [], "serverType": "VcnLocalPlusInternet"}]' | |
| Create two Security Lists, with the names workers and loadbalancers. These names are optional. | |
| oci network security-list create --compartment-id $COMPARTMENT_ID --vcn-id $VCN_ID --display-name workers --ingress-security-rules '[ | |
| { | |
| "icmp-options": null, | |
| "is-stateless": true, | |
| "protocol": "all", | |
| "source": "10.0.10.0/24", | |
| "source-type": "CIDR_BLOCK", | |
| "tcp-options": null, | |
| "udp-options": null | |
| }, | |
| { | |
| "icmp-options": null, | |
| "is-stateless": true, | |
| "protocol": "all", | |
| "source": "10.0.11.0/24", | |
| "source-type": "CIDR_BLOCK", | |
| "tcp-options": null, | |
| "udp-options": null | |
| }, | |
| { | |
| "icmp-options": null, | |
| "is-stateless": true, | |
| "protocol": "all", | |
| "source": "10.0.12.0/24", | |
| "source-type": "CIDR_BLOCK", | |
| "tcp-options": null, | |
| "udp-options": null | |
| }, | |
| { | |
| "icmp-options": { | |
| "code": 4, | |
| "type": 3 | |
| }, | |
| "is-stateless": false, | |
| "protocol": "1", | |
| "source": "0.0.0.0/0", | |
| "source-type": "CIDR_BLOCK", | |
| "tcp-options": null, | |
| "udp-options": null | |
| }, | |
| { | |
| "icmp-options": null, | |
| "is-stateless": false, | |
| "protocol": "6", | |
| "source": "130.35.0.0/16", | |
| "source-type": "CIDR_BLOCK", | |
| "tcp-options": { | |
| "destination-port-range": { | |
| "max": 22, | |
| "min": 22 | |
| }, | |
| "source-port-range": null | |
| }, | |
| "udp-options": null | |
| }, | |
| { | |
| "icmp-options": null, | |
| "is-stateless": false, | |
| "protocol": "6", | |
| "source": "138.1.0.0/17", | |
| "source-type": "CIDR_BLOCK", | |
| "tcp-options": { | |
| "destination-port-range": { | |
| "max": 22, | |
| "min": 22 | |
| }, | |
| "source-port-range": null | |
| }, | |
| "udp-options": null | |
| }, | |
| { | |
| "icmp-options": null, | |
| "is-stateless": false, | |
| "protocol": "6", | |
| "source": "0.0.0.0/0", | |
| "source-type": "CIDR_BLOCK", | |
| "tcp-options": { | |
| "destination-port-range": { | |
| "max": 22, | |
| "min": 22 | |
| }, | |
| "source-port-range": null | |
| }, | |
| "udp-options": null | |
| }, | |
| { | |
| "icmp-options": null, | |
| "is-stateless": false, | |
| "protocol": "6", | |
| "source": "0.0.0.0/0", | |
| "source-type": "CIDR_BLOCK", | |
| "tcp-options": { | |
| "destination-port-range": { | |
| "max": 32767, | |
| "min": 30000 | |
| }, | |
| "source-port-range": null | |
| }, | |
| "udp-options": null | |
| } | |
| ]' --egress-security-rules '[{ | |
| "destination": "10.0.10.0/24", | |
| "destination-type": "CIDR_BLOCK", | |
| "icmp-options": null, | |
| "is-stateless": true, | |
| "protocol": "all", | |
| "tcp-options": null, | |
| "udp-options": null | |
| }, | |
| { | |
| "destination": "10.0.11.0/24", | |
| "destination-type": "CIDR_BLOCK", | |
| "icmp-options": null, | |
| "is-stateless": true, | |
| "protocol": "all", | |
| "tcp-options": null, | |
| "udp-options": null | |
| }, | |
| { | |
| "destination": "10.0.12.0/24", | |
| "destination-type": "CIDR_BLOCK", | |
| "icmp-options": null, | |
| "is-stateless": true, | |
| "protocol": "all", | |
| "tcp-options": null, | |
| "udp-options": null | |
| }, | |
| { | |
| "destination": "0.0.0.0/0", | |
| "destination-type": "CIDR_BLOCK", | |
| "icmp-options": null, | |
| "is-stateless": false, | |
| "protocol": "all", | |
| "tcp-options": null, | |
| "udp-options": null | |
| }]' | |
| oci network security-list create --compartment-id $COMPARTMENT_ID --vcn-id $VCN_ID --display-name loadbalancers --ingress-security-rules '[{ | |
| "icmp-options": null, | |
| "is-stateless": true, | |
| "protocol": "6", | |
| "source": "0.0.0.0/0", | |
| "source-type": "CIDR_BLOCK", | |
| "tcp-options": null, | |
| "udp-options": null | |
| } ]' --egress-security-rules '[{ | |
| "destination": "0.0.0.0/0", | |
| "destination-type": "CIDR_BLOCK", | |
| "icmp-options": null, | |
| "is-stateless": true, | |
| "protocol": "6", | |
| "tcp-options": null, | |
| "udp-options": null | |
| }]' | |
| Export the OCIDs for the two security lists: | |
| export WORKERS_SECURITY_LIST_OCID=ocid1.securitylist.oc1.iad.aaaaaaaa6jn | |
| export LOADBALANCERS_SECURITY_LIST_OCID=ocid1.securitylist.oc1.iad.aaaa | |
| List the availability domains: | |
| oci iam availability-domain list --compartment-id $COMPARTMENT_ID | |
| Using the OCID values for the Availability Domains, create subnets | |
| oci network subnet create --compartment-id $COMPARTMENT_ID --vcn-id $VCN_ID --display-name workers-1 --availability-domain <OCID AD-1> --cidr-block 10.0.10.0/24 --route-table-id $ROUTE_TABLE_OCID --security-list-ids "[$WORKERS_SECURITY_LIST_OCID]" | |
| oci network subnet create --compartment-id $COMPARTMENT_ID --vcn-id $VCN_ID --display-name workers-2 --availability-domain <OCID AD-2> --cidr-block 10.0.11.0/24 --route-table-id $ROUTE_TABLE_OCID --security-list-ids "[$WORKERS_SECURITY_LIST_OCID]" | |
| oci network subnet create --compartment-id $COMPARTMENT_ID --vcn-id $VCN_ID --display-name workers-3 --availability-domain <OCID AD-3> --cidr-block 10.0.12.0/24 --route-table-id $ROUTE_TABLE_OCID --security-list-ids "[$WORKERS_SECURITY_LIST_OCID]" | |
| oci network subnet create --compartment-id $COMPARTMENT_ID --vcn-id $VCN_ID --display-name loadbalancers-1 --availability-domain <OCID AD-1> --cidr-block 10.0.20.0/24 --route-table-id $ROUTE_TABLE_OCID --security-list-ids '[$LOADBALANCERS_SECURITY_LIST_OCID]' | |
| oci network subnet create --compartment-id $COMPARTMENT_ID --vcn-id $VCN_ID --display-name loadbalancers-2 --availability-domain <OCID AD-2> --cidr-block 10.0.21.0/24 --route-table-id $ROUTE_TABLE_OCID --security-list-ids '[$LOADBALANCERS_SECURITY_LIST_OCID]' |
by At this point, all resourced are created and you can attempt the provisioning of the OKE Cluster, either through the Console or trough the CLI or the REST API.
(beginning of a) Shell Script for Provisioning the Required OCI resources for an OKE Cluster Instance
Executing all these steps manually and constantly exporting environment variable is still a lot of work. The next level of automation is to take these individual steps and create a Shell script out of them. Below you will find that script – or at least the beginning of one, as I have not yet had the time to complete it.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| # in order to run, first install jq with: sudo apt-get install jq | |
| oci() { docker run --rm --mount type=bind,source=$HOME/.oci,target=/root/.oci stephenpearson/oci-cli:latest "$@"; } | |
| # set global variable OKE_COMPARTMENT_ID with the OCID for the compartment with name passed in $1 | |
| set_oke_compartment() | |
| { | |
| echo set_oke_compartments for $1 | |
| compartments=$(oci iam compartment list --compartment-id $ROOT_COMPARTMENT_ID --all) | |
| # echo "Compartments: $compartments" | |
| # echo $compartments | jq -r --arg cn "$1" '.data | map(select(.name == $cn)) | .[0]' | |
| # echo $compartments | jq -r --arg compartment_name "$1" '.data | map(select(.name == $compartment_name)) | .[0] | .id' | |
| OKE_COMPARTMENT_ID=`echo $compartments | jq -r --arg compartment_name "$1" '.data | map(select(.name == $compartment_name)) | .[0] | .id'` | |
| # echo "OKE_COMPARTMENT_ID now is $OKE_COMPARTMENT_ID" | |
| } | |
| echo $ROOT_COMPARTMENT_ID | |
| #policies=`oci iam policy list --compartment-id $ROOT_COMPARTMENT_ID --all` | |
| #echo $policies | |
| #echo $policies | jq '.data | .[1] ' | |
| set_oke_compartment "oke-compartment" | |
| echo "OKE Compartment OCID : $OKE_COMPARTMENT_ID" | |
| CREATE_VCN="false" | |
| if [ $CREATE_VCN = "true" ] | |
| then | |
| # create a virtual cloud network | |
| # https://docs.cloud.oracle.com/iaas/Content/ContEng/Concepts/contengnetworkconfigexample.htm | |
| OKE_VCN=`oci network vcn create --compartment-id $OKE_COMPARTMENT_ID --cidr-block '10.0.0.0/16' --display-name oke-vcn --dns-label oke ` | |
| #echo $OKE_VCN | |
| OKE_VCN_ID=`echo $OKE_VCN | jq '.data | .id'` | |
| echo "VNC OCID $OKE_VCN_ID" | |
| # create internet gateway | |
| OKE_IGW=`oci network internet-gateway create --compartment-id $COMPARTMENT_ID --vcn-id $VCN_ID --is-enabled true --display-name internet-gateway-oke` | |
| echo $OKE_IGW | |
| OKE_IGW_ID=`echo $OKE_IGW | jq '.data | .id'` | |
| echo "Internet Gateway OCID $OKE_IGW_ID" | |
| # create routing table | |
| ROUTE_RULES='[ | |
| { | |
| "cidrBlock": "0.0.0.0/0", | |
| "destination": "0.0.0.0/0", | |
| "destinationType": "CIDR_BLOCK", | |
| "networkEntityId": "'$OKE_IGW_ID'" | |
| } | |
| ]' | |
| echo $ROUTE_RULES | |
| echo `oci network route-table list --compartment-id $COMPARTMENT_ID --vcn-id $VCN_ID ` | |
| OKE_RT=`oci network route-table create --compartment-id $COMPARTMENT_ID --vcn-id $VCN_ID --display-name routing-table-oke --route-rules "$ROUTE_RULES"` | |
| echo $OKE_RT | |
| OKE_RT_ID=`echo $OKE_RT | jq '.data | .id'` | |
| echo "Routing Table OCID $OKE_RT_ID" | |
| OKE_DHCP=`oci network dhcp-options create --compartment-id $COMPARTMENT_ID --vcn-id $VCN_ID --options '[{"type": "DomainNameServer", "customDnsServers": [], "serverType": "VcnLocalPlusInternet"}]'` | |
| echo $OKE_DHCP | |
| else | |
| OKE_VCN_ID=ocid1.vcn.oc1.iad.aaaaaaaadzp2obutldz2aqpi22i7x5fic4ybimoaa47qk7qxle7feiwvtlca | |
| OKE_IGW_ID=ocid1.internetgateway.oc1.iad.aaaaaaaavmgvzktyknzf6p3u74co5l6hjanll2x5iht2wtr3heirsnbnywqq | |
| OKE_RT_ID=ocid1.routetable.oc1.iad.aaaaaaaaqpm7i2kgps4oo6yoliquoemoqmf7rkhpmgvqfihr3vjm6rq7s7ga | |
| fi | |
| echo Creating Security Lists | |
| OKE_SL_WORKERS=`oci network security-list create --compartment-id $COMPARTMENT_ID --vcn-id $VCN_ID --display-name workersXX --ingress-security-rules '[ | |
| { | |
| "icmp-options": null, | |
| "is-stateless": true, | |
| "protocol": "all", | |
| "source": "10.0.10.0/24", | |
| "source-type": "CIDR_BLOCK", | |
| "tcp-options": null, | |
| "udp-options": null | |
| }, | |
| { | |
| "icmp-options": null, | |
| "is-stateless": true, | |
| "protocol": "all", | |
| "source": "10.0.11.0/24", | |
| "source-type": "CIDR_BLOCK", | |
| "tcp-options": null, | |
| "udp-options": null | |
| }, | |
| { | |
| "icmp-options": null, | |
| "is-stateless": true, | |
| "protocol": "all", | |
| "source": "10.0.12.0/24", | |
| "source-type": "CIDR_BLOCK", | |
| "tcp-options": null, | |
| "udp-options": null | |
| }, | |
| { | |
| "icmp-options": { | |
| "code": 4, | |
| "type": 3 | |
| }, | |
| "is-stateless": false, | |
| "protocol": "1", | |
| "source": "0.0.0.0/0", | |
| "source-type": "CIDR_BLOCK", | |
| "tcp-options": null, | |
| "udp-options": null | |
| }, | |
| { | |
| "icmp-options": null, | |
| "is-stateless": false, | |
| "protocol": "6", | |
| "source": "130.35.0.0/16", | |
| "source-type": "CIDR_BLOCK", | |
| "tcp-options": { | |
| "destination-port-range": { | |
| "max": 22, | |
| "min": 22 | |
| }, | |
| "source-port-range": null | |
| }, | |
| "udp-options": null | |
| }, | |
| { | |
| "icmp-options": null, | |
| "is-stateless": false, | |
| "protocol": "6", | |
| "source": "138.1.0.0/17", | |
| "source-type": "CIDR_BLOCK", | |
| "tcp-options": { | |
| "destination-port-range": { | |
| "max": 22, | |
| "min": 22 | |
| }, | |
| "source-port-range": null | |
| }, | |
| "udp-options": null | |
| }, | |
| { | |
| "icmp-options": null, | |
| "is-stateless": false, | |
| "protocol": "6", | |
| "source": "0.0.0.0/0", | |
| "source-type": "CIDR_BLOCK", | |
| "tcp-options": { | |
| "destination-port-range": { | |
| "max": 22, | |
| "min": 22 | |
| }, | |
| "source-port-range": null | |
| }, | |
| "udp-options": null | |
| }, | |
| { | |
| "icmp-options": null, | |
| "is-stateless": false, | |
| "protocol": "6", | |
| "source": "0.0.0.0/0", | |
| "source-type": "CIDR_BLOCK", | |
| "tcp-options": { | |
| "destination-port-range": { | |
| "max": 32767, | |
| "min": 30000 | |
| }, | |
| "source-port-range": null | |
| }, | |
| "udp-options": null | |
| } | |
| ]' --egress-security-rules '[{ | |
| "destination": "10.0.10.0/24", | |
| "destination-type": "CIDR_BLOCK", | |
| "icmp-options": null, | |
| "is-stateless": true, | |
| "protocol": "all", | |
| "tcp-options": null, | |
| "udp-options": null | |
| }, | |
| { | |
| "destination": "10.0.11.0/24", | |
| "destination-type": "CIDR_BLOCK", | |
| "icmp-options": null, | |
| "is-stateless": true, | |
| "protocol": "all", | |
| "tcp-options": null, | |
| "udp-options": null | |
| }, | |
| { | |
| "destination": "10.0.12.0/24", | |
| "destination-type": "CIDR_BLOCK", | |
| "icmp-options": null, | |
| "is-stateless": true, | |
| "protocol": "all", | |
| "tcp-options": null, | |
| "udp-options": null | |
| }, | |
| { | |
| "destination": "0.0.0.0/0", | |
| "destination-type": "CIDR_BLOCK", | |
| "icmp-options": null, | |
| "is-stateless": false, | |
| "protocol": "all", | |
| "tcp-options": null, | |
| "udp-options": null | |
| }]'` | |
| oci network route-table create --generate-param-json-input route-rules |
Resources
Blog article: Create OKE Kubernetes Cluster on Oracle Cloud Infrastructure – including Service Request to increase limit https://technology.amis.nl/2018/10/16/create-oke-kubernetes-cluster-on-oracle-cloud-infrastructure-including-service-request-to-increase-limit/
Blog article: Running OCI CLI using Docker container – https://technology.amis.nl/2018/10/14/get-going-quickly-with-command-line-interface-for-oracle-cloud-infrastructure-using-docker-container/
First steps with Oracle Kubernetes Engine–the managed Kubernetes Cloud Service – https://technology.amis.nl/2018/05/25/first-steps-with-oracle-kubernetes-engine-the-managed-kubernetes-cloud-service/
OCI Docs Preparing for Container Engine for Kubernetes- https://docs.cloud.oracle.com/iaas/Content/ContEng/Concepts/contengprerequisites.htm?tocpath=Services%7CContainer%20Engine%7CPreparing%20for%20Container%20Engine%20for%20Kubernetes%7C_____0
