Coming to this solution was a journey. I was looking for a Kubernetes installation which was easy to deploy and cleanup on my own laptop (I didn’t want to have to pay for a hosted solution). I did want a solution which was more or less production-like because I wanted to be able to play with storage solutions and deal with cluster challenges / loadbalancers. Things I couldn’t do easily on environments like Minikube and Microk8s. Also, since I was running on a single piece of hardware, I needed a virtualization technology to deploy parts of Kubernetes to different hosts. Each virtualization technology comes with its own challenges. On some of them it is difficult to get storage solutions to work, for example LXC/LXD (a Linux container technology). Some of them are Windows specific like Hyper-V and some of them just don’t perform well like VirtualBox. I also needed a solution to provide some form of automation to create/destroy/access my virtual environments. A long list of requirements and this is what I ended up with. A production-like environment which is quick to create, destroy or reset, running on my laptop with easy management tools.
Charmed Kubernetes is a pure upstream distribution of Kubernetes by Canonical (you know, the people from Ubuntu). It can be deployed using so-called Charms with a tool called Juju. Read more here. Charms consist of a definition of applications and their required resources. Using juju you can deploy charms to various environments such as AWS, Azure, OpenStack and GCP. You can also deploy locally to LXC/LXD containers or to a MaaS (Metal as a Service) solution. Juju allows easy ssh access into nodes and provides several commands to easily manage the environments. In addition, it comes with a nice GUI.
During my first experiments I tried deploying locally to LXC/LXD. It looked nice at first since I was able to deploy an entire Kubernetes environment using very few commands and it was quick to start/stop/rebuild. When however I tried to do more complex things like deploying StorageOS, CephFS or OpenEBS, I failed miserably. After trying many things I found out I was mainly trying to get LXC/LXD do what I want. My focus wasn’t getting to know LXC/LXD indebt but playing with Kubernetes. Since LXC/LXD isn’t a full virtualization solution (it uses the same kernel as the host, similar to Docker) it was difficult to pull the required devices with the required privileges from the host into the LXC/LXD containers and from the containers into the containerd processes. This caused different issues with the storage solutions. I decided I needed a full virtualization solution which also virtualized the node OS, thus KVM or VirtualBox (I had no intention of running on Windows so dropped Hyper-V).
I already mentioned I wanted an easy way to manage the virtualized environments. I started out with Vagrant but the combination of getting a production-like environment out of the box using Vagrant scripts was hard to find. Also it would require me to define all the machines myself in Vagrant code. The Kubernetes distributions I would end up with were far from production like. They were automated mostly by developers who had similar requirements as my own but those solutions were not kept up to date by a big company and shared by many users. I wanted automation like juju but not LXC/LXD. What were my options without having to pay anything for it? MaaS, Metal as a Service, was something to try out. Juju could deploy to MaaS. I found that MaaS could be configured to create KVM machines on Juju’s request! Nice, let’s use it!
My main challenge was getting networking to work. MaaS can create virtual machines in KVM using virsh. The machines boot and then they have to be supplied with software to allow management and deployment of applications. MaaS does this by ‘catching’ machines on DHCP requests and then use PXE to provide the required software. This is all automated though. You only have to provide the networking part.
Bringing it all together
I started out with a clean Ubuntu 18.04 install (desktop, minimal GUI) on bare metal. I could also have chosen for 20.04 since that has also recently been released but I thought, let’s not take any chances with the available tutorials by choosing a distribution which might contain changes which would make the tutorials fail. I think this can also be applied to 20.04 since it is not really that different compared to 18.04 but I haven’t tried.
First I installed some packages required for the setup. I choose to use MaaS from packages instead of Snap since I couldn’t get the connection to KVM to work with the Snap installation. The package installation created a user maas which was used to connect to the libvirt daemon to create VMs.
sudo apt-add-repository -yu ppa:maas/stable
sudo apt-get update
sudo apt-get -y install bridge-utils qemu-kvm qemu virt-manager net-tools openssh-server mlocate maas maas-region-controller maas-rack-controller
Configure the network
Probably the most difficult step. Part of the magic is done by using the following netplan configuration:
cat << EOF > /etc/netplan/01-netcfg.yaml
This creates a bridge interface/subnet which allows the VMs to communicate with each other and with the host. The stp parameter is needed to allow PXE (the network boot solution MaaS uses) to work.
The below part creates an interface which is available in KVM for the VMs to use. It is a bridge interface to the subnet. The below commands replace the default KVM interface.
sudo virsh net-destroy default
sudo virsh net-undefine default
cat << EOF > maas.xml
virsh net-define maas.xml
virsh net-autostart default
virsh net-start default
Since we’re at it, KVM also needs to have some storage defined:
virsh pool-define-as default dir - - - - "/var/lib/libvirt/images"
virsh pool-autostart default
virsh pool-start default
In order to allow the hosts on the subnet (your KVM virtual machines) to communicate with the internet, some iptable rules are required. In the below script replace enp4s0f1 with your interface which is connected to the internet (you can determine it from the ifconfig output)
sudo iptables -t nat -A POSTROUTING -o enp4s0f1 -j MASQUERADE
sudo iptables -A FORWARD -i enp4s0f1 -o br0 -m state --state RELATED,ESTABLISHED -j ACCEPT
sudo iptables -A FORWARD -i br0 -o enp4s0f1 -j ACCEPT
echo 'net.ipv4.ip_forward=1' | sudo tee -a /etc/sysctl.conf
echo 'net.ipv4.conf.br0.proxy_arp=1' | sudo tee -a /etc/sysctl.conf
echo 'net.ipv4.conf.br0.proxy_arp_pvlan' | sudo tee -a /etc/sysctl.conf
#For saving the iptables rules. After install the installer asks to save the current state
sudo apt-get install iptables-persistent
#If you want to save the state afterwards, do:
#iptables-save > /etc/iptables/rules.v4
It is a good idea to do a reboot now to make sure the state of your network is reproducible. The sysctl code is executed at boot before the br0 bridge becomes available so after a reboot, you’ll need to execute the below 3 lines. I do not have a solution for that yet.
sudo sysctl -w net.ipv4.ip_forward=1
sudo sysctl -w net.ipv4.conf.br0.proxy_arp=1
sudo sysctl -w net.ipv4.conf.br0.proxy_arp_pvlan=1
Also sometimes it helps (after you have already installed MAAS) to restart MAAS services such as:
sudo service maas-rackd restart
Next step is setting up Maas.
sudo maas init
This will ask you for an admin user and you can also supply an SSH key. After you’ve answered the questions, you can use the user you have created to login to:
You need to enable DHCP for the subnet you’ve created (10.20.81.0/24)
Also the maas OS user needs to be able to manage KVM. Add the maas user to the libvirt group:
sudo usermod -a -G libvirt maas
Create a public and private key for the user maas and add the public key to the maas user s authorized keys so the private key can be used to login to the maas user:
sudo chsh -s /bin/bash maas
sudo su - maas
ssh-keygen -f ~/.ssh/id_rsa -N ''
cat id_rsa.pub > authorized_keys
chmod 600 ~/.ssh/authorized_keys
The public key needs to be registered in maas. Click on admin in the webinterface and add the SSH public key
Now maas needs to be configured to connect to KVM. Click pods and add the virsh URL: qemu+ssh://email@example.com/system
You also installed virt-manager so you can manually create a VM and check it becomes available in MaaS (does the PXE boot and starts provisioning).
Now you have the network, KVM, MaaS configured and it’s time to move to the next step. Configure Juju.
snap install juju --classic
#Add your MaaS cloud environment
juju add-cloud --local
Use the following API endpoint:http://10.20.81.2:5240/MAAS. Do not use localhost or 127.0.0.1! This IP is also accessible from hosts on your subnet and allows Juju to manage the different hosts.
Add credentials (the MaaS user). First obtain an API key from the MaaS GUI.
juju add-credential maas
Bootstrap JuJu. This creates a controller environment. Juju issues its commands through this. If this step succeeds, installing Charmed Kubernetes will very likely also work. Should this fail, you can issue: juju kill-controller to start again. Do not forget to cleanup the VM in MaaS and KVM in such a case.
juju bootstrap maas maas-controller
Installing Charmed Kubernetes
A full installation of Charmed Kubernetes requires quite some resources. Most likely you don have them available on your laptop. Resources like 16 cores and more than 32Gb of memory.
Charms contain a description of the applications and the resources required to run them so hosts can automatically be provisioned to accomodate for those applications. You can override the resource requirements by using a so-called overlay.
For example, the original Charm looks like this: https://api.jujucharms.com/charmstore/v5/bundle/canonical-kubernetes-899/archive/bundle.yaml
I reduced the number of workers, the number of cores available to the workers and the amount of memory used by the individual master and the worker nodes so the entire thing would fit on my laptop.
cat << EOF > bundle.yaml
description: A highly-available, production-grade Kubernetes cluster.
constraints: cores=1 mem=3G root-disk=16G
constraints: cores=2 mem=6G root-disk=16G
Create a model to deploy Charmed Kubernetes in:
juju add-model k8s
Now execute the command which will take a while (~15 minutes on my laptop):
juju deploy charmed-kubernetes --overlay bundle.yaml --trust
This command provisions machines using MAAS and deploys Charmed Kubernetes on top.
Sometimes you end up with several machines in KVM which have not been provisioned and are turned off. You can remove these (all machines which are turned off including their storage) with a command like:
for i in `virsh list --state-shutoff --name`; do virsh undefine $i --remove-all-storage; done
In order to access the environment, you can do the following after the deployment has completed (you can check with juju status or juju gui)
sudo snap install kubectl --classic
sudo snap install helm --classic
juju scp kubernetes-master/0:config ~/.kube/config
The resulting environment
After juju is done deploying Charmed Kubernetes you will end up with an environment which consists of 8 hosts (+1 juju controller). Below is the output of juju status after the deployment has completed.
You can also browse the juju GUI. ‘juju gui’ gives you the URL and credentials required to login.
You can look in the MAAS UI to see how your cores and memory have been claimed
In virt-manager you can also see the different hosts
‘kubectl get nodes’ gives you the worker nodes.
You can enter a VM by for example ‘juju ssh kubernetes-master/0’
If this does not work because provisioning failed or the controller is not available, you can also use the juju private key directly.
ssh -i /home/maarten/.local/share/juju/ssh/juju_id_rsa firstname.lastname@example.org
You can execute commands against all nodes which provide an application:
juju run "uname -a" kubernetes-master
Of course you can access the dashboard which is installed by default:
Next access http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/login
You can use the .kube/config to login
If you need to cleanup, the best way to remove Charmed Kubernetes is:
juju destroy-model k8s
This will remove the application and the machines so you can start over if you have destroyed your Kubernetes environment. Reinstalling it can be done with the below two commands:
juju add-model k8s
juju deploy charmed-kubernetes --overlay bundle.yaml --trust
If you want to go back further and even remove juju, I recommend the following procedure:
sudo snap remove juju --purge
sudo snap remove kubectl --purge
sudo snap remove helm --purge
rm -rf ~/.kube
rm -rf ~/.local
I’ve mainly used the following sites to put the pieces of the puzzle together