Kubernetes is a system for running and coordinating containerized applications across a cluster of machines. Minikube runs a single-node Kubernetes cluster and can be used for local development. In this blog post I’ll compare 2 different ways to get a working Minikube environment on Windows based on experience with a workshop which we’ve created. One based on using Vagrant and VirtualBox (in which an Ubuntu environment is created) and one which uses Hyper-V (and an out of the box Minikube Linux distribution running on top). Do mind that many of the things in this blog post are a personal opinion.
Why Minikube?
At first I was not convinced using Minikube would provide sufficient environment to get to learn Kubernetes. There are several reasons though why I decided to go with Minikube.
Differences in Kubernetes distributions
When setting up something for a workshop or a blog, a purpose is to have it applicable to as many people as possible. The knowledge provided should be applicable in different environments. When using a complete Kubernetes distribution or a PaaS solution, only a part of the knowledge is reusable for different platforms. Minikube distributions are far more comparable than Kubernetes distributions (running locally or provided as PaaS).
Kubernetes distributions differ significantly
I started checking out complete Kubernetes distributions such as the one provided by Canonical and the one provided by Oracle in Vagrant boxes. Quickly I discovered that they differ significantly in many aspects such as virtualization technology (LXC for Ubuntu vs containerd for Oracle Linux) and installation procedure (conjure, snap, juju for Ubuntu, Vagrant, shell scripts for Oracle Linux). Also the resulting environments were different in configuration and in what was pre-installed inside them.
PaaS Kubernetes providers provide different experiences
Also most PaaS providers provide a different experience and tools. Usually PaaS providers have specific CLI tools for their environment and specific web interfaces. Large PaaS providers are Google (GKE), Azure (AKS) and Amazon (EKS) thus should you be required to specialize, it is probably safest to choose one of those. Google donated Kubernetes to the CNCF and is probably most advanced in its implementation but I do not know this for sure. On most cloud providers you can do unmanaged Kubernetes (do it yourself on IaaS) or managed (on PaaS). Again, the choices are largely provider specific.
Light on resources and easy to install
Since my target audience for blog posts is mostly developers and developers often develop locally on their own laptop, the ease of installation and resource usage are important.
Minikube on Windows using Hyper-V
When using Hyper-V, you cannot use VirtualBox (5.x) at the same time to run VMs; you need to make a choice and switching afterwards requires a Windows restart. The reason for this is that when Hyper-V is running, Hyper-V claims the hardware resources required to provide virtualization and provides an abstraction on top. The only way to access those resources is through an Hyper-V interface. This is also the reason why Hyper-V running on top of Hyper-V works and other virtualization technologies on top of Hyper-V are often an issue (unless they implement the Hyper-V interface). When VirtualBox is running, VirtualBox asks the host for access to the required resources and not Hyper-V. When Hyper-V is not running underneath the host, the host can provide this. Otherwise the host is (more or less) just a VM running on Hyper-V and cannot help VirtualBox with its requirements (something like ‘the wrong person to ask’).
Getting started with Minikube on Windows requires the use of several tools and some configuration;
- Hyper-V needs to be enabled and the Hyper-V manager installed. Hyper-V is an out of the box Windows 10 feature.
- Chocolatey. Chocolatey is a package manager for Windows which takes away the manual steps required for installing and updating software on Windows. Chocolatey is not required to install CLI tools and Minikube but saves you a lot of time. I can also recommend it for wider use and not just for Docker, Kubernetes, Minikube, Helm and such.
- Hyper-V configuration. You need to create a Virtual Switch which is linked to the network adapter which obtains an IP for the Minikube or Docker Desktop Hyper-V VM to be accessible. Be aware that when your network changes (e.g. cable, wireless), you will need to select a different adapter. Hyper-V has a bug with using dynamic memory and you need to disable it.
- Docker Desktop. Software running in Minikube is packaged as Docker containers. The Minikube installation creates a VM which has a Docker registry. You use Docker to publish images to that registry. Docker Desktop includes Docker Machine which allows you to manage Docker hosts. In order to set the environment Docker requires to access your Minikube Hyper-V VM, you can use Minikube docker-env.
- Kubectl. This is the Kubernetes CLI. You will use this often.
- Minikube itself of course. Usually also Helm but this is out of scope for this post.
Minikube in Ubuntu in VirtualBox using Vagrant
This setup requires you to build your own Ubuntu installation with a Vagrantfile. For this setup you need:
- VirtualBox. This provides the virtualization.
- Vagrant. This makes creating a VirtualBox VM and prosioning it easy. Vagrant requires a Vagrantfile and provisioning scripts. Obtaining or writing these might seem the hardest part but it is actually pretty easy. You can find examples of such a Vagrant files here and here.
No additional tools are required for this since the provisioning scripts will take care of creating the Ubuntu VM and install the required CLI tools such as docker, kubectl, minikube.
You cannot do nested virtualization within VirtualBox 5.x or earlier. Thus you will need to run minikube with –vm-driver=none within the VirtualBox machine to start Minikube. Also since it is a custom VM, you also manually need to start minikube proxy in order to have a consistent endpoint (host:port) for the API and the dashboard. In VirtualBox 6, nested virtualization is possible though the feature is still young (read more here). This might allow you to run VirtualBox, an Ubuntu VM on top and for example use KVM for the Minikube VM. This way you do not use a custom Minikube environment but an out of the box one making the environment easier to maintain and upgrade. Have not tested this yet though.
This does mean though that the CLI tools will be there also and that building Docker containers will need to be done within the VirtualBox container or you will need to work with a shared folder in order to obtain Docker images from your Windows environment. You might be able to expose this environment to the host and use the Windows CLI tools but I have not tried this yet and I’m not planning to. Using the Hyper-V solution would be easier if you want to use the Windows CLI tools. Docker Desktop explicitly states only Hyper-V support so there might be issues.
Comparing the environments
In order to compare the environments it is helpful to look at different aspects.
How easy is it?
Installation
The Hyper-V path requires the installation of several tools. Once this is done, it does not require much maintenance. The VirtualBox/Vagrant path requires less tools (quicker to get up and running), however it does require provisioning scripts which creates an environment for you. Setting up these scripts yourself can take some time (more than the Hyper-V environment installation). Once you have them, the recreation of the entire environment (Minikube + CLI tools) is automated and documented (infrastructure as code). You can also use provided scripts such as here and here to save yourself the work.
Environment
The environment created when using the Hyper-V path, require the local CLI tools to access the Minikube VM running on Hyper-V. The Hyper-V VM and the tools are separated and need to be managed that way as two separate things. The tools need connectivity to the VM (Hyper-V virtual switch configuration) and the host needs configuration to allow this connectivity. This can cause issues when for example you change your laptops network from wireless to wired. These connectivity challenges might be a benefit though since when using a full Kubernetes, you will need to configure and connect to a similar remote instance and you already know how that works when you have to deal with it regularly (frequency reduces difficulty). A question is whether you want to deal with it regularly.
When using VirtualBox and have everything within that VirtualBox, connectivity is no issue since everything is located on the same machine; a single environment. Also creating backups of the complete environment (tools, configuration, running Minikube) becomes easy with clones and snapshots. The environment can quickly be rebuild and distributed (for a workshop for example) if necessary. On Hyper-V you can only easily do that for the Minikube machine but the tools need a local installation.
Where are the CLI tools located?
Hyper-V
The tools are installed natively on Windows. This is useful since if you are using a development environment on Windows, building a Docker container and deploying it to Kubernetes can be done from the same environment. Management of tools is also done on Windows. When using Chocolatey this is pretty easy. For large groups of developers, this tends to become everyone’s own responsibility, which has some risks. Benefit of having the tools native on Windows is that integration with other tools which are installed on Windows (such as an IDE) is easy.
VirtualBox/Vagrant
The tools are installed within the VirtualBox VM. If you are developing on Windows and want to use images within the VM, you have 2 options; figure out how to expose the VirtualBox VM to Docker Desktop (though it says only Hyper-V is supported) or develop completely from the VirtualBox VM. Developing from the VirtualBox VM has benefits such as isolation of the environment and you have the option to manage the development environment as code, allowing easy updates for large groups of developers. A drawback is the overhead of the virtualized environment / OS. This overhead is visible in CPU, memory and disk usage.
Virtualization
When using Hyper-V, a virtual machine runs in the background which can be managed from the Hyper-V manager. You can connect to it using RDP and can also use docker-machine to manage the machine from the CLI. Minikube runs within the Hyper-V VM.
You can login to this VM with user root. It uses docker-containerd inside to run containers. The custom Minikube Linux distribution is created by using Buildroot (although you do not have to care). It runs kube-proxy and a single node Kubernetes cluster.
Hyper-V is a Type 1 hypervisor. This means it works directly on hardware without having an OS layer in between. The host OS also runs through Hyper-V (installing Hyper-V actually puts Hyper-V between the host OS and the hardware). Since the host OS is the Parent Partition or Root Partition, there is not much delay of the host even though the host also accesses the hardware through the hypervisor. Since Hyper-V is dependent on the Root Partition for many things, there is some debate on whether Hyper-V is truly Type 1 and not Type 1+2.
VirtualBox is a Type 2 hypervisor. It depends on the host OS for access to hardware. This means it is slower since there is an extra layer between the VM and the hardware; the host OS. Also since it cannot directly access hardware, VirtualBox needs to provide an abstraction of the hardware provided by the host OS to the guest and provides several drivers for this. For example you can choose different NICs for the guest.
Hyper-V manager manages VMs running on Hyper-V but does not run them. To display a VM, RDP is used. VirtualBox actually runs VMs itself, in my experience VirtualBox provides a more seamless experience (between host/guest and hypervisor/guest) for developers when compared to Hyper-V + Hyper-V manager. With VirtualBox, the VMs run on VirtualBox which runs on the host. With Hyper-V the VM runs on the same hypervisor the host runs.
Both Hyper-V and VirtualBox provide paravirtualization drivers. These drivers allow the guest to access the hypervisor in an optimized manner.
Finally
Summary
To summarize see the below table. Of course many things are personal opinions. Yours might differ!
Which setup works best for you depends on your personal tastes. Generally speaking; if you like Windows, go for Hyper-V/Chocolatey and if you prefer Linux, go for VirtualBox/Vagrant. I haven’t done much with Apple products so can’t compare those with the above setups. I can imagine it would be similar to the Windows setup.
The Minikube environments created by both setups are comparable in usage. It is useful to get some experience with both in order to increase your personal experience. When getting started with Kubernetes or want to develop locally, I can definitely recommend Minikube! When learning Kubernetes, do not expect it to be easy. It will take some time to get the hang of it. As indicated, some of the knowledge you gain might be platform specific and not generally applicable. Kubernetes basics however are portable.
Alternatives
There are of course many alternatives worth investigating
- Using VirtualBox 6 nested virtualization feature
- Using Minikube to create a standard VirtualBox Minikube VM. See for example here. This way you can still use the Windows CLI tools but don’t have to manage a custom VM. You might have connectivity issues tools/VM but since VirtualBox is (in my experience) easier to configure (you will not need to fiddle with Hyper-V memory bugs, virtual switches), this setup would probably be easier to use than Hyper-V.
- Getting rid of Windows altogether (or dual boot or virtualized) and switch to for example Ubuntu or other developer friendly distributions. This would probably make working with Minikube easiest. You could use KVM for the Minikube host and install the CLI tools locally. Docker on Linux is also easier than on Windows.
