Setup and use of oVirt on CentOS7

Today, I’ll explain how to install and use oVirt: a nice virtualization tool for Linux, which is based on KVM. I installed the current production version 4.3.8 on a test machine, a Medion machine with an extra SSD drive. As usual, you’ll find the installation file for pxe that I used for my environment in my github repository [1]. You will need a third computer with a browser (f.e. a laptop) to get access to the management console.

When you go to the website oVirt.org > Download, installation seems quite easy: it’s just a matter of six commands (assuming you work with CentOS7):

sudo yum update -y

sudo yum install https://resources.ovirt.org/pub/yum-repo/ovirt-release43.rpm

sudo yum install cockpit cockpit-ovirt-dashboard -y

sudo systemctl enable –now cockpit.socket

sudo firewall-cmd –add-service=cockpit

sudo firewall-cmd –add-service=cockpit –permanent

After that, you can run the website: https://nodename:9090, and click a button to install a virtual machine with the administrator menu. The website uses a self-signed certificate, it is normal for your browser to give errors.

This all sounds simple, but you might (like me) run into trouble. In fact, it took me several days to get it all working. Some things to be aware of:

1) You need a DNS entry for both the management console and the host that oVirt is running on. That also means that the DNS cannot run on the virtualized oVirt machine that we are creating now. In my case, I’ll call the management VM mgt-ovirt.fra.nl and the physical host where oVirt is running on medion2.fra.nl. Later, we will add another physical node called medion.fra.nl to this environment. All of these are running in a DNS that is installed on a VM on my laptop.

2) The fact that your hardware can run Windows with Hyper-V or VirtualBox doesn’t mean that the oVirt environment will run without further changes in the default settings. Fortunately, there is a very good tool to help you to analyse which settings should be changed: this tool is called virt-host-validate, it is installed when the cockpit and cockpit-ovirt-dashboard packages are installed. See for output of this command the photo below:

In my case, I need to start the kernel with an extra parameter and I have to do a modprobe fuse before the virtualization will work.

3) When these preparations are in place and you start the wizard, you might (like me) run into the problem that the wizard will say “ERROR Host does not support domain type kvm for the virtualization type ‘kvm’ arch ‘x86_64′”. This was very strange, because not too long ago I installed qemu-kvm and used KVM virtual machines on the same physical machine without problems. I tried several things: because many web pages warned that the default user wouldn’t have the right permissions, I changed the file /etc/libvirt/qemu.conf to use user root and group root. The oVirt installation software didn’t allow that, it commented these lines out during the installation. I therefore tried not to use root, but used +0 instead. That fixed the problem in the first part of the installation, but in the second part my trick was discovered and the installation file was changed again, which lead to the same error message on a later place in the installation…

I finally came to the idea to do a simple reboot (!) before starting the web configuration. That reboot solved my problems: when I did the reboot, no changes were needed in the file /etc/libvirt/qemu.conf… I like the idea, however, to use user vdsm and group kvm for all the virtual machines that I start. This change is allowed by the installation software, these changes will not be commented out.

4) Though some error messages in the wizard suggest otherwise, I had to re-install from the start several times: simply restarting the wizard will lead to problems. This is where a pxe configuration file will save you lots of time, because you can always start again in a fresh environment where you change your configuration every time you gain more knowledge.

NFS

oVirt works with the assumption that you have a central storage environment to store the virtual machine images. I’ll use the same machine for storage as the machine where the virtualization software runs, so I need to configure NFS before starting the wizard. I’d like to have the NFS directory in the directory nfs-ovirt under /var/lib/libvirt: I created a mount point for a logical volume under this directory with disk space that is reserved for virtualization. This directory should be accessible to everybody (the access is restricted by the NFS layer). To make this directory better accessible, I create a soft link /nfs-ovirt in the root directory.

For NFS, the file /etc/exports should be changed. In my case, I grand read/write access for both the physical machines that I want to use for virtualization and for the management VM. The file then looks like:

systemctl enable nfs

systemctl start nfs

systemctl enable nfs-mountd

systemctl start nfs-mountd

systemctl enable rpcbind

systemctl start rpcbind

firewall-cmd –add-service=nfs –permanent

firewall-cmd –add-service=mountd –permanent

firewall-cmd –add-service=rpc-bind –permanent

firewall-cmd –reload

Wizard

The wizard will help you with lots of defaults, most of them are good. I think the management VM has too many CPU’s, I gave it two CPU’s instead of the suggested four. The amount of memory for the HostedEngine VM is default 16Gb, I gave it 8Gb. You can use the (existing) root password to give access to this VM (as I did, see the images), but it is also possible to generate an SSH key first and use that. Unfortunately, it isn’t possible to use another system account than root to give access to the physical host.

In the second part of the wizard, you should give the administrator password. This is a new password, which is used with the default admin account to give access to the management console. On this screen, it is also possible to configure the e-mail settings.

After checking the settings, you can start the setup. Please mind, that especially the step to install the ovirt engine appliance can take quite some time (on my machine: about 6 minutes). While you are waiting, you might want to look in the logs in /var/log/ovirt-hosted-engine-setup directory, where the detailed logs of the installation are kept. You will see that Ansible is used to do all kind of checks and to configure the software.

…and, after waiting for several minutes…

When step 3 has been finished, the wizard continues with configuring the storage: you can choose between NFS, iSCSI, Fibre Channel and Gluster. In the advanced settings, there is a default of 58GB of disk space, but when you use NFS, the system will fortunately recognize the real size of the NFS partition and it will ignore the limit of 58GB.

After another confirmation screen, the setup continues with configuring the storage. When that is complete, the system is ready to use.

First use of oVirt

The total installation, including the OS itself, the installation of oVirt with the post install script (see pxe file) and the wizard, will take 20-30 minutes (!). After the installation you can browse with https to the web address that you gave in the wizard, which is in my case https://mgt-ovirt.fra.nl (it will use the default port 443 for https). On this screen, you see the option to download a certificate. Click this link and install the certificate on local machine by double clicking on it. Many elementary actions (like uploading an iso file) are not possible when you don’t first install this certificate first.

When you installed the certificate, click in the main menu of the management website on administration portal and log on with username admin and the administrator password (that you provided in the second step of the wizard).

Storage: upload ISO’s

VM’s can be installed both from DVD (iso file) and PXE. When you deploy from DVD, the ISO must first be uploaded to oVirt. Go (with Chrome!) in the left menu to Storage > Disks  and click in the Upload menu (upper menu) on Start.

First, click on the button “Test connection”.  When the connection succeeds, upload an ISO file (f.e. CentOS 8). When the test fails, retry to add the certificate to your computer.

When the test succeeds and OK is pressed, you will first see the newly created image in Locked state. After a few seconds, the transfer will be started automatically.

Create a new VM

Now it’s time to start using the environment by deploying our first VM. In the left menu, go to compute > Virtual Machines and click on New. Don’t forget to add storage (via the Create button) and a network card on the first screen.

In the System submenu, change the memory size to 2048MB and give the VM two virtual CPU’s.

In the boot options menu, choose for CD-ROM as first device and click on the checkmark before Attach CD.

In the drop down window, you can select the ISO that you uploaded earlier. When you finish the wizard and start the VM (via the Run button in the upper menu), use the console button to download a settings file for virtual machine viewer.

When you installed virtual machine viewer [2] and double click on the downloaded settings file, the console of the VM will be started. When the installation process asks you to reboot, you can change the settings of the VM (Edit button in the upper menu of the website interface, boot options, boot from hard disk instead of CD Rom) while the VM is running. Press Ok on the warning and press Reboot in the VM console to reboot the VM. The VM should now start from the hard disk, your VM is ready to use.

Create a new VM using PXE

You know how I love PXE to configure new VM’s by scripting. Using PXE in the oVirt environment wasn’t that easy: the PXE boot is not too picky about the DHCP offers it gets. In my case, it always took the DHCP request of the DHCP server that wasn’t configured for PXE. When I searched for a solution on the internet, I found a site where it seemed to be possible to configure oVirt to use a specific DHCP server: by changing the vNic settings to allow-dhcp-server and by configuring the IP-address in the network card of the VM (keyword DHCPSERVER). This network filter was deleted a few releases ago, because it didn’t work.

There is another solution: oVirt supports internal networks. You can create an internal network via Network > Networks, New and after giving it a name. Check the “Create on external provider” box. You now should check the “create subnet” tickbox in the Subnet submenu on the left. When you don’t tick this box and don’t give a name and subnet to this subnet, the network cards of the VM’s will not see each other. When the subnet is created, there is automatically a DHCP server on this network. This is in most situations a good idea, however, you guessed right, this DHCP server is in general also faster than the PXE server you configured yourself.

I searched a lot (and long) for a solution. In the end, it was the iPxe documentation that gave a solution: when you press ctrl-B after the PXE machine started, you will enter the command line of iPxe. You can type

autoboot

here. When the DHCP server without PXE configuration is found, iPxe will give an error. With arrow-up, you get the autoboot command again. Try this until your PXE server is found (in general: the second try).

It is possible to embed a script which has the same functionality in the rom-files of iPxe. See the documentation of iPxe to show how this works (you will find the roms in /usr/share/qemu-kvm) [3].

Shutdown and reboot

When you shutdown your machine, the VM’s that are still running will be stopped. This can take quite some time. After the start of your machine, you will have to wait about 10 minutes (!) before the HostedEngine VM is started. In the meantime, you can follow the start up via tail -f /var/log/ovirt-hosted-engine-ha/broker.log. When you see that the HostedEngine VM has come up, you can log on to the management service.

And now… the nice things!

You might have got the impression that I don’t like oVirt. This isn’t true: I think it’s a great product, but it does have a rather steep learning curve. Two nice things of oVirt are the dashboard and the way it deals with high availability. Let’s look at this.

Dashboard

The dashboard of oVirt is impressive, let’s look at the main screen:

When you click on the red or orange flag, you will see (either) the warnings (or) the errors from today.

When you click on one of the circles on the main dashboard, you see the different virtual machines, with their CPU or memory usage:

When you look at the virtual machines, you will see per VM what the amount of memory, CPU and network is being used:

High availability

Obvious, but true: for high availability, you need at least two physical machines. On the second machine, install oVirt the same way as on the first physical machine. You don’t need to use the wizard, because the HostedEngine VM is already running.

Go in the management website to Compute > Hosts, and click on the New button in the upper menu. Type the name of the host, the full host name (Full Qualified Domain Name, in my case medion.fra.nl) and the root password. When you pressed OK, then the new host is added, in state installing. When you want to see what is happening, go to the node and use tail -f /var/log/messages. After a few minutes, the second host is also up.

Now, add the storage of the second node to the oVirt environment: go to Storage > Domains, and add a new storage domain. I will call my domain medion_domain, the export path is medion.fra.nl:/nfs-ovirt.

Let’s move a VM from medion2 to medion: go to Compute > Virtual machines, and select a VM. In the context menu, the option “Migrate” is visible, click on it:

There is just one host to select, so this choice is not too difficult to make. The next option, Migrate VM’s in affinity, makes it possible to look at dependencies with other affinity’s and rearrange those VM’s as well. The migration doesn’t take too long, but this is because only the run environment of the VM is changed, not the disk. You can also move disks to storage on another host: go to Storage > Disks, right-click on a disk and select “Move”. When the disk is attached to a VM, you will get a warning that the disk is moved while the VM is running.

The action is, however, allowed. When I tried this, the disk was successfully transferred, though it took quite some time to do so. Unfortunately, the only thing you know is that the disk is locked, you don’t see any progress. This will be solved in the next major release, which is now in alpha stage. On the positive side: both the transfer of the VM and the transfer of the disk were possible without having to shutdown the machine.

Templates and scripting

One of the other nice features is that it is possible to create a template for new VM’s. In these templates, you can define options for f.e. the startup of the VM, or the amount of CPU’s or memory. After creating a template, the context menu of the template will have an option “New VM”, which you can use to create a new VM based on this template.

It is also possible to use ansible to script the creation of VM’s. I might write another blog about that in the future.

As before, I used PXE to deploy the hosts with the initial software. You can download the PXE script from my repository [1]. When you use this pxe file, please mind that this configuration is made for my personal medion2-machine. This machine has two SSD drives, sda with about 80 Gib and sdb with about 1000 Gib, the PXE script will reconfigure those disks. You will most likely have to change the settings of the partitions. You might also get (other) warnings from virt-host-validate, change your pxe-file based on the advice of this tool.

Conclusion: home use

There are some things to keep in mind if you want to use oVirt at home. One of the most important things is the start up time (and the shutdown time) of the server: it takes more than 10 minutes to start up the server with the HostedEngine VM so you cannot use it directly after the physical machine is started. This is a huge disadvantage when you are used to f.e. Hyper-V or VirtualBox (which you can use as soon as the laptop or the server has been started).

The nice dashboard and the fact that this software is free is a big advantage for home users.

Conclusion: company use

When you use this software within a company, you’ll not be too glad about the enforced use of the root account within this solution. The need to visually check the management password in the wizard because the password is not checked a second time is also not correct from a security perspective. And the need for a DNS server outside the virtualized environment is not that great.

There are, however, many options in this software that make this software a good solution for companies: the possibility to create a high available environment by creating clusters and data centers, the high granularity of permissions that you can give individual users, the nice dashboard – I think that’s really great. I strongly advice to script your solution, because I had to re-install the software very often before I got a working environment.

Foot notes

[1] https://github.com/FrederiqueRetsema/AMIS-Blog-Pxe , directory oVirt

[2] https://virt-manager.org/download

[3] https://ipxe.org/scripting