Last week I passed for my RHCSA exam (EX200) [1]. While I was studying for the exam, I asked myself which parts of this exam are still relevant when you use Linux servers and Linux containers in the cloud and which parts are not.
Assumptions
In the cloud, we prefer:
- SaaS (Software as a Service) solutions over Serverless solutions
- Serverless solutions over PaaS (Platform as a Service) solutions
- PaaS solutions over IaaS (Infrastructure as a Service) solutions
- Microservices (preferably: serverless or in containers) over Virtual Machines
- IaaC over manual changes
Red Hat still assumes that Linux machines are used for a very long time. Because of this, Red Hat exams stress for persistency: when a system boots, the configured solution should still work. Red Hat tries to avoid systems that have an uptime of months, sometimes even years. No operator dares to reboot these systems out of fear that that will break the system.
In auto scaling solutions in the cloud we use VMs in a different way. Linux machines in the cloud are in general configured in a way that the VM is stateless: when more capacity is needed, a VM with the same functionality will be started within minutes. When less capacity is needed, the VM is destroyed. Some virtual machines live just a few hours and are destroyed after that. A lot of Linux systems in the cloud are not designed to be patched or rebooted, they will just be replaced by a new image that contains new versions of the software.
On this moment there are still a lot of legacy systems running on Linux in the cloud. These systems are configured and used in more or less the same way that people used to do on-premises years ago. In the future legacy systems might be decommissioned or replaced by containers, PaaS or serverless solutions. This also means that certain Linux knowledge will become less relevant in the future.
Which parts of the exam are still relevant today?
Using keys for access to VMs
When you deploy a virtual machine in for example AWS, you can only do so by using an AWS generated SSH key. You then can use ssh -i key.pem ec2-user@1.2.3.4 to log on to the virtual machine. It is still important to know about scp and sftp to use encryption in transit. ssh-copy-id is not very common in a cloud environment.
What to do when you are not able to log on to a VM
It is good to know how to change the root password of a Linux system in case a root password is lost. At least, when you work on an on-premises Linux system.
When you follow best practices in the cloud, the server can be replaced easily with Infrastructure as Code. Operators will nowadays not be that often on the prompt of a virtual machine in the cloud as they might have been on an on-premises Linux machine in the past. It is impossible in the cloud to add the rd.break switch in the grub2 menu to stop the start-up process of the VM and then replace the root password.
It would be better to learn how to solve the issue when you lose your key.pem file and still want to log on to the virtual machine that you created earlier. In AWS this can be done by creating a new VM with a new boot disk (using a new key.pem file), then attach the boot disk of the original VM and copy the information from the new authorized key file to the old one. The nouuid option needs to be used in the mount command. This solves the issue that the two root disks might have the same UUID when the VMs were created using the same Linux AMI [2].
Using a different Linux kernel in the cloud
It is very good to know how to fall back on a previous kernel version via grubby. This is even more relevant in the cloud than on-premises: on-premises an administrator can select a different kernel version from the grub2 startup-menu, in the cloud this grub menu is not accessible for cloud operators.
Information about Linux commands
I think it is good to learn about man pages, info and the /usr/share/doc directory. Knowing how to use /var/log/messages and journalctl is good to know as well as you will not be able to see the information at boot time. Knowing how to make the journalctl information persistent is not that useful, most Linux systems in the cloud will send their logging information to a cloud logging service like AWS CloudWatch or Azure Monitor.
File systems
Permissions
Most Linux systems that are started in the cloud will not have many users. Learning the chmod and chown commands is enough for most workloads. Knowing about SUID, SGID, sticky bits and File Access Control Lists is useful for cases where you use NFS technology (EFS on AWS, Storage Accounts in Azure) as a file system.
User and group commands
I think that most companies will use a directory service (Active Directory or LDAP) for their user management. In the cloud you will not use tools like useradd, usermod, userdel a lot. This is even more true for tools like chage and visudo.
SELinux
SELinux will help creating least-privileged Linux VMs and containers. It’s good to know how SELinux works. Maybe the use of permissive mode (or not enabling SELinux in the first place) should not be part of the RHCSA exam, it is in no way a best practise…
Scripting
Knowing the basics about scripting (using if, while, for, case and dealing with environment variables) is good: it will discourage people to do configuration manually or solve repeating issues in different ways. It helps to script Infrastructure as Code in the right way as well. Environment variables and scripts and scripts are also used a lot in containers. Knowing how to use sed -i in boot scripts of VMs to change default configuration files is very useful.
Containers
In general it is good to know basic podman commands. I would have liked more content about the build file and pushing the container image to a generic container library. Most companies will use the AWS or Azure container registry for their own containers. The root user should not be used within the container, unfortunately this knowledge is not part of the RHCSA exam. When Red Hat would check this knowledge every time someone does an RHCSA exam we might get rid of insecure containers in production environments in the future. The way to start a container persistently in a user account is not relevant in cloud environments. You will in general use a serverless container orchestration service that is managed by AWS or Azure.
General commands
Tar, gzip, bzip2 are used quite a lot in software distribution. It’s good to know how that works. For trouble shooting everyone should be able to use commands like find and wc.
Software management via YUM
One might need a company-based repository instead of the general available libraries. I do think, however, that this is a more advanced topic. In most cases, the yum/dnf repositories that are deployed by AWS/Azure and the Cloud Platform Teams will do. Most Linux engineers will never touch the repo files.
This is, however, just one part of using yum. Knowing the most important yum switches (list –available, list –installed, search, provides) and knowing yum modules can be very useful.
NTP
Virtual Machines in AWS and Azure use time services as well. It is impossible to connect to AWS or Azure services with the command line interface when the difference in time within the VM and the time on the Cloud Platform is too big. It is therefore useful to know how chrony works.
Less useful knowledge for cloud environments
Multi-target Linux environments
Knowing that Linux can be started with a GUI might be useful, but knowing how to change from a command line Linux environment to a GUI and back is not used in a cloud environment very often.
Performance and tuning
I doubt if tuning processes with different priorities (nice) or tools like tuned are used a lot in cloud environments. Nice might be useful in legacy environments. Tuned will use by default the correct virtual-guest profile and I doubt if someone would ever want to change this. Tuning a VM to use resources in the most optimal way might be more expensive than just spinning up an extra VM. In general operators tend not to count the time (and therefore: money) they invest in optimization of VMs. They also tend to overestimate the costs of more or bigger cloud resources.
Partitioning and formatting disks
Most Linux administrators in AWS will store their long-term data in EFS or in S3 (AWS) or in Azure Storage. This makes attaching extra disks to VMs, creating partitions and file systems less common. The same is true for using tools like LVM, Stratis and VOD. Having some knowledge about LVM might be useful because LVM is installed on a freshly installed Red Hat Linux system by default, though the default configuration will do for most use cases. Knowing LVM is sufficient for most cases. Fortunately, VDO and Stratis seem to be gone in RHCSA version 9.
AT and cron
Tools like at and cron are not used a lot in the cloud: we use cloud native solutions for scheduling. Legacy servers might still use them.
Network tools for configuring network interfaces
When you start a virtual machine in the cloud, the IP addresses will be provided by AWS or Azure via DHCP and even though one can change the configuration within the VM this is one of the anti-patterns for cloud systems. In cloud environments one will probably never need tools like nmcli or nmtui.
Firewalls
I once had a lot of discussion with a colleague who told me that having ports closed in NACLs (Network Access Control Lists) is not useful when you close the same ports in Security Groups in AWS. Security Groups are better maintainable, so why bother about closing ports in NACLs? You can also argue the other way around: when someone is able to change Security Groups but is not able to change NACLs, then defining the same allow and deny rules in NACLs and Security Groups might make sense.
This same discussion can be held for using firewalls within Linux when you also use NACLs and Security Groups in the cloud. There is a point where adding more security tools just takes more time, more complexity and therefore more costs where it doesn’t add value.
Network tools for DNS
In the future tools like dig and knowing about /etc/resolv.conf, /etc/hosts will be less important and information how the cloud platform deals with DNS will become more relevant. I honestly hope no-one uses /etc/hosts anymore. /etc/nsswitch is a more advanced topic that is not relevant for most linux engineers.
Conclusion
The way Linux is used in cloud environments is fundamentally different than the way Linux is used on-premises. The RHCSA exam still wants us to know a lot of details that are not too relevant for Linux when it is used in containers or for Linux VMs running in the cloud. It could have been a lot worse, though: a lot of information that I had to learn for Linux exams in the past (f.e. how to build our own kernel, kernel modules etc) was fortunately not part of this exam.
Links
[1] https://www.redhat.com/en/services/training/ex200-red-hat-certified-system-administrator-rhcsa-exam
Photo by Andrea Piacquadio
