Linux Rolling Releases Head to Head Competition

My desire to always experiment with operating systems and the drumbeat of OpenSUSE updates on The Coder Radio Podcast had me give OpenSUSE for a whirl. It’d been awhile since I’d experimented with my one and only foray into rolling releases, Solus , so I decided to go with their Tumbleweed rolling release. Overall I’m pretty happy but I keep complaining about the time it takes to run updates. I’ve been told that’s pretty par for the course for a rolling release but I didn’t recall Solus having that sort of issue but memory is a funny thing. I figured the best thing to do would be to run the two head to head to see if it’s just a rolling release thing or an OpenSUSE thing. Then I figured why stop there. Thus the Linux Rolling Releases Head to Head Competition was born.

Introduction

For the uninitiated a rolling release is where there is no major version upgrade of an operating system. Instead the updates just roll out over time and the system just continually gets upgraded. Imagine a world where there was just Windows not Windows 7, 8, and 10. In the Apple world there would be no Catalina or Big Sur it’d just be macOS. Most Linux distributions still have major version updates. Ubuntu has 18.04 (from 2018), 19.04 (from 2019), and so on. In the modern Linux world many of these distributions can be upgraded in place pretty easily. However there is still an upgrade process. So why bother with rolling releases?

For people that want the latest and greatest Linux at all times it’s pretty nice. When working with stock configurations most people find it a painless process. It sounds scary to always be running the latest OS but in practice it isn’t as bad as some may think. I’m not saying I’d personally run production systems on it. For example you may get an automatic upgrade to a package that creates an incompatibility you wouldn’t be happy with and there is always the possibility, albeit slim, that the update makes the system unbootable. But for many they wouldn’t want it to be anything but the latest and therefore best. The other question is what is the overhead of all of this keeping up with the latest and greatest.

One of the things I love about my Linux machines over Windows is the seamlessness and speed of system updates. I am the opposite of one of those people that only updates once in a blue moon. I actually keep things updated several times a week. Windows seemed to constantly be having updates that would take me down for several minutes or more several times a week. It’s not the speed of the actual update in the background, which was sluggish, but the “okay let’s reboot to finish up” which would then drag on and on. Worse was when it’d decide to do the reboot automatically for me while I stepped away from the machine or overnight. Invariably that would happen when I was doing some sort of an analysis run or build overnight thus causing me to lose a day of work. Linux on the other hand is almost all in the background but with the reboot taking not much longer than normal, if any longer at all. I’ve been able to do entire version updates of a Linux machine in a fraction of the time a normal update was being applied to Windows. That changed with OpenSUSE’s Tumbleweed rolling release though.

Tumbleweed on shutdown often asks if I want to apply updates. Of course I do! I figured it’d be like the other Linux’s with it being pretty fast and then a shutdown. These seem to drag on for awhile. Worse was on the other side of it the reboot seemed to take substantially longer. How much of this is imagination versus actual behavior? How much of this is just the penalty if always rebuilding the latest kernel, having to install major version upgrades of the various packages, etc? I’m a numbers guy so let’s make some real numbers.

Methodology

To give a fair competition I need to compare apples to apples. That means I am going to compare only rolling release distros for the daily updates. Also to make it fair these will not be machines that are actually used. These are virtual machine images all with identical settings, running on the same hardware, running off the same disk. For each distribution I’m doing their standard desktop installs. None of these are completely minimalist versions. That means that they all have a reasonable collection of software for their users to be able to get started with: media players, office suite, text editors, maybe a couple games etc. They don’t all necessarily have the exact same configuration but it’s all relatively comparable. Lastly for distributions that have a “long term support” (LTS) version and a “latest” version which has a more recent kernel I chose the “latest” option. That way these should, hopefully, all track major Linux kernel updates pretty similarly.

The execution environment for all of these is VirtualBox running on my 2016 System76 Gazelle laptop that I brought out of retirement for the task. It is running a fresh install of Linux Mint and has nothing else running on it. The machine has a very fast NVMe main disk and a slightly slower but much bigger Samsung 850 EVO SATA SSD. All of the VM’s files are being hosted on this drive. The computer has an Intel Core i7-6700HQ processor with four physical cores and eight virtual cores. The system also has 32 GB of RAM. The virtual machines for each of these machines will be:

• 2 cores
• 4 GB RAM
• 40 GB Hard Disk using Fixed, not dynamic allocation

In order to ensure fairness and not some weird VirtualBox idiosyncrasies with dynamically resizing the disk I’m actually using the pre-allocated fixed disk size instead. I rarely do this on every day VMs but for this the accuracy of the speed is important. Similarly I want to make sure there isn’t periodic host system resource issues thus only giving each VM half of the total physical cores. Lastly because these machines aren’t going to be doing anything besides running updates and rebooting I felt that 4 GB of RAM was far more than adequate. Each of the machines are running comparable desktops as well. Gnome 3 when available is preferred. However some only had one version, KDE, and others supported mostly lighter weight versions. For those “lighter weight” versions I chose what I felt was the heaviest weight of them, Cinnamon.

For each operating system the machines were installed and then updated to the latest version using the standard command line package/update managers. This is what I’m considering the baseline of each system. On a regular basis, every few days or weeks, each of these VMs will be brought up individually. The command line updating process will be initiated. The system will then be rebooted. The things that will be tracked include:

• Time it takes to run the package manager various steps
• Time it takes to reboot the system after the package manager update
• The kernel version number after update
• The amount of disk space that the system is using

In order to keep things tidy the package manager systems will also go through an orphaned packages/unused packages cleanup process, if it is available. The actual times of the package manager stages are measured with the time CLI tool. The times of the reboot will be measured manually with a stopwatch. Some distributions have prompts during boot up which can add several seconds of delays to the process. I’m going to try to click through each as fast as possible. All of the systems have auto-login enabled so that the total time of a reboot will be from when I hit “enter” on the reboot command to when the desktop renders.

I’m also going to measure the performance of the various versions using Geekbench . I’m not expecting these to change much over time. In fact I’m finding Geekbench to be pretty noisy so I have to run it several times to get some good estimates.

Lastly, I will be doing VM snapshots before each update process so that if there is a problem, or I have a problem measuring a timing, I can revert the machine back and do it again. It’d be a shame to have a long run of information get noisy because of forgetting to hit a timer, or not having focus when a prompt is presented.

Distributions

I wanted to have a representation of the various rolling distributions that are available. I therefore did some research on the topic and found the below candidates. Some of these are proxies for bigger distributions. For example Manjaro is a more user-friendly Arch Linux distribution. This is a side project and I wanted to get this setup as fast as possible and to make the process as painless as possible. These were better alternatives to the base distributions. Yes I may live and breath Linux all day but I’m not one of the usual tinkerers. I’m the “I want it to ‘just work’” people. I generally go with graphical installers that will configure it for a workstation. I generally don’t muck with the partitions and just use whatever it recommends for me.

Below is the information of each distribution and the commands I’ll be using for the update process. First is a screenshot of all the VMs stacked up next to each other though:

Manjaro

One of the most popular rolling releases is Arch Linux , but I didn’t want to spend hours or days figuring out how to configure it. Manjaro is a popular packaging of the Arch distribution that is meant to be more approachable. I dabbled with it previously but nothing about it drew me in to try it more than that. The package manager for Arch, and therefore Manjaro, is called “pacman”. I can’t help but think of the video game and cartoon character. For copyright infringement reasons I imagine they have a story behind it sufficient to not get a cease and desist letter. Pacman handles all of the updates but the standard update command doesn’t clean up the cache or orphaned packages. I therefore have a series of three commands that are executed each time:

• pacman -Syu which does the update/upgrade step
• pacman -Sc which looks for the orphaned packages
• paccache -r which removes unused cached package files

The fully patched current system as of April 1st is running is running the 5.10.23 kernel and is using 7.7 GB of disk space. The time it takes to run the update sequence when there are no updates is 0.103 minutes (or just over 6 seconds). The time it takes to reboot when there are no updates is 0.383 minutes (23 seconds).

Solus

Solus was the only rolling distribution I ever really tried before. I had good experience with it and used it for part of my daily routine for about a year. At the same time nothing about it was so dramatically better that I ever considered moving away from my Debian/Ubuntu/Mint baselines. It is an independent Linux distro with its own package management system called eopkg system. Solus has an LTS and a “current” version where “current” is updated more frequently. The “current” system is the default and the one that is installed. The update process commands for Solus are:

• eopkg upgrade which does the update/upgrade step
• eopkg rmo which finds and removes orphaned projects
• eopkg dc which clears the eopkg cache of unused items

The fully patched current system as of April 1st is running is running the 5.11.9 kernel and is using 7.5 GB of disk space. The time it takes to run the update sequence when there are no updates is 0.073 minutes (or just over 4 seconds). The time it takes to reboot when there are no updates is 0.3 minutes (18 seconds).

OpenSUSE

OpenSUSE is the distribution that started this whole contest. It actually comes in both a traditional distribution form factor, called Leap, as well as the rolling distribution version called Tumbleweed. The names of each fit perfectly to the update style. The traditional distribution “leaps” ahead for upgrades while the Tumbleweed package rolls around picking up more updates just like a tumbleweed in the real world does. SUSE’s package management system is called Zypper. The commands executed for updates are:

• zypper dup which performs the update/upgrade process. “dup” is short for “distribution upgrade”
• zypper cc -a which performs the cache clean up of all data

The fully patched current system as of April 1st is running is running the 5.11.6 kernel and is using 5.2 GB of disk space. The time it takes to run the update sequence when there are no updates is 0.022 minutes (or just over 1.3 seconds). The time it takes to reboot when there are no updates is 0.67 minutes (40 seconds).

Redcore

I wanted Gentoo to be represented since it was one of the earliest Linux distributions I successfully played with almost two decades ago. It is also as close to a building Linux from scratch as you can get in an actual distribution. It is ironic that I didn’t want to go through all of the setup mishegas around Arch but I did spend four hours working through the Gentoo installation process. In the end it was becoming an impediment to getting this off the ground so I instead looked for something like Manjaro but for Gentoo. Unfortunately there isn’t a lot in that space. There was a now-defunct project called SabayonOS which has actually been undergoing some re-branding and a new life as MocaccinoOS . However it is currently in an alpha state. I did try getting it to install with only middling results. There is also Calculate Linux which showed some promise but the torrents were barely trickling when I finally got the installation of the distro I actually chose done.

So the Linux that will be representing Gentoo is Redcore Linux . I was hoping to have all Gnome 3 desktops but this one only comes in the KDE flavor. It actually just recently went through integrating some of the new Gentoo profile management capabilities that went from nice to have to necessary to stay based on Gentoo (link) . So this is actually a beta of the other side of their new rolling release system built on that. It supports Gentoo’s package management systems but it uses its own wrapper tool around them called sisyphus . All of the pre-packaged components are delivered in binary not source code form. Using sisyphus command line installing other packages you can choose between the source code/compile path or the pre-built binaries. It’d be interesting to see how much slower the source/build path would be but that’s not an exercise for this project. The update commands for Redcore are:

• sisyphus update to update the package cache
• sisyphus upgrade to perform the upgrade

I can’t find comparable package/cache removal commands for sisyphus so for the time being I won’t be doing any of that cleanup. This will potentially give it a big disadvantage in total disk usage as time goes on. It is already starting off relatively heavy in that department.

The fully patched current system as of April 1st is running is running the 5.10.25 kernel and is using 16.6 GB of disk space. The time it takes to run the update sequence when there are no updates is just over 1.5 minutes (94 seconds). The time it takes to reboot when there are no updates is 0.53 minutes (32 seconds).

Void

Lastly I wanted to get another independent rolling release distribution in the form of Void Linux . I’ve heard about it before but I’ve never played with it in practice. In the end I kept getting foiled with the final Grub boot loader installation step of the install. As I wrote above I generally don’t mess around with partitioning and instead just use the defaults therefore I’m a bit out of practice but not entirely. There is some text about this being a known problem with the installer but none of the steps that were suggested could overcome it. If some Void Linux lovers read this and have suggestions I’m all ears to try them and to add this to the experiment.

Starting Data

So with full installations and patches done late last night and early this morning today was the first day where the actual update cycle was executed. None of the operating systems had any updates so it created a great minimum execution and reboot time benchmark. I put all of that above but in compact tabular form:

Moving Forward

I’ll be running these update cycles on a regular basis. It will not be literally daily or even necessarily weekly. I will try to make it as infrequent as weekly but there are no guarantees. All of the OS’s will be updated on the same day in the same way and on their own though. I am publishing the data set, and potentially code that may eventually be used to process it, in an open Gitlab repository here under an AGPLv3.0 license for any code and Creative Commons Attribution (CC-BY) license for data and content. I envision regularly updating the blog here with charts, metrics, and commentary as well.