State Of Linux Usability 2020

TL;DR

We've carried out a series of daily tasks on TOP 20 Linux distros as well as Windows and macOS to test whether Linux has a chance to compete in daily use space. And the answer is - Yes since neither Mac nor Windows came on the top. There is still a very long way to go for all of them, however.

Motivation

I've been a Linux evangelist and a strong user (as in regular usage, not just for servers) for almost two decades since I've been introduced to Mandrake Linux with a mysterious penguin in the background.

That's why I've been the one who consistently annoys family, friends,
and colleagues until they're forced to give up their bellowed non-Linux life.

Of course, the change is often difficult since they are thrown in a never-before-seen world with never-before-understood rules. Being a CLI-first person, I often overlook what is so hard about it.

But over the years, I've received fewer and fewer complaints, and since there have been almost no cases of people going back unless it's for work, I've decided to do a proper test to see whether someone who's not a technician can live a proper Linux life.

The Subject

I wanted the subject to be someone with little-to-no technical skills/background as well as available for close supervision during the tests (and quarantine) to get accurate results.

So the background of the chosen subject is: a human, an illustrator, a long-time Mac-only user, spends 40% of work drawing on paper, 30% on Adobe suite and 30% on social media, saw me working numerous times and referred to the terminal as a "mysterious black window," never installed an OS before, required assistance towards installing Adobe software, not a native English speaker.

What I want to emphasize is that the subject is not a tech-friendly person at all.

So before we started I had to teach the subject a few simple things: what ISO is, how to use Etcher and how to select USB as a bootable device.

No further advice or assistance has been given.

Of course, we need to put a disclaimer here - the subject is human after all and humans learn over time so that test results in similar desktop environments or derivatives can be influenced by previous experience, but from the obvious dissatisfaction that has been witnessed, I can confirm that previous experience has not benefited too much.

Rules

General rules for making the results more relevant:

• Any outside support to solve a problem is forbidden, including other people, blogs, and search engines (except for the chrome installation task). Only out-of-box resources shall be used.
• Terminal use is also prohibited. Some apps provide terminal-based installation instructions - no one can expect newbies to rely on a "mysterious black box" to perform simple tasks.
• If a single task reaches the 20min mark, usage of a search engine is permitted.
• If a single task reaches the 60min mark, the entire distro is deemed unusable and the test group is considered to have failed.

Tasks

There are 16 tasks in total:

• Download ISO | tests web navigation UX, mirrors availability, clarity of choices, ISO size
• Flash ISO to USB | tests ISO size and file count
• Install encrypted system | tests installation process experience, encryption can be omitted only if it is not available
• Change wallpaper | tests basic personalization interface and options
• Play song on Spotify desktop | tests support for popular proprietary software, the initial internet connection
• Send message from telegram desktop | tests support for popular proprietary software
• Check email on google chrome | google chrome is rarely available in the app stores so this is the only exception where the installer can be downloaded, tests support for installing downloaded packages
• Create a PDF document with your name in it | tests initial tools availability for document editing
• Take a screenshot | tests initial tools availability for a screenshot
• Add a circle in the screenshot | tests initial tools availability for basic image editing
• Resize or crop the picture to 100x100px | tests initial tools availability for basic image editing
• Rename the picture to "my New Pic" | tests interface behavior for basic file adjustment
• Set picture as a user profile picture | tests interface availability for basic user personalization
• Install Steam | tests support for popular software
• Delete steam | tests ease of cleanup
• Power off the computer | tests power off clarity, availability, and pre-shutdown behavior

Each task is measured by the length of time it took to complete with additional subject scores for general UX (system intuitiveness, tool availability, etc.) and UI (out-of-box interface esthetics).

This will split evaluation into 4 total categories:

• The time required to set up (installing the system on the device)
• The time required to complete tasks (all other tasks apart installation)
• UX Score (subject to subject's subjective opinion)
• UI Score (subject to subject's subjective opinion)

Evaluation

Each system is ranked from the best to the worst in each category and given a point for the respective reverse position (1-22). If the scores are equal, the maximum is given for the respective placement and continues from the minimum (e.g. 18,17,16,16,16,13,12,11).

Tools

To be completely consistent with the test results, the same machine and network were used for all tests.

Machine: Thinkpad T480s (i5-8250U, 256GB Samsung 970 Plus NVMe, 16GB RAM).
For macOS installation late-2017 MBP (i5, 128GB, 8GB RAM) was used.

Network: A dedicated (all other devices were disconnected) 100mbps, pretty stable, always through LAN.

Data Set

The data set consisted of Windows and macOS as a control group, then picked up TOP 20 distros from DistroWatch over 12 months, removed Arch as it would require CLI knowledge and included Regolith instead (just because it's my go-to distro).

So the final list is macOS, Windows, MX Linux, Manjaro, Mint, Debian, Ubuntu, Elementary, Solus, Fedora, Zorin, Deppin, Antix, KDE Neon, OpenSUSE, CentOS, Pop OS, ArcoLinux, PCLinuxOS, Kali, ReactOS, Regolith.

I am aware that most of the tasks are software-specific activity, so the test is mostly about OS compatibility with day-to-day apps and means to obtain them.

What Went Wrong

Since all tasks were expected to go smoothly by default, I have only noted failures/pain points when using a particular system that I noticed when observing the subject.

I know there are workarounds for some problems, different paths and so on.
But if they have not been discovered by the subject in time, they are too hidden from the user and are therefore considered invalid.

So the results are as follows:

Failed Tests:

Antix

• Installation sometimes froze during install, if it succeeded it never booted (tried all grub options).

openSUSE

• Connecting to already attached LAN cable was hell, still required to manually "configure" the connection to use it, even if "automatic" mode was selected.
• The struggle to find Spotify ended up in multiple restarts and even the Discovery" app store failed to launch.

CentOS

• There was no progress at all with finding Spotify - no 3rd party repositories or snap or Flatpak were available.

ArcoLinux

• Software installation utility started crashing because of some file conflict, failed to find any way to repair through GUI.

Kali

• No software managing app was available and it was not possible to install one without CLI usage.

ReactOS

• Download was at ~70% when it hit the 60min mark (SourceForge auto-mirror & no issues with the network).

Successful Tests:

macOS

• Installation took a long time. As I understood it is a general issue with all macOS installs.
• When installing Telegram it was unclear what "drag to Applications" meant after mounting the downloaded image. Telegram from the app store was not being installed for some reason, no errors were shown.

Windows

• During the installation a permission to collect/user personal data was asked too many times (at least 8), it was annoying.
• When powering off the system, surprise - windows update was enforced without a visible option to bypass it.

MX Linux

• It was unclear how to find anything remotely close to the app store, the subject kept bumping into the software update page when finally found the "software installer".
• When installing anything live-action log with hard to comprehend errors was too overwhelming.
• It took a long time until the Flatpak" tab was discovered as a software source. It's confusing when it is the first time seeing this word.

Manjaro

• It was very confusing and time-consuming to find out that the desired apps are either in "AUR" or "Snap" sections, which from the names does not sound like places to find apps.

Mint

• Setting the profile picture when selecting the file redirected to /root by default, not /home/$USER so it took time to find the required file in the file system.

Debian

• It was a long fight but essentially to install a Spotify app without using CLI you have to install the Discovery" app store from which you need to install a snap plugin for gnome store" and then go back to gnome store to find Spotify.

Ubuntu

• The default ISO mirror location is not optimized. Custom mirror selection is not obvious.

Elementary

• Changing the background is impossible through a usual right-click on the desktop and no app handles remotely close to that, the journey to the system settings for this simple matter was a painful one.
• After downloading chrome .deb, trying to open it (double-click) does not bring the window of the installer to the front if it's already open (discovery app store) - this was especially frustrating.
• No default office suite is available

Solus

• Install process hung up on detecting local storage devices. After restarting and failing multiple times it finally worked (same settings).
• Spotify was available in the software app under 3rd party apps but after installing it did not appear anywhere so it was impossible to launch until "check for updates" was pressed and it finally appeared in the app list.
• After installing telegram all apps stopped working nothing would open until after the complete reboot of the system
• No native tools available for image editing since using LibreOffice Draw kept crashing.

Fedora

• Spotify was available only after enabling 3rd party repositories, installing updates and rebooting the whole system then installing discovery on which installing snap backend, rebooting again and fallback to search engine usage where rpm package was found, lots of loops until Flatpak backend was discovered for Discovery and Spotify Flathub URL was added (tutorial online)
• When opening system settings for 2nd time it stayed on the last open section (note - the system was restarted before), it was not clear how to go back to the settings menu to find where to change the user picture.

Zorin

• Very slow ISO download.
• Installing steam errored several times regarding "unmet dependencies", took a while to try to update the system, then reboot and only then to succeed in steam install.

Deppin

• It was confusing which app actually was meant for document creation and even then it was hard to find how to export to pdf.
• Steam was not in the app store, had to be downloaded separately, afterward when doing right-click on the app to uninstall, the app store would throw an error, eventually reopening the same .deb and then finding the "remove" solved the issue.

KDE Neon

• Very slow ISO download.
• No default picture editing software had to download separately.

PCLinuxOS

• Spotify was preinstalled but the software manager UI was so overwhelming that it took a long time to notice it.
• After installing Telegram it did not appear in the app list until the reinstall.
• Installing Steam required using a search engine to find that the only way to get it is to install Playonlinux first and only then to install Steam inside it.

Regolith

• It was a first time the subject used tiling window manager, no onboarding was available so all actions were incredibly slow

Final Results

I have decided to provide two sets of results - for overall score and score without installation evaluation for cases where the long-term use is more important than the time to get it running.

Bear in mind that this study only affected the general light-use cases of daily software consumption. Therefore, I will not give any further opinions as to why one is better than the other, and vice versa, since my personal insights may not be consistent with the general needs of everyday users.

The key problem tends to be a basic UX of software installation and management. Proper section naming or broader default repositories would solve most of the problems that day-to-day users are facing if sticking to Linux.

A thought regarding Windows/macOS score - there may be some issues with the way the test was designed, but I think that these results are pretty accurate.

Overall Score

Overall Score Without Install

Raw results are available here: linuxusability/2020/results

Final Thoughts

In the end, one of the key reasons why we have a handful of Linux distributions available is that each of them is designed to solve a specific problem and serves a specific function for a specific audience.

A lot of multi-purpose distros aim to be Windows-like or MacOS-like to ease the Linux transition. Although there is nothing wrong with it, the true strength of Linux lies in its uniqueness, not its similarity, and I think that this uniqueness could be "sold" to the consumer as long as it is presented/explained correctly (i.e. recall your first move from Symbian to Android or Windows to macOS).

However, I do believe that the Linux ecosystem as a whole is certainly ready to replace Windows / macOS not just for day-to-day users, but also for educational/corporate/government use.

On a final note, I'd like to encourage you to try using a Linux system for a week and if you're already a Linux user then go and make someone else use Linux for a week!

$ echo '#preachingpenguins'$ exit