Ubuntu Usability Test Report Bentley University: Human Factors in Information Design HF750: Usability Testing & Assessment December 15, 2008 Dan Fitek, Don Goetz, Karen Gosciminski, Sunil Muniraju, Allison Yale
Six individuals participated in a usability test of Ubuntu version 8.04.1, a free, open-source operating system.
The test aimed to assess Ubuntu’s usability and identify barriers that might hinder Windows’ users from converting.
Participants commented on Ubuntu while performing six tasks and completed a questionnaire afterward about their experiences.
Participants completed some tasks with relative ease (reorganizing files, using the word processing application). However, participants struggled to complete other tasks (connecting to a wireless network, changing the screen resolution).
A couple of participants reported interest in using Ubuntu, others hesitated to abandon their current Windows operating system and voiced concerns about application incompatibility and leaving their comfort zone.
This report presents summarizes, the project’s goals, our usability test methodology, and our findings and associated recommendations.
The most popular free Linux-based operating system available.*
Ubuntu (pronounced "oo-BOON-too“) is a Zulu word meaning “'Humanity to others,” or “I am what I am because of who we all are.” The Ubuntu distribution brings the spirit of Ubuntu to the software world.
Philosophy: “Every computer user should have the freedom to download, run, copy, distribute, study, share, change and improve their software for any purpose, without paying licensing fees. “
Ubuntu version 8.04.1 * http://www.desktoplinux.com/articles/AT5816278551.html
Test participants (n = 6) All participants used computers at least once daily to perform common tasks (e.g., emailing, word processing, browsing the internet). None of the participants had experience developing computer software or using Linux-based OS.
We presented participants with the following scenario to guide the test session.
Your college was selected as a marketing test site for a new operating system – Ubuntu. The company sponsoring the market test has agreed to give participants a laptop with the Ubuntu operating system to keep if the participants report back on their experience. You decide to sign up as a participant and you’ve been selected. You pick up your laptop and bring it home.
We identified frequent and important tasks based on 30-40 survey responses we received from our target user group regarding operating system use.
Directed tasks 1. You decide that the current display setting makes on-screen information hard to see. Decrease the monitor’s display size to the 1024 x 768 setting (time limit: 5 min) . 2. You are trying to get online to check your email and you realize there is no Internet connection. Connect wirelessly to the Internet with the following information (time limit: 10 min) : Network name: campus Password: 11bb22bb33bb44bb55bb66bb77 Encryption: open WEP, 128-bit 3. In your inbox, there is an email from a friend telling you that the most watched video on YouTube.com is great and that you should watch it. Go to YouTube.com to view this video (time limit: 15 min) . Participant will need to install an Adobe Flash plug-in to complete this task. (continued on next slide)
Directed tasks (continued) 4. You recently took a trip home to visit family for the holidays and you took some pictures. Three of the pictures are on the desktop. Create a new folder called ‘Family’ in an appropriate place to store the pictures. Move them off of the desktop and into this folder (time limit: 5 min) . 5. Earlier, you downloaded a few songs to the Music folder. Play one of these songs (time limit: 10 min). Participant will need to download a codec to complete this task. 6. Your grandmother sent you a batch of homemade cookies for your birthday and since you have terrible handwriting and your grandmother has poor vision, you want to type up a “thank you” letter in 20-point font. Type the following letter and save it onto the hard drive so that you can print it out later (time limit: 7 min). Dear Grandma, Thank you very much for the delicious cookies. I can’t wait to see you next month for dinner! Love, Pat
Participants provided basic information about their background and computer experience via a pre-test questionnaire (see Appendix C). During each task, we collected the following data directly into a Microsoft Excel spreadsheet:
Participants’ path while navigating through the operating system
Any assistance the moderator provided
Participants’ ease of use rating (1 = poor, 5 = excellent)
After completing the directed tasks, participants provided feedback on Ubuntu by completing a post-test questionnaire (see Appendix D).
There were certain tasks which participants had no problems performing.
There are some aspects of the Ubuntu operating system that could be improved.
Many of the usability issues described in this report fall within a couple overall themes.
The menu item labels and organization are very developer-centric. To increase the system’s usability, design the user interface with the user in mind. Given that the Linux and open-source philosophy may be at odds with this recommendation, try to strike a balance between developers’ needs and those of users.
Controls and settings do not directly support common user tasks as effectively as possible. This became particularly evident as participants attempted to establish a wireless connection (Task 2). From a system’s perspective, it is a matter of changing a network setting. However, the user is focused on his/her goal of connecting to the Internet and surfing the web. The system should be designed to support this goal from beginning to end. As it stands, the task is designed from a functional perspective and users are required to determine exactly which system changes they need to make in order to accomplish their goal.
The next group of slides describes some of the usability issues participants encountered while performing tasks with Ubuntu. To help the Ubuntu developer community address and prioritize our findings, we provide a brief recommendation and assign one of the following priority levels.
We considered the problem’s severity, frequency of occurrence during the test, and potential affect on the business goals when assessing each issue:
High : Something that prevented successful task completion or that caused significant user frustration or significant errors.
Medium : Something that interfered with task completion or led to user frustration. Fixing this should improve user productivity or operating system acceptance.
Low : Something that was a minor annoyance or resulted in minor delays in task completion. Typically considered a cosmetic or preference issue.
2 participants filled in the 802.1x wired networking form to try and connect to the wireless network.
They did not notice the word “wired” instead of “wireless.” The configuration window looks very similar to a wireless connection window so participants assumed it would enable them to connect to a wireless network.
Highlight the word “wired” in the menu option label by placing it first and prominently display the term “wired” once the window is open.
Place the menu option in a system administration menu, rather than presenting it on a typical Windows user’s path to configure a wireless network, since it is a tempting, yet incorrect path to take.
Participant quote: (after filling it out) “Looking at the screen now, it seems to be a wired network.” Connect to 802.1X protected wired network window
2 participants opened the Wireless Networks window, by clicking Edit Wireless Networks , and tried to enter the connection information. This window does not allow the user to connect to a wireless network, although it appears as though it should.
Rename this option.
Move the option to a less prominent menu location.
Many participants expressed confusion about the three main menu options’ labels: Applications, Places , and System.
4 participants looked for certain options under System > Preferences and S ystem > Administration . The items under these two menu hierarchies seem to overlap, and there is no clear distinction between them.
Seek opportunities to reword the primary menu options to more clearly communicate the options contained within the menus.
Avoid using technical jargon that average computer users might not understand.
Combine and consolidate menus to simplify the overall menu structure and eliminate redundancy.
Participant quote: “I’m not clearly understanding what the difference is between them.” Main menus (located in top left corner of desktop) System > Preferences menu System > Administration menu
Multiple network-related labels Priority: Medium
Participants seemed confused between the multiple network-related menu options which are available in various areas of the user interface.
Applications > Internet (n=3) Fig. A
Places > Network (n=1) Fig. B
System > Preferences > Network Proxy (n=3) Fig. C
System > Administration > Network Tools (n=1) Fig. D
Consolidate networking options into a single networking window with multiple tabs.
Figure A Figure B Figure C Figure D
Non-intuitive path for changing screen resolution Priority: Medium
4 participants had difficulty locating where to change the screen resolution .
2 participants right clicked on the desktop and arrived at the Appearance Preferences .
Include screen resolution setting in the Appearance Preferences window, which is accessible by right-clicking the desktop.
Place all display options in a dedicated, display settings window.
Rename the right-click menu option to say “Change Display Settings.”
Appearance Preferences window Participant quotes: “For Windows, normally I right-click and go for the desktop settings or something like that." "I right-clicked on the desktop because that is the general windows shortcut to access the display properties." Desktop right-click menu System > Preferences > Appearance menu option
When opening a music file, the video player opens as the default application.
When participants chose to install codecs, they are presented with both the video and audio codecs as options. This led 3 participants to initially select the wrong codec. One participant did not successfully install the audio codec.
One participant checked to see which of the two files had a higher popularity rating.
Present codecs based on the selected file type. For example, present users with only the audio codec when they select an audio file.
Reduce the prominence of the popularity ratings and present the codec’s file name, extension, and description in larger font.
Participant quote: “I’m going to try installing the first one and I’ll see what happens.” Install multimedia codecs window
Five of 6 participants said they were unlikely to replace their Windows OS with Ubuntu due to one or more of the following reasons (paraphrased):
Satisfaction and comfort with current operating system (n = 3)
No clear advantages over current operating system (n = 2)
Incompatibility with frequently used programs (n = 2)
Barriers to conversion
Likelihood of supplementing Windows with Ubuntu: Two participants would consider using Ubuntu in addition to their current Windows OS. Other participants were less enthused about Ubuntu in general and/or thought running two OS was unnecessary or overly complicated. Barriers to conversion (continued)