Web Site Usability Test - Client Report - Victorian Deaf Society (Ver 1....

Prepared by: ADGS Consulting - Monash University
Usability Test Client Report
Project:
Website Usability Assessment
Client:
Victorian Deaf Society (Vicdeaf)
Team:
Adrian Caceres
Greg Ortega
Steve Remington
Di Zhang

ADGS Consulting - Monash University Usability Test Client Report
Victorian Deaf Society (Vicdeaf) Website Usability Assessment
Date: 12/02/2014 Version: 1.0 Page 2
Commercial In Confidence
DOCUMENT INFORMATION
Document Details
Document Title:
Usability Test Client Report
Project Name
Website Usability Assessment
Document Version:
1.0
Document Date:
12/022014
File Name:
Client Report - Victorian Deaf Society (Ver 0.3).Docx
Document Authorisation
Prepared by:
Adrian Caceres; Greg Ortega; Steve Remington; Di Zhang
Authorised by:
Manoj Kathpalia; Julie Fisher
Prepared for:
Ms Lidia Risicato - Victorian Deaf Society (Vicdeaf)
Company Details
Name:
ADGS Consulting - Monash University
Address:
Monash University Caulfield Campus
Caulfield East, Victoria, 3165

EXECUTIVE SUMMARY
A usability test was conducted on the Auslan course enquiry and enrolment, and online donation functionality of the Vicdeaf website. These two areas of functionality were chosen after a discussion with Michael Parremore about concerns Vicdeaf had about the website. After a review of the website content and functionality two user scenarios and a set of tasks related to the areas of concern were developed.
The usability tests were conducted in the period 03/02/2014 to 08/02/2014. The test protocol required the participants to complete a small number of tasks related to each of the scenarios. The tests were observed by members of the project team. At the completion of the test the participants completed a questionnaire that sought their feedback on the overall usability of the functions tested and suggestions on how the functions of the site could be improved. Six participants were used for the test. The participants were members of the Victorian general public known to the project team members. Five of the participants completed the usability test for both scenarios and one participant only completed the tasks for the Auslan course scenario.
The test results indicated that the participants’ opinion of the usability of the online donation functionality was moderately positive (i.e. 3.6 out of 5.0), while the opinion of the Auslan course functionality was slightly positive (i.e. 3.4 out of 5.0). The responses to open ended questions about the online donation process showed that participants thought the process was generally easy to use however the information on the left side of the donation page is difficult to read, and that it would have been better if there was more obvious access to information about how Vicdeaf uses donations. The responses to the open-ended questions about the Auslan course information and enrolment process showed that participants found it difficult to find information about courses because the information is spread across many pages. Moreover, the layout of the course enrolment page made it difficult to find the particular course they wanted to enrol in.
Based on the test findings and best-practice website usability guidelines a five key changes to the Vicdeaf website are recommended. Additional minor recommendations are presented in the body of this report. The key recommendation are outlined below.
Recommendations – Auslan Course Registration
 Create a single Auslan course information page that outlines the course levels, duration, typically available days / time, costs and available locations. The information should ideally be presented in the form of a set of “frequently asked questions” about Auslan courses.
 Reorganise the Auslan course timetable so that it clearly shows level, location, date, time of day, and open or closed status of a course and present the information in a tabular format that make it easy for the user to scan. The format of the list should allow users to sort and filter the courses by each of the categories.
 Simplify the Auslan course main page so that it only has three links below the video and introductory text, namely; “Course Information”, “Find a Course and Enrol” and “Gift Vouchers”
Recommendations – Online Donations
 On the home page near the donation link add a link titled “Learn More” which will take the user to a page that briefly outlines how donations are used. The “Learn More” page should include a “Donate Now” link that takes the user to the existing online donation page.
 Change either the font colour to the tone of the background image in the donation page to increase the readability of the information on the left side of the page.

TABLE OF CONTENTS
EXECUTIVE SUMMARY ................................................................ 3
TABLE OF CONTENTS ................................................................. 4
LIST OF TABLES ...................................................................... 4
1 Introduction .............................................................. 5
1.1 Purpose .................................................................... 5
1.2 Outcomes ................................................................. 5
1.3 Scope ....................................................................... 5
2 Description of the Test ............................................... 5
3 Summary of Results and Findings ................................ 6
3.1.1 Usability Test ............................................................. 6
3.1.2 Heuristic Evaluation .................................................... 7
3.1.3 General Observations ................................................. 7
4 Recommendations ..................................................... 8
Appendix 1 – Usability Test Detailed Results ......................... 10
LIST OF TABLES
Table 1: Scenarios and task used in the usability tests ............. 6
Table 2: Recommended usability changes and benefits ............. 9
Table 3: Summary of participant demographics ..................... 10
Table 4: Summary of participant experience statement responses ............................................................................. 10
Table 5: Summary of key themes mentioned in participants’ responses to open-ended questions ........................... 11
Table 6: Summary of results of heuristic evaluation of the Auslan course enrolment functionality based on Nielsen’s 10 Usability Heuristics for User Interface Design .............. 13

1 INTRODUCTION
1.1 Purpose
This document briefly describes the usability test on the Auslan course enquiry and enrolment and the online donation functionality on the Vicdeaf website, summarise the results and findings of that usability test and document recommended changes that will improve the usability of those areas of functionality.
1.2 Outcomes
The outcome of this document is Vicdeaf receiving a set of actionable recommendations that will improve the usability of the Auslan course enquiry and enrolment, and the online donation part of the Vicdeaf website.
1.3 Scope
The scope of this project is limited to two specific use scenarios on the Vicdeaf website, namely; Auslan course enquiry and enrolment, and online donation parts of the Vicdeaf website. All other parts of the website were not tested in this project.
2 DESCRIPTION OF THE TEST
The usability test method used for this project was a hybrid of the hallway testing and think-aloud protocols of usability testing. The hallway testing approach recruits a small random group of five or six participants who loosely match the profile of the core group of users of the website. Research has shown that the hallway testing approach can identify approximately 80% of the usability problems with a given website scenario. The think- aloud protocol is a variation of the typical usability testing approach in which the participants are asked to talk through the actions they are performing on the site. The think-aloud protocol helps the observer more clearly understand what the user is looking at, thinking, doing, and feeling as they go about the task.
The usability test involved two scenarios and related tasks. The choice of the functionality on the Vicdeaf website on which the scenarios were based resulted from a phone meeting discussion with Michael Parremore on Friday 30 January. Michael said that a recent review of web search analytics showed that many users were finding the Vicdeaf Auslan course page after searching for terms related to learning sign language in Melbourne, but only a small percentage of those users were enrolling in a course. Michael suggested that it would be good to test if there were any usability problems with the Auslan course part of the site. Michael also suggested that online donations were a growing source of income for Vicdeaf and that it would be useful to know if there are any usability problems in that part of the site.
After a review of the website functionality two scenarios and related tasks corresponding to the two areas of concern were developed. The specific text of the scenarios and the tasks is outlined in Table 1.
At the completion of the scenario tasks the participant was asked to fill in a questionnaire. The questionnaire was divided into three sections, namely; a demographics questions section that captured a few personal attributes of the participant, a website experience statements section that allowed the participant to indicate their overall rating of the usability of the site for the scenario, and an open-ended questions section that allowed the participant to give his or her opinion about and offer suggestions to improve the site.

Scenario Tasks
Online Donation
You have a friend who is deaf. In a recent conversation she told you how much the services provided by Vicdeaf have helped her through her life. You recently received a good pay rise at work and decided that you would regularly donate part of the pay rise you received to a charity that you like. After hearing about the work that Vicdeaf has done for your friend you decide to choose them as the charity you will support.
1. Find out how Vicdeaf uses the money from donations.
2. Establish a monthly donation of $35 using a credit card.
Enrolling for Auslan Course
A new colleague at your workplace is deaf and primarily communicates using “sign language”. You would like to communicate more effectively with your new work colleague and as a result you have decided to enrol for a course in sign language. Your work colleague told you that in Australia sign language is called “Auslan” (Australian Sign Language) and that Vicdeaf regularly runs Auslan courses in various suburbs in Melbourne. You decide to visit the Vicdeaf website to find out about and enrol on an Auslan course.
1. Find out what is the duration of the Auslan courses and how much do they cost.
2. Find out if there are any courses specifically related to learning Auslan to communicate with a work colleague. Make an enquiry to find out more information about the course.
3. Register for an Auslan class that runs on a weekend at a venue closest to where you live.
Table 1: Scenarios and task used in the usability tests
The usability test were conducted in the period 03/02/2014 to 08/02/2014. The tests were observed by members of the project team. Six participants used for the test. The participants were members of the Victorian general public known to the project team members. Five of the participants completed the usability test for both scenarios and one participant only completed the tasks for the Auslan course scenario. A breakdown of the participant demographics is shown in Table 3.
3 SUMMARY OF RESULTS AND FINDINGS
3.1.1 Usability Test
The ratings of the usability experience for each scenario showed that the participants, overall had a more positive experience with the online donation scenario compared to the Auslan course scenario. The key points from the experience ratings were:
 In both scenarios the participants’ responses were moderately negative about the screens generally being too much information on the screen. This indicates that there is an opportunity to review the amount of content on these the pages.
 The participants’ responses were slightly positive about the ease of use and visual appeal of the Auslan course pages. This indicates that there is an opportunity to improve the clarity of the content and layout of these pages.
For a more detailed breakdown of the usability experience ratings refer to Table 4 in Appendix 1.
The participants generally had positive comments about the usability of the online donation scenario. It was generally considered easy to make a donation. A few users, however, commented on the difficulty they had finding information about how Vicdeaf

uses money from donations. On participant summarised it thus: “You must first make the decision to donate to understand / obtain information on how Vicdeaf use the donations”. The participants’ comments about the Auslan course scenario were on the other hand more generally more negative. The common themes raised in the comments were:
 The information about the Auslan courses is unnecessarily spread across too many pages, and sometimes pages on a similar topic present inconsistent information.
 It was difficult to find out if the desired type of course, is available at the desired location on the desired day.
 Even when the desired course was found the participant often clicked on the registration link only to find that the course was full.
The general feeling of the participants about the usability of the Auslan course scenario was summarised thus: “Couldn’t find all the information together. Not all links were working”. A summary of the themes raised by the participants’ responses to the open- ended questions is presented in Table 5 in Appendix 1.
3.1.2 Heuristic Evaluation
In light of the usability issues identified in the Auslan course scenario a heuristic evaluation of the Auslan course scenario pages was conducted to provide an alternate and wider ranging assessment of the usability issues on the relevant web pages. The results confirmed the findings of the participants’ usability test. In particular the heuristic evaluation identified the following key issues:
 Related information (e.g. course types, location and cost) is spread over many different pages which requires the user to rely too heavily on remembering what is on each different page to fully understand the details of the course offered.
 The current course list page does not advise the user if the course is full until the user clicks on the course resulting in unnecessary clicks and potential frustration for the user.
 The course registration page allows the user to submit the page before all required field are filled in. A better user experience would be achieved if the form submission is prevented until all required fields are filled in.
The fully documented results of the heuristic evaluation, including severity rating of the issues found is presented in Table 6 in Appendix 1.
3.1.3 General Observations
While watching the participants conducting the usability tests the observers identified a small number of technical issues with the website that should be addressed. The key items are outlined below:
 When the user is directed to the online donation and course registration pages provided by bCommerce the links in the top menu bar and the search field return an error page from bCommerce web server when clicked.
 A number of links on the Auslan course pages (e.g. course timetable link, and some of the links in the page “breadcrumb” trail) redirect the user to pages served from the Bliss Media staging web server.
 The page “breadcrumb” trail on some of the Auslan course pages do not accurately reflect the position of the page in the site hierarchy.

4 RECOMMENDATIONS
Based on the findings outlined in the previous section and best-practice website usability guidelines we make the recommendations as outlined in Table 2 below. Recommended Change Benefit of Change Scenario: Enquiring About and Registering for Auslan Courses
 Create a single Auslan course information page that outlines the course levels, duration, typically available days / time, costs and available locations.
 Present the information in the form of a set of “frequently asked questions” about Auslan courses. Using this format would present users with information is a format that they are familiar with from other websites.
 Presenting information about Auslan course to user this way means that it is possible to find all information from a single page.
 A single course information page will provide a more targeted page for search engines to land on when search terms such as “learn sign language Melbourne” are used.
 Reorganise the Auslan course timetable so that it clearly shows level, location, date, time of day, and open or closed status.
 Present the information in a clear tabular format that make it easy for the user to scan. Ideally the list of courses should be (a) sortable by the user on each of the categories, and; (b) filterable by the user (e.g. only show me level 1 course on the weekend).
 Do not show other course information on this page, but rather provide links to the above course information page.
 Presenting the course timetable in this format with sorting and filtering options will make it easy for the users to find and register for exactly which course they need.
 If it is easy for users to find the course and then register then there will be a reduced level of user frustration and site abandonment.
 Present the information about and purchase of the Auslan course gift vouchers on a separate page and remove information about gift vouchers from the course list page.
 Presenting the Auslan course gift voucher information on a separate page will make it clear that a person is paying for a course sometime in the future rather than enrolling in a specific course.
 Reduce the number of image links below the welcome video and introductory text on the Auslan course. The reduced set of image links should be:
o Course Information
o Find a Course and Enrol
o Gift Vouchers
 Reducing the image links on the Auslan course main page will not only align the links with the above recommend changes, but also simplify the page and bring it more in line with how most users would approach obtaining information about and enrolling for courses. Scenario: Making and Online Donation
 On the home page near the donation link add a link titled “Learn More” which will take the user to a page that briefly outlines how donations are used. The “Learn More” page should include a “Donate Now” link that takes the user to the existing online donation page.
 Adding this page will allow potential donor to quickly find out how their money will be used without the perceived pressure of the online form. If the user likes what they read he/she can in one click go directly to the donation page.

Recommended Change Benefit of Change
 Change either the font colour to the tone of the background image to increase the readability of the information on the page.
 Making this change will make it easier for a potential donor find out more detail about how donations to Vicdeaf are used. Moreover, research has shown that a user will read more information on a web page and are more likely to complete a transaction if the page is well laid out easy to read. General Site Issues
 Change the Vicdeaf site main menu bar that is displayed on the bCommerce e- commerce pages (e.g. course registration and donations) so that the links return the user to the correct pages on the Vicdeaf site.
 Correct the small number of links on the site that are pointing to the Bliss Media staging server.
 Update the format of the “breadcrumb” trail links so that it more accurately reflect the position of the page on the site.
 Making these changes, particularly correcting the e-commerce links will reduce the chance of users seeing unnecessary and confusing error messages. Moreover eliminating these errors will increase the trust that the user has in the Vicdeaf site generally but in the e-commerce parts of the site specifically.
Table 2: Recommended usability changes and benefits

APPENDIX 1 – USABILITY TEST DETAILED RESULTS
Participant Demographic Attributes Scenario Auslan Course (Count) Online Donation (Count)
Gender
Male (2) | Female (4)
Male (2) | Female (3)
Age Group
18-34 (3) | 35-49 (3)
18-34 (3) | 35-49 (2)
Internet Experience
Some (1) | Extensive (5)
Some (1) | Extensive (4)
Vicdeaf Experience
None (5) | Limited (1)
None (5)
Table 3: Summary of participant demographics
Participant Experience Statements Scenario Auslan Course (Average Score) Online Donation (Average Score)
1. I found the layout of the website easy to understand. 3.2
4.0
2. The design of the website is appealing. 3.2
4.0
3. When assessing the information, the number of steps required was acceptable.
3.8
4.0
4. It was easy to navigate through the website.
3.5 3.4
5. I did not get lost when completing the task.
3.8
3.6
6. The information of the page was effectively laid out.
3.3
3.8
7. Generally there was too much information on the screen. 3.2 2.6
Note: Shaded scores are two worst scores for each scenario.
Table 4: Summary of participant experience statement responses

Open-Ended Question Auslan Course Online Donation
1. Were you able to easily complete the task set? If yes, what aspect(s) of the site helped you complete the task? If no, what aspect(s) of the site made it difficult complete the task?
 Yes
o It is easy to find course related information the home page
 No
o Purpose of gift voucher is confusing
o Auslan page very confusing. There is a lot of information in spread across many places
o Hard to find a weekend course
 Yes
o Easy to find donation and make an online donation
 No
o It is not easy to find how donation money is used without directly to the donate page
o Could not find link to donation page on menu
2. What was the one element of the website that you feel made it easier for you to complete the task(s)?
 There was none
 Radio buttons to select payment options
 The main Auslan page is specific
 The donation image link was easy to find
 The site menu is useful after the home page
 Filling donation details was easy
3. If you could change one thing about the website that would make your experience better, what would it be?
 Improve layout of the course timetable
 Group related course information together
 Reduce size of banner on home page as it distracts from main image links below it
 Focus less on appearance more on content
 Change white text on light coloured background on donation page
 Too many distracting colours on the home page
 Make more obvious on home page link to information on how donations are used
4. Were there any elements of the website that caused confusion or slowed down your retrieval of information when completing the task? If yes, please explain what it was and why.
 Inconsistent link destinations
 Repeated but inconsistent information
 The menu links on all four edges of the pages
 Lack of clarity on status of courses
 It is not clear what types of course are available
 Distracting content on the home page
 The frequency options
 It is possible to set an expiry date that in the past
 No instructions on the donation page
 Not clear on pages other than the donation page how donations are used.
5. Are there any other comments or suggestions about the website that you would like to add?
 Move main link images on home page to above banner
 Clearer presentation of course information
 Make the site less fancy
 Search option in multiple place on home page
 Inconsistent colour of donate image on home page and other pages.
 No images on the home page makes it feel impersonal
Table 5: Summary of key themes mentioned in participants’ responses to open-ended questions

Nielsen’s Usability Heuristic Issue Location Issue Details Issue Severity
 Visibility system of status
 Course registration form
 Auslan courses list
 Form field error messages should be displayed next to the fields that causes the error, not as a list at the top.
 User should not be able to click on a course that is not available or full.
1
2
 Match between the system and the real world
 Grammar mistake, “To register in class”, reads as one can register yourself on the day of the course.
1
 User control and freedom
 Information for a specific Auslan course and location
 The breadcrumb trail does not show the previous page
2
 Consistency and standards
 When an Auslan course is full
 When an Auslan course is full
 When clicking on “back to events” link, the page layout of the course list changes. Register and enrol buttons confuse the user
 The contact us and make an enquire page have a different page layout.
2
2
 Error prevention
 Course Registration form
 Auslan course list
 Proceed to payment button should not be active if the fields are not filled in
 There is no confirmation dialog before registering and payment. When alternate payment options.
 There is no map for the new St Kilda course location.
2
2
1
 Recognition rather than recall
 Enquire or contact about a specific course that is full or not available
 The form should be pre filled with the course information that the user is enquiring about.
2
 Flexibility and efficiency of use
 No issue identified
 Not relevant to this case
0
 Aesthetic and minimalistic design
 Button colours for new course locations and gift vouchers should be different because they serve different purposes.
1
 Help users recognise, diagnose and recover from errors
 Form field error messages should be displayed next to the fields that causes the error, not as a list at the top.
1

Nielsen’s Usability Heuristic Issue Location Issue Details Issue Severity
 Help and documentation
 Auslan Course Information
 Message reads click “Enquire” for more payment options, but there is no enquire button or link.
 The course list should not show courses that are not available, or that are full.
 Related information (e.g. course types, location and cost) is spread over many different pages
2
1
3
Issue Severity Rating
0. There is no usability problem.
1. Cosmetic problem only: Need to fix only if time permits.
2. Minor usability problem: fixing this should be given low priority.
3. Major usability problem: important to fix, so should be given high priority.
4. Usability catastrophe: imperative to fix this before product can be released
Table 6: Summary of results of heuristic evaluation of the Auslan course enrolment functionality based on Nielsen’s 10 Usability Heuristics for User Interface Design

Web Site Usability Test - Client Report - Victorian Deaf Society (Ver 1....

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Web Site Usability Test - Client Report - Victorian Deaf Society (Ver 1....

Similar to Web Site Usability Test - Client Report - Victorian Deaf Society (Ver 1.... (20)

Web Site Usability Test - Client Report - Victorian Deaf Society (Ver 1....