• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
eReading Experience
 

eReading Experience

on

  • 592 views

The School of Informatics hosted technology employers (Cisco, Oracle, Disney Animation Technology, Intuit, etc) for a day to better express what IU's human-computer-interaction design program. As a ...

The School of Informatics hosted technology employers (Cisco, Oracle, Disney Animation Technology, Intuit, etc) for a day to better express what IU's human-computer-interaction design program. As a member of the committee to design this interaction, we thought there was no better way to see our students in action, then to have our guests design for experience with us. This is the playful, situated, experiential design problem the committee created (and I put together and presented).

Statistics

Views

Total Views
592
Views on SlideShare
592
Embed Views
0

Actions

Likes
0
Downloads
4
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    eReading Experience eReading Experience Document Transcript

    • ABC Center User Testing Report Client: ABC Center for Research Website: http://www.abccenter.org Contact: John Walton Consultants: Bazler, Jae Kim, Erik Brook, Heather Beery, Lora Ramseier Date: January 15, 2007
    • Table of Contents ABC CENTER USER TESTING REPORT........................................................................................................... 1 TABLE OF CONTENTS .......................................................................................................................................... 2 EXECUTIVE SUMMARY ....................................................................................................................................... 3 STUDY DETAILS ..................................................................................................................................................... 4 PURPOSE OF STUDY ................................................................................................................................................. 4 METHOD .................................................................................................................................................................. 4 PARTICIPANTS ......................................................................................................................................................... 4 PROCEDURES ........................................................................................................................................................... 4 TASKS ...................................................................................................................................................................... 5 RESULTS .................................................................................................................................................................. 5 TASK PERFORMANCE.......................................................................................................................................... 7 OBSERVATIONS AND RECOMMENDATIONS................................................................................................ 9 SCREEN LAYOUT/DESIGN ........................................................................................................................................ 9 TERMINOLOGY ...................................................................................................................................................... 11 CONTENT ORGANIZATION ..................................................................................................................................... 11 NAVIGATION ......................................................................................................................................................... 12 OPERATING SYSTEM AND BROWSER ISSUES ......................................................................................................... 15 POST-TEST SATISFACTION RATINGS........................................................................................................... 16 SITE SATISFACTION RATINGS ................................................................................................................................ 16 SATISFACTION QUESTIONNAIRE – POST-TEST QUESTIONS ................................................................................... 17 Question 1 – What do you consider the most valuable aspect of the system? ................................................. 17 Question 2 – What is the biggest problem with the system?............................................................................ 17 Question 3 – Additional comments .................................................................................................................. 17 APPENDICES.......................................................................................................................................................... 19 APPENDIX A – TESTING PROTOCOL SCRIPT........................................................................................................... 19 APPENDIX B – PARTICIPANT CONSENT FORM ....................................................................................................... 20 APPENDIX C – VIDEO RELEASE FORM................................................................................................................... 21 APPENDIX D – SATISFACTION QUESTIONNAIRE .................................................................................................... 22 APPENDIX E – SPACE UTILIZATION ....................................................................................................................... 23 APPENDIX F – IMAGE FONTS ................................................................................................................................. 24 APPENDIX G – CONTRAST BETWEEN TEXT AND BACKGROUND ........................................................................... 25 APPENDIX H – TERMINOLOGY ............................................................................................................................... 26 APPENDIX I – LINK FORMAT ................................................................................................................................. 27 APPENDIX J – MOUSE-OVER EFFECT..................................................................................................................... 28 APPENDIX K – LINKING TO PREVIOUS PAGE ......................................................................................................... 29 APPENDIX L – OPERATING SYSTEM AND BROWSER ISSUES .................................................................................. 30 APPENDIX M – BROWSER COMPATIBILITY ISSUES ................................................................................................ 31 User Experience Group Indiana University 2 of 31
    • Executive Summary In March 2005, a usability study of the ABC Center website was conducted with fifteen participants in the Bloomington, Indiana area. Results showed that users were generally satisfied with the amount and quality of the content available on the ABC site. However, a number of usability issues were identified that hindered locating information, and the site was perceived as being unprofessional in appearance. One of the greatest concerns for usability in the ABC site was the lack of a consistent, reliable navigation system. Users were often left without navigation options to return to the previous page or to the main menu, and were forced to use the browser buttons to navigate the site. This issue was compounded by the fact that some links took the users unexpectedly out of the site. The overall unprofessional appearance of the site was another concern. When searching for scientific research, the perceived credibility of a site will play a large role in whether people are willing to use the site and trust the information provided. Participants expected the images used in the site to be of higher quality, page space to be utilized more efficiently, and the content to be formatted correctly (capitalization, text alignment, etc.). Even though these issues may seems somewhat minor, the combined effect reduces the sites credibility in the users’ eyes. Other usability issues included linking to software-specific files without warning and unpredictable content presentation format. This report describes the purpose and methods of the study, and provides a summary of the observations made from each testing session. Recommendations are made for the ABC Center web development team to consider as they prepare for the redesign of the website. User Experience Group Indiana University 3 of 31
    • Study Details Purpose of Study The ABC Center website is a resource for students to find research studies on the web. The site provides a variety of features including a research article database, links to other online research sources, and a directory of researchers. A new site design is going to be introduced in July, 2005, and the ABC Center web development team wants to explore opportunities to improve the website by obtaining user feedback. As part of the usability evaluation for the ABC Center, user testing was conducted in March of 2005, in Bloomington, Indiana. Method Usability testing methodology1 was used to evaluate the website. This approach involved having authentic users perform authentic tasks using the system, while evaluators observed and recorded their actions and comments. Users were asked to perform a think-aloud protocol2 to help the evaluators understand their behaviors and gain insight into the design of the website. Sessions were performed on an individual basis with each session lasting approximately 1 to 1 ½ hours. ABC Center provided users with $50 gift certificates in return for their participation. Following the last session, qualitative and quantitative data were analyzed and recommendations for redesign were made. Participants Participants recruited for the testing sessions were identified and recruited by the UXG team. An effort was made to target participants who fit a wide range of user profiles due to the heterogeneity of the site’s potential users. The goal was to recruit a sample of 16 participants to achieve adequate representation across the following four demographic categories: 1. Gender – approximate equal split between male and female 2. Age – three age ranges (20-39, 40-59, 60+) 3. Computer experience – novice (little to no computer use), intermediate (office applications, web), or expert (complex applications and/or programming) 4. Experience with research websites A total of 15 participants were tested including 7 men and 8 women. Five participants were in the 20-39 year age-range, six were in the range of 40-59 years of age, and four participants were over 60. Procedures Participants were read an introductory protocol at the beginning of the testing session (see Appendix A for script) and asked to read and sign two consent forms. The first consent form (see Appendix B) is a standard consent form indicating the participants’ understanding of what will be expected of them during the sessions and their willingness to participate. The second consent form (see Appendix C) is a video release form indicating the participants’ awareness that the sessions are videotaped and granting their permission for the video to be used in the 1 Dumas, J. S., & Redish, J. C. (1993). A practical guide to usability testing. Norwood, NJ: Ablex. 2 Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (Revised ed.). Cambridge, MA: MIT Press. User Experience Group Indiana University 4 of 31
    • evaluation and reporting of results. Participants were given a copy of the consent forms for their records. Each participant completed 10 tasks using the abccenter.org website. Tasks were completed one at a time and were recorded as either 1) success with ease, 2) success with difficulty, or 3) failure to complete the task. Success with ease included those tasks that were completed on the first or second try. Success with difficulty included tasks that participants adequately completed but more effort and persistence was needed. For example, trying numerous paths to locate necessary information before finding the correct screen. Failure to complete a task was given if the participant gave up on the task or failed to locate the correct information. In some “failure” cases, participants may have been satisfied that they completed the task; however, the correct information was not located or portions of the task were not completed. Following the session, the users were asked to rate their satisfaction with the system using the Survey of User Satisfaction developed by Digital Equipment Corporation (see Appendix D). Tasks were performed using Internet Explorer 6.0 and the Windows XP operating system on a Dell, Pentium 4, 3.80 GHz computer. Tasks The ABC Center team developed a draft list of 10 tasks for testing based on key site information and features. UXG consultants reviewed and refined the tasks in order to create appropriate scenarios for user testing. Tasks were presented in random order to eliminate order effects from influencing the testing results. Results The results of the study have been divided into four sections: 1. Task performance (e.g. successes and failures) Tasks are listed and the number of participants experiencing successes and failures are noted. 2. Observations and recommendations The results of the study are presented in a table format. The table includes three columns of information: Observations, Interpretations, and Recommendations. Each column is described in more detail below: • Observations – an objective description of participants’ actions and comments during a session. • Interpretations – a proposed explanation for participants’ observed behaviors and justification for recommendations based on known design principles and rules. • Recommendations – suggestions for maintaining aspects of the current design that work, changing aspects that are problematic, and considering alternative possibilities for presenting information to the user. The data within this table are organized into categories that represent the major types of issues raised during the study. The categories include: User Experience Group Indiana University 5 of 31
    • • Screen Layout/Design – Items concerning the placement of elements on the screen, use of screen space and opportunities for modifying items to increase user understanding of presented information. • Terminology – Items concerning the user’s understanding of the language used on site. • Content Organization – Items concerning the organization of information within the site. • Navigation – Items concerning the users’ ability to maneuver throughout the site. • Operating System and Browser Issues – Items concerning compatibility and consistency of site presentation across major computer configurations. 3. Post-test satisfaction ratings and comments The results of the SUS (System Usability Scale) questionnaire and a brief description of how the overall SUS scores are calculated is presented. Participants’ comments are included. User Experience Group Indiana University 6 of 31
    • Task Performance Task Success Success with Fail with Ease Difficulty 1 You are looking for employment at the ABC Center. Find out if 12 2 1 there is anything open at this time that you could apply for. Are you interested in the position? 2 You are interested in taking an online course offered by the ABC 11 2 2 Center. Find a course in “Referencing in APA Style.” 3 You are writing a research paper, and you realize that you are 7 3 5 missing the author information for one of your sources. Find the author of the research article “Intercultural Discourse.” 4 You have been asked to contact a researcher by your 4 7 4 teacher/professor. Find the contact information for “Ellen Miller.” 5 You are interested in visiting the ABC Center in person. Where is 13 2 0 the ABC Center located? 6 With so many online research databases to choose from, you 4 6 5 wonder how they compare. Find the discussion forum where the users rate them. 7 After spending some time browsing through the ABC Center 2 6 7 research database, you found some articles that you might need in the future. After logging in, find and save the research article “Benningham DNA Study” in your “Online Bookmarks” for later retrieval. 8 You have heard that ABC Center offers several workshops 7 5 3 throughout the year. What are the titles of the two that are offered in May (2005)? 9 ABC Center offers several free research aid software for 12 1 2 download. Find and download “Ace Reference Tracker 2”. 10 Thank you for visiting the ABC Center website. Can you please 10 3 2 use the online form to rate us? User Experience Group Indiana University 7 of 31
    • The graph above illustrates the percentage of users who accomplished the tasks with ease (easy), with difficulty (hard), or failed. The tasks that caused users the most difficulty were locating the contact information for researchers (task #4) and using the Online Bookmarks feature (task #7). User Experience Group Indiana University 8 of 31
    • Observations Interpretations Recommendations Observations and Recommendations Screen Layout/Design The copyright information on Using the same colored text and 1. Use black or other dark the homepage is invisible to the background does not provide any colored text for this users since it is white text on contrast for the text to be visible, information. white background. unless they are drag-selected with the mouse by accident. At least three different font and When the same information is 2. Standardize the font and color schemes are used in the presented in different formats, it color format if the site to graphically represent the causes inefficiency in user organization does not have organization’s name. behavior because they have to re- a graphical logo. If it does, interpret the data. Furthermore, a replace the current titles chance to reinforce the with the logo. organization’s image/identity is lost. The organization’s name is The organization’s name is not 3. Change the layout of the difficult to read. easy for users read. It looks like a organization’s name. list including; “the center, education, research, retailing”. The information currently The homepage does not help 4. Remove the current home displayed on the homepage is users understand what the site is page and make the “ABC not helpful to users. For about or where they can navigate. Center Menu” page the example, the organization’s site’s home page. name is shown twice, the picture displayed is not associated with the center, and there is no site menu or navigation on this page. The “Research Time Line” link Content should be presented in a 5. Change the font of the link is presented in a different font consistent graphical format, to be consistent with the than the other links in the same especially if the items belong to the other links in the list. list. same list. Any deviation would suggest that the particular link is different from the others in content, importance, etc. Too much screen space (80+ The top of a web page is valuable 6. Move the page title image pixels) is wasted on the top of and draws the attention of the up to reduce the white the menu page (see Appendix users. The current use of the page space. E). layout does not utilize this potential. Many images used in the web Low quality images degrade the 7. Replace the existing site are of low quality and/or overall look of the web site, making images with higher quality stretched from smaller samples, the site and its affiliated images. making the overall look of the organization less credible. 8. Limit the use of images site unprofessional. Images should enhance the look of because they increase site a site without being distracting or load time and can distract overwhelming. users from more important content on the site. The participants commented Low quality text images and/or 9. Replace all text images that some items were blurred adjusting image size (shrinking with normal text, unless and difficult to read (see and stretching) can cause images specific design Appendix F). to appear blurry. considerations require it (such as within a logo). User Experience Group Indiana University 9 of 31
    • Observations Interpretations Recommendations There is no consistent format Users have difficulty knowing what 10. Choose a format for page for content text, links, and constitutes a link, a header, plain layout, title, content text, visited links. text and imbedded links. Without a links and visited links and consistent format, users must apply the format explore each item to determine if it consistently across the is a link or not. This requires more site. time and energy than many users are willing to spend. Every page begins with a title Providing a clear title on each 11. Choose a font, color, size that describes the contents of page helps users know where they and position for the title the page, but the format is not are and feel more comfortable and use the same format consistent across the site. navigating the site. However, an consistently throughout the inconsistent format can cause site. confusion. The repetition of frame at the The site identification is important. 12. Apply a consistent format left hand side could be site The repetition of screen layout, to the page layout as well identification; however, there is font size and color contribute to as the contents. no design scheme or stylistic site identification. treatment for the content frame. The link on the left frame turned The purple text on a red 13. Choose a color schemes to purple when user clicked on background does not provide that provide good contrast the link and used the “Back” sufficient contrast and is therefore between the text and button on the browser to go difficult to read. background. back to the page (see Appendix G). Participants were not able to read the link. Links to external sites are The main purpose of navigating 14. Replace the links with the included in what should be the the site is to locate information site navigation structure. site’s main navigation. about the organization instead of 15. Put the external links linking to other sites. Most- under the “resources” frequently used links and the site category or put them in the navigation structure should be footer. available on every page. An empty check box in front of Inappropriate use of symbols or 16. Remove the empty check the “Return to ABC Center images causes confusion. Users box and the check box Menu” link causes confusion. may not know whether to click on animation from the links. Participants were unsure the underlined text or to click the whether it was a bullet point or box. something requiring an action. The check box animation is not On some pages, there is a relevant to the link or its content checkbox animation in front of and is distracting to the user. the link. The navigation image map Using large images for navigation 17. Do not use large images used on “About Our Center” is not an efficient way of organizing unless absolutely takes too much screen space content on a web page. necessary. Replace the (550 x 413 pixels). image map with a bulleted list of links. There are no visual cues for User could easily miss/ignore a 18. Avoid using image maps. users to know that the 4 red link if it does not look like a link. 19. If an image map is triangles on the corners of the necessary, provide visual image map on “About Our cues (underline the text on Center” page have imbedded graphics, for example) to links. tell the user that there is an User Experience Group Indiana University 10 of 31
    • Observations Interpretations Recommendations imbedded link. Participants clicked on some There is no visible distinction 20. Choose a font, color and texts and then realized that it between text and embedded links. style for imbedded links was not a link. Also, there is no consistent format and use the format for imbedded links. consistently across the site. Bulleted links do not line up on Proper item alignment is an 21. Make sure all items are the ‘Programs for Students’ important element for professional properly aligned. page. presentation. Most, but not all, link text is A consistent capitalization scheme 22. Change the link text to capitalized on the ‘Programs for is an important element for the adhere to a consistent Students’ page. professional appearance of the capitalization guideline. site. Terminology Only the homepage has a Windows should be given a clear 23. Provide an appropriate title window title and it did not identifiable title that will display in for every web screen. describe the website clearly. the browser address bar. This will also help the user bookmark the page appropriately. The “Welcome to the ABC The sentence is more like a 24. Phrase all the links Center” link on the homepage is greeting instead of a link. User appropriately. The link text vague as it did not tell user would not be able to tell that it is a should clearly identify the anything about the purpose of link if the link was not underlined. purpose of the link. the link. Participants expected to get The manner in which the images 25. Change the title to be access to an application by were displayed led the user to “Screenshots of Software clicking on images under believe they could access the Applications” to tell users “Software Applications” (see various applications shown. that those images are just Appendix H) some screenshots from the applications. 26. A brief description for each image helps users understand the content. The names of the links to the Using a consistent word or phrase 27. Make sure the name of a menu page are not consistent. to link to any given page will help link is identical on every On some pages, it is called the user learn the structure of your page. “ABC Center Welcome Page”; site and allow them to accurately on other pages, it is called select the appropriate link no “ABC Center Menu”. matter what screen they are on. The menu item and the page Users are not sure where they are 28. Make sure the link reflects title are not the same. when they click on a link and get a the title of the target page. page with a title different from the Page titles may be longer link they clicked. and more descriptive than links, but key words should be the same. Content Organization Contact information is buried Users should not need to browse 29. Consider placing basic too deep into the web site and deep into the site in order to find contact information only available after following a basic contact information. Many (address, e-mail, and narrow navigation route. users will look for this information phone number) at the and become frustrated if it is not bottom of each page on easily located. the web site, or providing a User Experience Group Indiana University 11 of 31
    • Observations Interpretations Recommendations top-level link to the contact information. A number of links open an Users can easily become confused 30. Only contents of the external web page within the about which site they are visiting if website should display in content frame. an external site opens within the the content frame. An content frame. external web page should open in a new window so users know that it is another site. A number of links on the site Users like to be informed before 31. Provide the content in a open a PDF file, a word downloading/opening a file. They format that users can view document, a PowerPoint file or become frustrated when they click online. That gives user the even a streaming video without on a link and get unexpected control to choose to read alerting the users beforehand. results. the document online or download it. 32. Clearly label any links that will start a download or attempt to open an application (such as Acrobat Reader, Word, PowerPoint, etc.) so users know what to expect before clicking. ‘Customer Interface Technology Users would like to be informed 33. Warn the users that the Lab’ opens a Real Media file, before trying to open a file that link will open a Real Media which requires a media player requires an application they may file, and provide a link to a that has to be downloaded. not have. download page where the required media player is available. Navigation Participants spent too much Users expect to see a clear 34. Remove the current time scanning the homepage navigation scheme that can help homepage from the site, with the mouse to see what determine the next page to visit. and establish the main links were available. Time and effort is wasted when the menu page as the users have to explore the page to homepage. find what navigation options are 35. If the current homepage available. has to be kept, make sure that the users know that there is only one navigation option available by moving the “welcome” link to the center of the page. A clear association between the The menu page specifically 36. Provide the bulleted list top-level navigation presents a bulleted list of content presented in the descriptions and sub-level information that is available in menu page for the sub- content was not apparent to each of the top sections. This level pages. participants. leads users to believe that these categories will be clearly marked in their respective sub-pages. The current layout, however, does not distinguish these categories. A quick, obvious link back to The current navigation scheme lets 37. Remove the current the menu page is not provided the users go back to the homepage from the site, User Experience Group Indiana University 12 of 31
    • Observations Interpretations Recommendations throughout the site. homepage from most of the site and establish the main pages. However, there should be menu page as the little need for this, since the homepage. homepage only provides a link to 38. If the current homepage the menu page. It is more logical has to be kept, replace the that the users will want to return to universal link to the the menu page from their location homepage with link to the in order to restart their interaction menu page. with the site. Participants do not know where The current site design does not 39. Provide information about they are and where they came indicate where the user is within current location to facilitate from when navigating the site. the site. Show the user where navigation. they are by using breadcrumbs at the top of screen or by highlighting the user’s location within a persistent navigation structure. User would feel more confident if they know exactly where they are and where they came from when navigating a site. Participants relied on the back Users frequently miss the link 40. Provide a sound navigation button on the browser to because they would have to scroll mechanism. navigate the site. A ‘go back to to the bottom of the page to find it. 41. Make the go back to the previous page’ link is In addition, the link is not always previous page link provided at the bottom of some available on every page. Users available at the same pages, but not all. would be frustrated if they scroll position on every page. down to the bottom of a page but cannot get the link they expected. Some participants could not Users normally click on an item’s 42. Make the name/title of an locate information on the name to get more details about the item a link instead of center’s director as they did not item instead of clicking on the imbedding the link in a know that the link was bullet. bullet point. imbedded in the bullet point A dark red border around the red 43. Links should be presented (see Appendix I). rectangular bullet point is a hint to in an easy to recognize tell users that it is a link, but it is and consistent manner. not easy to tell as the colors are quite similar. Participants did not realize that The use of fancy technique is not 44. Remove the mouse-over the image above the facility lists always appropriate as users might effect and allow users to would change when the mouse not know how to use it. They may get more details of a moved over an item of the list. easily miss the information if they facility item by clicking on Participants clicked on an item do not know how to get access. it. as the cursor turned to a finger User would be frustrated as the when moving over the item. cursor turns to be a finger when moving over an item, but those are not links and they cannot get more information by clicking on it. On the research on retailing User would not be able to know 45. Use the title of the study as page, links of two study reports even the title of a report unless a link and clearly labeled have the same title. clicking on the link. the link to tell user what to The image above the sentence It causes confusion when the site expect when clicking on of “Click for a PDF copy of the cues the users for action that does the link. study” is not a link although it not lead to the expected result. 46. Change the image to act indicated to be. as a link. User Experience Group Indiana University 13 of 31
    • Observations Interpretations Recommendations Participants were confused Users expect different links to go 47. Either combine the links to when the “shopping to different pages. Many users will be “shopping simulations & simulations” link and the not understand that the links point software applications” or “software applications” link took to different sections on the same separate the contents into them to the same page. page. two separate pages. When the user mouses over a The list format and position of the 48. Remove the mouse-over category in the menu, it popped pop-up information create effect and use available up a brief description of the confusion because it looks like a screen space to display category. Participants regarded pop-up submenu. Users become the brief description right it as a pop-up submenu and frustrated trying to click on this below each category. wanted to click on it but the image that disappears every time 49. Change the format to pop-up description disappeared they move their mouse. make it more like a brief once the cursor moved away description of a category in from the category. Participants stead of a list moved their cursors back and forth between the menu and the pop-up information and then realized that it was not a submenu (see Appendix J). The link at the bottom of the Bad navigation mechanisms force 50. Simplify the link at the page always returns the user to the user to recall where they came bottom of the page to be the previous page (see from. “back to previous page”. Appendix K). However, Users have more processing 51. Provide a mechanism that participants were confused by capacity for the contents when helps users to recognize links that contained the title there is less cognitive load for where they are in the because they did not always locating their position. website instead of recalling recognize the names of the their path. previous pages they visited. ‘KPMG’ link in the ‘Sponsoring Dead links damage the credibility 52. Fix the dead link to lead to Partners’ page is a dead link. of the site and frustrates the users the correct, working site or who wish to visit the linked page. if the site no longer exists, remove the link and associated text. The ‘Marshall Field’s’ and Link names should always 53. Fix the ‘Marshall Field’s’ ‘Mervyn’s’ links on the correctly refer to the site they link and ‘Mervyns’s’ links to ‘Sponsoring Partners’ page do to. Unexpected behaviors, such as lead to their respective not take the user to the correct the wrong site opening, lead to sites. Check all links sites. Rather, both of these user confusion and frustration. (internal and external) for links lead to target.com. accuracy and periodically verify that external links still point to published sites. On the ‘Activities and Users expect similar behaviors 54. Group links together that Programs’ page, various links in from links that are grouped behave in the same the same bulleted list behave together. Furthermore, users’ manner. Provide a differently. Some open up new mental model of the site navigation distinction or visual browser windows, others open scheme is weakened when separation for links that PowerPoint-like slides, and unpredictable links are present. behave differently. others go to pages deeper in the hierarchy. Clicking any items on the The users expect the side 55. Fix the links so that the “Timeline” page takes the users navigation bar to be present at all pages are displayed within out of the site navigation frame. times. the content frame in the Clicking on the “Home” link on browser window, leaving User Experience Group Indiana University 14 of 31
    • Observations Interpretations Recommendations the “Timeline” page takes the the side navigation intact. users out of the site navigation frame. The “Home” link at the upper- Different links with the same name 56. Rename the link to avoid left hand corner of the causes confusion. confusion with the link to “Timeline” page is confusing. the center’s homepage. Participants did not know whether they would be taken to the site’s homepage or the center timeline’s homepage. Operating System and Browser Issues The red rectangle with the Some operating systems and 57. Test the most popular imbedded link on the “About browsers are more flexible than browsers and operating Our Center” page has a 1 pixel other in rendering code systems to make sure the bolder to indicate that a link. site works properly. The border does not display properly on a Macintosh (see Appendix I) The checkbox image in front of Some operation systems and 58. Test the most popular the “Return to Research Center browsers are more flexible than browsers and operating Menu” works on PC, but the other in rendering code systems to make sure the image did not display properly site works properly. on a Macintosh (see Appendix L). The mouse-over link color does Some operation systems and 59. To increase the likelihood not change for Netscape 7.2 on browsers are more flexible than that pages will display and PC, Netscape 7.01 on other in rendering code function properly, make Macintosh and Firefox 1.0 on sure that code is formed both PC and Macintosh correctly with proper tags computers. (always use closing tags), etc. 60. Test the most popular browsers and operating systems to make sure the site works properly. If all browsers and operating systems will not be supported, provide text stating which browsers the site supports. User Experience Group Indiana University 15 of 31
    • Post-test Satisfaction Ratings Site Satisfaction Ratings Satisfaction User Survey 1 5 Median Mean strongly strongly (Raw score by SUS question) disagree agree I think that I would like to use this 5 5 2 4 4 3 3 4 3 5 4 2 3 2 3 system frequently. 3 3.47 I found the system more complex than 1 2 5 3 3 5 1 2 3 1 3 5 3 5 1 necessary 3 2.87 I thought the system was easy to use. 4 5 2 2 3 3 4 5 3 5 3 2 2 2 4 3 3.27 I think that I would need the support of 1 1 1 3 1 1 1 2 2 1 2 1 1 1 1 an experienced person to be able to use this system. 1 1.33 I found the system visually appealing 5 5 3 4 2 3 4 4 5 5 3 5 3 3 4 4 3.87 I thought there was too much 1 1 3 1 3 2 1 1 3 1 3 3 4 3 1 inconsistency in this system. 2 2.07 I would imagine that most people 5 5 1 4 2 2 5 4 3 5 3 2 3 1 5 would learn to use this system very quickly. 3 3.33 I found the system very cumbersome to 1 1 4 2 2 2 1 4 4 1 3 5 3 4 1 use. 2 2.53 I felt very confident using the system. 5 4 5 3 2 4 5 5 2 5 3 3 4 5 5 4 4.00 I needed to learn a lot of things before I 1 1 1 3 3 1 1 1 3 1 4 2 1 1 1 could get going with this system. 1 1.67 Satisfaction User Survey (total SUS Score (0-100) adjusted SUS scores by participant) Participant 1 97.5 Participant 2 95 Participant 3 47.5 Participant 4 62.5 Participant 5 52.5 Participant 6 60 Participant 7 90 Participant 8 80 Participant 9 52.5 Participant 10 100 Participant 11 52.5 Participant 12 45 Participant 13 57.5 Participant 14 47.5 Participant 15 90 Median 60 Mean 68.67 Raw SUS scores are adjusted as follows: * Questions 1, 3, 5, 7, and 9: adjusted score = raw score – 1 * Questions 2, 4, 6, 8, and 10: adjusted score = 5 – raw score User Experience Group Indiana University 16 of 31
    • Calculating an individual’s total SUS score for an application: * The sum of the adjusted scores is multiplied by 2.5 to yield an overall score in the range of 0 – 100. Higher total SUS scores indicate greater overall satisfaction with the website. The median and mean scores of 60 and 68.67 respectively for the ABC Center website represent a somewhat lower level of user satisfaction than desired. Correcting the various usability issues identified in the preceding sections of this report are likely to improve user satisfaction with the site. Satisfaction Questionnaire – Post-Test Questions The table below shows participants responses to the three SUS post-test questions. Participants wrote their responses on the SUS form and they are included here verbatim. Question 1 – What do you consider the most valuable aspect of the system? P1 Easy to use, straight forward. P2 Access to rich, original research. P3 Easy to find research based on topic of interest. P4 Getting information. P5 There is a lot of information. P6 Fairly easy to find what I needed. P7 Wealth of information. P8 Links to cross references. P9 Rich information. P10 Providing easy links for additional research material. P11 P12 This site has many research materials that are not available anywhere else. P13 I liked the links to other research resource sites. P14 Contact information for the research authors is a good resource. P15 Easy to find the research I am looking for. Question 2 – What is the biggest problem with the system? P1 Some of the links took me out of the site completely without warning. P2 The images that are used are not that good in quality. It makes the site look unprofessional. P3 Organization is not clear. P4 Make it user friendly. P5 There seem to be a lot of space wasted – I would rather have more content than fancy titles and graphics. P6 Did not like the images used. P7 It was difficult to navigate the site. P8 The pages in the site looked too different from each other. They don’t look like they belong to the same site. P9 Difficult to navigate. P10 Lack of professionalism in site presentation. P11 N/A P12 Full screen is not utilized. P13 Site looked unprofessional. P14 Navigation options are sometimes unavailable. P15 Many linked files required programs that I did not have on the computer. Question 3 – Additional comments P1 Nice site, lots of information. P2 Change the images to better ones, and the site will look much nicer. P3 P4 About time! [do not know what the participant was referring to ] User Experience Group Indiana University 17 of 31
    • P5 P6 It was frustrating because navigation options would disappear on me at times. Those links should be there at all times. Also, there were instances where the links that I clicked took me to an external link. I think those should open up new windows. P7 P8 P9 Needs more organization. P10 P11 I look forward to using it in the future, nice content. P12 Overall it was fine, but I think the images should be changed to better ones. P13 P14 Make the navigation bars available at all times. This will make the site much easier to use. I liked the content available on the site, lots of original research. P15 User Experience Group Indiana University 18 of 31
    • Appendices Appendix A – Testing Protocol Script Participating in Usability Sessions Thank you for agreeing to participate in the usability study of the ABC Center website. Before we begin, I will briefly go over what a usability session consists of, what will be expected of you, and what our goals are in conducting this study. The first thing that I want to make clear is that we are testing the application, NOT you. If you find errors or if you have any difficulties with the application, it is very likely that other people visiting the site will also experience those same difficulties. We will use this information to better understand how we might improve the site. During the Session: 1. Tasks: You will be asked to perform a series of tasks using the ABC Center website. The tasks are structured to determine if the most important information and features of the site are easy to locate and use. We ask that you try to accomplish the tasks without assistance, as if you were at home trying to find the information on your own. This gives us a better idea of the things that work well or the difficulties people experience. Anytime during the session, if you find yourself thinking that you would quit the task, use help, contact someone for assistance, or visit another site to accomplish your goal, please let us know. 2. Test Facilitator and Observers: One member of the User Experience Group will act as the test facilitator assisting you in getting started and answering any questions you may have. Another member of the User Experience Group will observe and take notes. Representatives from the ABC Center team may also be present to observe the session and take notes. 3. Think Aloud Protocol: To help the observers understand the way you use the application, you will be asked to “think aloud” as you complete the tasks. This simply means that we ask you to talk about what you are doing, what you are looking for, clicking on, wishing you could find, etc. In other words, any task-related thought that comes into your mind we would like for you to share aloud. Some people find this easy right away, while others need to be reminded a little. It may seem a bit odd to talk about every step that you are doing, but it really helps the observers understand how you are interacting with the application and how the application is working. After the Session: 1. User Satisfaction Questionnaire The user satisfaction questionnaire is a short questionnaire that asks you about a few general aspects of the system. It will only take a few minutes for you to complete. 2. Questions If you have any further questions for the facilitator or members of the ABC Center team, you will have the opportunity to ask them at that time. User Experience Group Indiana University 19 of 31
    • Appendix B – Participant Consent Form Participant Consent Form The purpose of this usability study is to evaluate the design of the ABC Center website. We are interested in determining if people can accomplish common tasks and easily find information using this website. The session will not ‘test’ you or your ability, rather the session will test the website in order to provide information on areas that might be improved. Please be advised that there are no foreseeable risks associated with participation in this session. During this session, you will be asked to complete some tasks using the ABC Center website and fill out a user satisfaction questionnaire. As you complete the tasks, members of the User Experience Group and ABC Center will observe and take notes. In addition, the session will be captured on video for future review. The session will last no longer than one hour and fifteen minutes. If for any reason you are uncomfortable during the session and do not want to complete a task, you may say so and we will move on to the next task. In addition, if you do not want to continue, you may end the session and leave at any time. Approximately 16 people will participate in this study. Results from all sessions will be included in a usability report to be presented to ABC Center. Your name will not be included in the report nor will your name be associated with any session data collected. If you wish to speak with someone about your participation in this study, or if you feel you were not treated as described above, please contact the User Experience Group manager at 812-855-4499. I, ______________________________________________, have read and fully understand the extent of the study and any risks involved. All of my questions, if any, have been answered to my satisfaction. My signature below acknowledges my understanding of the information provided in this form and indicates my willingness to participate in this user testing session. I have been given a blank copy of this consent form for my records. Signature:______________________________ Date:________________ User Experience Group Indiana University 20 of 31
    • Appendix C – Video Release Form Video Release Form The signature below indicates my permission for University Information Technology Services User Experience Group of Indiana University to use video footage recorded during the usability session conducted for ABC Center Website on _______________, 2005 in which I served as a participant. My name will not be reported in association with session results nor will my name be included on the video footage. This video footage may be used for the following purposes: • Analysis of research and reporting of results • Conference presentations • Educational presentations • Informational presentations I will be consulted about the use of the video recording for any purpose other than those listed above. There is no time-limit on the validity of this release nor is there any geographic specification of where these materials may be distributed. This release applies to video footage collected as part of the usability session listed on this document only. I have been given a blank copy of this release form for my records. Name (please print): Date: / / Signature: Address: Phone: E-mail: User Experience Group Indiana University 21 of 31
    • Appendix D – Satisfaction Questionnaire System Usability Scale © Digital Equipment Corporation, 1986. w/ Revision by Usability Consulting Services, 2002. Strongly Strongly disagree agree 1. I think that I would like to use this system frequently 2. I found the system more complex than necessary 3. I thought the system was easy to use 4. I think that I would need the support of an experienced person to be able to use this system 5. I found the system visually appealing 6. I thought there was too much inconsistency in this system 7. I would imagine that most people would learn to use this system very quickly 8. I found the system very confusing to navigate 9. I felt very confident using the system 10. I needed to learn a lot of things before I could get going with this system User Experience Group Indiana University 22 of 31
    • Appendix E – Space Utilization Too much screen space (80+ pixels) is wasted on the top of the menu page. Move the location of the page title image up to reduce the white space that occupies the top of the page. User Experience Group Indiana University 23 of 31
    • Appendix F – Image Fonts The participants commented that some items were blurred and difficult to read. Low quality text images and/or adjusting image size (shrinking and stretching) can cause images to appear blurry. Replace all text images with normal text, unless specific design considerations require it. User Experience Group Indiana University 24 of 31
    • Appendix G – Contrast Between Text and Background Figure.1 Figure.2 Figure 1 shows what a link looks like on the red frame. The link turned to purple when user clicked on the link and used the back button on the browser to go back to the page (Figure 2). Purple font on a red background does not provide sufficient contrast and many participants will have difficulty reading the text. Choose a color scheme that provides good contrast between the background and text, links, and visited links. User Experience Group Indiana University 25 of 31
    • Appendix H – Terminology Figure.1 Screenshot of the Virtual Showcase page Figure.2 A screenshot of an application When clicking on an image under the “Software Applications” session (see Figure 1 above), users expect to get access to the application. Users tried to click on the image (Figure 2) to navigate the system and then realized that it was just a screenshot of the application. User Experience Group Indiana University 26 of 31
    • Appendix I – Link Format The image above shows the current design scheme on the ‘About Our Center’ page. Users easily miss the links since bullet points are not a common place to locate links. The dark red border may have been meant to indicate an embedded link, but the border is difficult to see due to low contrast between the colors. The image above shows what the rectangular bullet points look like on Macintosh computers using Internet explore 5.2 or Safari 1.2.4 It is difficult for users to get more details since there is no visual cue (border around the bullet point) to tell user that the bullet points are links and using bullets as links is not a common practice. If users notice that links are available, it will likely be due to accidentally mousing over the bullet point and noticing the cursor change. The example above uses the name/title of an item as a link instead of the bullet point. Using the name/title to link to more details of an item is a common method that users are familiar with, and so they would be less likely to miss the link. In addition, applying a consistent format (such as underlining) to all links will help users easily distinguish links from other items on the site. User Experience Group Indiana University 27 of 31
    • Appendix J – Mouse-Over Effect Figure.1 Figure.2 Figure 1 above shows the appearance of the center menu page. When the user moves the mouse over a category of the menu, the category was highlighted and it popped up a short description about the category (see Figure 2). The short description displayed in list format; users regarded it as a pop-up submenu and tried to click on it. They got frustrated as the “submenu” disappeared when the mouse moved away from the category. User Experience Group Indiana University 28 of 31
    • Appendix K – Linking to Previous Page The link at the bottom of the page always returns to the previous page. However, participants were confused by links that contained the title of the target pages (as shown in the example above) as they did not always recognize the names of the previous pages they visited. Simplify the link at the bottom of the page to “back to previous page”. In addition, provide a mechanism, such as the breadcrumbs shown above, that helps users to recognize where they are in a website instead of recalling their path. User Experience Group Indiana University 29 of 31
    • Appendix L – Operating System and Browser Issues There is a checkbox image in front of the Return to Research Center Menu link on Windows using Internet Explore 6.02, Firefox 1.0.2, and Netscape. The checkbox image does not display correctly on Macintosh computers using Internet Explorer 5.2 The checkbox did not show up on the Macintosh computer using Safari 1.2.4 User Experience Group Indiana University 30 of 31
    • Appendix M – Browser Compatibility Issues Using the most recent version of Internet Explorer on both the Mac and pc, the links change color with mouse- over to indicate that they are selected and the cursor is changed to indicate a hotspot. In Netscape 7.2 (pc), Netscape 7.01 (Mac), or Firefox 1.0 (pc and Mac), however, the mouse-over does not work, and the cursor stays the same. User Experience Group Indiana University 31 of 31