Usability Testing Report
Upcoming SlideShare
Loading in...5
×
 

Usability Testing Report

on

  • 309 views

 

Statistics

Views

Total Views
309
Views on SlideShare
309
Embed Views
0

Actions

Likes
0
Downloads
3
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Usability Testing Report Usability Testing Report Presentation Transcript

    • NEXTRAVELUsability Testing Findings and RecommendationsFebruary 2012
    • Usability research goals• Primary usability research goals • Observe Tumbler user experience to determine:  whether evaluators understand the tumbler functionality  how evaluators perceive and interact with the tumblers  Interest of using tumblers to leverage their searches • Observe Detail View user experience to determine:  whether evaluators can easily find out the detail view  how well the detailed information can serve their needs • Observe Filter/NLA user experience to determine:  whether evaluators can easily locate the filter/ NLA button  how well the filters/NLAs can help narrow results • Observe Map user experience to determine:  whether evaluators can easily locate the map functionality  whether evaluators can accurately interpret all the information in the map  Interest and engagement with the map • Observe Ambiguous Search user experience to determine:  whether evaluators understand the ambiguous search functionality  how evaluators perceive and interact with the recommendations in the tumblers  Interest of using Ambiguous Search ©2012, Proprietary and Confidential property of Nextag, Inc. 2
    • Key insights: summary• Key insights: • Tumbler Search: clearly understood, but not always useful • Detail View: considered useful and comprehensive • Filter/NLA: considered useful but not easy to find out • Map: not easy to find or understand • Ambiguous Search: consistent with evaluators’ expectations, but the quality was not good enough ©2012, Proprietary and Confidential property of Nextag, Inc. 3
    • Key insights: Tumbler Search• Tumbler service clearly understood • All evaluators could use tumblers to narrow their search results • 7 out of 9 evaluators thought it was cool to search hotels by using tumblers  “I think the spinners are a good idea. It does actually help you find a hotel quicker that close to particular areas that you would like to get to.”----P2  “I definitely like using the spinners. Ive never used anything like this before. The spinners were so convenient and easy to use, even for a first time user of the website, like myself. ” ----P8 • Even though P2 said the tumblers were cool to use, she also mentioned to get rid of the tumblers:  “Its frustrating me that I cannot see this(the hotel list) in a bigger way. I dont see a way to get rid of this(the tumblers) either. I would really get this(tumblers) a little smaller… maybe just focus on one at a time. ”----P2 • Only P7 expressed strong dislike of using the tumblers because she couldn’t stand too much movement  “I wish to get rid of the screen(tumblers) and only look at the hotel list below the spinners. I would rather have only one thing move at a time. I wish the movement on the hotel list could be toned down.”----P7 • One out of 9 evaluators didn’t express explicitly his attitude for using the tumblers to narrow searches  “I don’t know how much I like the spinners.” ©2012, Proprietary and Confidential property of Nextag, Inc. 4
    • Key insights: Tumbler Search• Thoughts  In the usability tests, evaluators were dictated to use the tumblers to accomplish certain tasks  The tumblers took up from 1/3 to 1/2 of an evaluator’s screen vertically  Even though most of them felt “cool” about using the tumblers, we don’t know how much they would like to keep using the tumblers for all hotel searches• Recommendations  Need to further explore whether should make the tumblers hidable ©2012, Proprietary and Confidential property of Nextag, Inc. 5
    • Key insights: Tumbler Search• Neighborhood feature not always useful • 3 out of 9 evaluators thought the neighborhood feature was only useful for people who had been to the place before  “Show neighborhood function only to people who have been to the place before. The neighborhood function looks annoying and stupid to me.”----P4  “Have a prompt saying have you been to this place before?, and use that to control the comp up of neighborhoods and landmarks. ”----P4  “I can only recognize Loop, Museum Campus and Airport of the neighborhoods because I’m not living there. The neighborhood tumbler can get confusing if I’m just casually browsing. The least useful feature for me is the neighborhoods.” ----P6  “I’ll use the neighborhood and landmark function if I know the city.”----P10 • One evaluator(P7) skipped task 2, and she strongly disliked using tumblers • Even thought one evaluator(P9) claimed he thought the neighborhood feature was useful, he didn’t use it to find a hotel before asked explicitly to • Recommendations  Experiment using landmarks as the default content for the 2nd tumbler  Shorten the neighborhood list ©2012, Proprietary and Confidential property of Nextag, Inc. 6
    • Key insights: Tumbler Search• Landmark list too long • One evaluator said the landmark list in the 2nd tumbler was too long and needed too many clicks • Most evaluators thought it was helpful to have the landmark information in a city • Recommendations  Shorten the landmark list to only give most famous landmarks  Rank landmarks by its popularity ©2012, Proprietary and Confidential property of Nextag, Inc. 7
    • Key insights: Tumbler Search• Tumbler names/links not clear and easy to see • P4 scrolled the “Destination” tumbler to look for Loop neighborhood, and she said she didn’t notice the “Destination” tumbler was the “City” tumbler [4:30] • P6 said it’s hard to see the links above the tumblers  “Oh…I see! You can change this(the 2nd tumbler) to landmarks! That’s interesting. I didn’t know that. “  “If they(filter options above the tumblers) were a little easier to see, I would have started using them sooner.” • Recommendations  Change the name destinations to cities  Make links above the tumblers more prominent ©2012, Proprietary and Confidential property of Nextag, Inc. 8
    • Key insights: Tumbler Search• Scrolling not fast enough • One evaluator thought the tumblers were a little sticky, and 3 other evaluators tried to type to accelerate the scroll of the tumblers  “They got a little bit stuck, but they are not so bad.”----P2  “I would like to type and see the spinners scroll to the destination.” ----P5  “It’ll be cooler if I can just type to let the spinners spin.”----P6  “I want the tumblers to go faster because it took a long time going from A to Z.”--- -P10 • One evaluator wished that the tumblers could keep scrolling when he pressed the arrows • Other evaluators didn’t see any problem of scrolling the tumblers, and some of them thought the scrolling was very convenient and seamless. • Recommendations  Enable type to accelerate the scroll ©2012, Proprietary and Confidential property of Nextag, Inc. 9
    • Key insights: Tumbler Search• Scrolling surprises • One evaluator dragged the tumblers to scroll, but she unconsciously clicked the empty space somewhere in the tumbler and surprisingly found that the tumbler spinned in the reverse direction(P2 [10:30])  “Sometimes the spinners are a little funny. I think the spinners is fine…Maybe a little bit tweaky” ----P2 • One evaluator used the scroll arrows in the 1st tumbler and found it was a little frozen (P3[00:15]) • Recommendations  Clearly inform users all the ways they can scroll or possibly move the tumblers ©2012, Proprietary and Confidential property of Nextag, Inc. 10
    • Key insights: Tumbler Search• Defaulted sort order not clear • 2 evaluators(P2 & P9) changed the sort order from “Best Match” to “Price low-high” • 2 evaluators questioned the sort order of hotels  “It gives you hotels recommended. I’m not sure what they based that on. ” ----P5  “to be helpful to know why you recommend it. Ur…probably values, right?”----P9 • One evaluator said she might want to see the hotel list ordered by price  “They(the hotel list) are sorted by best match. I guess thats OK. I may want to look by price though. ” ----P8 • Recommendations  Further explore what’s the best default order ©2012, Proprietary and Confidential property of Nextag, Inc. 11
    • Key insights: Tumbler Search• Poor readability • Some evaluators reported problems of reading the contents in the tumblers  One evaluator said his first impression was that the website was too text-heavy(P3)  One suggested to use another color, like yellow or light orange, for the highlighted search result in the tumblers to differentiate it from other un- highlighted text • Recommendations  ????I don’t know…it’s the problem of using tumblers ©2012, Proprietary and Confidential property of Nextag, Inc. 12
    • Key insights: Detail View• Detail View is useful and necessary • almost all evaluators liked the information in detail view  8 evaluators thought the detail view gave them all the information that they could possibly need. Only one evaluator said he disliked the detail view and it was not clean, but he didn’t give any reasons for this opinion.  All evaluators could easily find how to open the detail view, get hotel rates and book a hotel. 2 evaluators mentioned it would be more desirable if they could book a hotel directly from the website. ©2012, Proprietary and Confidential property of Nextag, Inc. 13
    • Key insights: Detail View• Detail View problems • 2 evaluators expected to see long and detailed descriptions  2 evaluators found there were only one or two sentences for the hotel description, so they complained it was too short • One evaluator wanted to know breakfast information • 2 evaluators suggested other ways of displaying amenities  One would like to have both amenity icons and an amenity list  One suggested not to show the icons of unavailable amenities---” I dont see the points of actually displaying them, if something is not available. It created confusion. If something is not available, either put a line through it or throw it. ”(P3) ©2012, Proprietary and Confidential property of Nextag, Inc. 14
    • Key insights: Detail View• Detail View problems • One evaluator clicked the highlighted hotel name in the 3rd tumbler to open the detail view  “That would maybe open up a pop-up window, but its not.”---P8 • Thoughts  Once the tumblers become the main focus, user will expect to get all the information from the tumblers • Recommendations  Need to further explore all the advantages and defects of using 2 exactly same hotel lists ©2012, Proprietary and Confidential property of Nextag, Inc. 15
    • Key insights: Detail View• Detail View problems • most evaluators thought the star ratings were guest/user ratings  One evaluator expected to see user reviews in detail view by clicking the hotel stars, and he was surprised that he couldn’t find any recommendations for giving 3 stars to the hotel • Recommendations  Include user rating contents from TripAdvisor ©2012, Proprietary and Confidential property of Nextag, Inc. 16
    • Key insights: Filters/NLAs• Filters/ NLAs are clear & easy to use • all evaluators liked the filters/NLAs  One said the filter/NLA options covered all the thing he wanted  One said the filters/NLAs were perfect and extremely clear. She rated it ten out of ten. ©2012, Proprietary and Confidential property of Nextag, Inc. 17
    • Key insights: Filters/NLAs• Filters/ NLAs problems • 2 evaluator said the filter/NLA button might be hard to see • Some evaluators said the amenity icons were hard to recognize when not highlighted  “At first it (the pool icon) looked like jacuzzi, but if you hove over, it would tell me.”- ---P2  “That’s a pool?! Could be a little clearer about that!”----P9  Some evaluators didn’t notice there were tool tips until they 2nd time hovered over amenity icons  Evaluators were guessing the meanings of each amenity icon before they checked through all the tool tips • Some evaluators thought star ratings were guest/user ratings • Recommendations  Further explore ways to place the filter/NLA button  Re-design amenity icons to make them easier to recognize  Include user ratings as another filter/NLA option ©2012, Proprietary and Confidential property of Nextag, Inc. 18
    • Key insights: Filters/NLAs• Narrow it? • Immediately, P3 clicked on “Narrow your results” to find a hotel around Navy Pier, even though he had already noticed & opened the landmark list • Immediately, P4 clicked on “Narrow your results” to find a hotel closest to the beach • Thoughts  Some well-known travel websites allow users to select hotels close to a certain neighborhood or landmark under the filters/NLAs ©2012, Proprietary and Confidential property of Nextag, Inc. 19
    • Key insights: Filters/NLAs• Narrow it? • Immediately, P3 clicked on “Narrow your results” to find a hotel around Navy Pier, even though he had already noticed & opened the landmark list • Immediately, P4 clicked on “Narrow your results” to find a hotel closest to the beach • Thoughts  Some well-known travel websites allow users to select hotels close to a certain neighborhood or landmark under the filters/NLAs ©2012, Proprietary and Confidential property of Nextag, Inc. 20
    • Key insights: Ambiguous Search• Ambiguous Search approximated user expectations • 7 out of 9 evaluators thought the tumblers should show places that actually had nice ocean beaches when they searched “beach”  the 7 evaluators seemed very excited about the ambiguous search feature. One said the function seemed unique.  the 7 evaluators checked the entire region list to prove their initial guess, most of them questioned the recommendation quality since they saw Arizona, Indiana and Kentucky were also in the list • Recommendations  Tell users explicitly what will happen after searching ambiguous keywords  Improve the search quality ©2012, Proprietary and Confidential property of Nextag, Inc. 21
    • Key insights: Ambiguous Search• Ambiguous Search approximated user expectations • 2 out of 9 evaluators couldn’t understand the defaulted recommendations in the tumblers by searching “beach”  One couldn’t understand why the 1st tumbler changed from a city list to a region list after searching “beach”. He said it was very confusing and not user-friendly. “Its putting me in Florida now, which I dont understand. It(the 1st tumbler) should be a search filter by actual city and I dont know where it is. Its not user friendly. ”-- -P3  One said the search was not clear.” It seemed to go to the destination with the word ”beach” in it, not the beach I want to go to.”----P10 • Recommendations  Inform novice users the website is capable of ambiguous searches ©2012, Proprietary and Confidential property of Nextag, Inc. 22
    • Key insights: Ambiguous Search• Ambiguous Search approximated user expectations • One evaluator disliked the default recommendation by searching beach  “I can see why pick Miami Beach because that was… I dont know…maybe its the most popular search under the word beach. I dont know Miami Beach is the destination I want to go. Maybe I want to go to West Palm Beach or somewhere that just simply has a beach, like St. Augustine”----P5 • Thoughts  No users knows how the search engine works in our website (our website hasn’t established its reputation yet)  The recommendation in the tumblers looks too strong and pushy, but it doesn’t give any explicit reasons  Personalized recommendations are more desirable • Recommendations  Need further explore ways to give “desirable recommendations” ©2012, Proprietary and Confidential property of Nextag, Inc. 23
    • Key insights: Map• Map feature is desirable • All evaluators thought the map feature was great  Some evaluators liked clean maps and they thought no more information was needed in the hotel map  Almost all evaluators could properly interpret the information in the maps ©2012, Proprietary and Confidential property of Nextag, Inc. 24
    • Key insights: Map• Map problems • 6 evaluators wished the map button could be more prominent  One said she didn’t notice the map until she was asked to find it out explicitly by the task  One wished the map could be more obvious because he almost didn’t see it • one evaluator(P10) didn’t find the, and he supposed the map in detail view was the one mentioned in the final task • 3 evaluators had problems of closing the map  The map close button went down below the bottom of one evaluator’s browser, so he couldn’t find how to close the map(P6[21:35])  One evaluator wished the map close button could be bigger • Recommendations  Further explore ways to place and design the map button/icon  Make the map shorter or place the close button in the upper right corner of the hotel map ©2012, Proprietary and Confidential property of Nextag, Inc. 25
    • Key insights: Map• Map problems • All evaluators didn’t notice the city map before they were asked explicitly to tell the intention of 2 maps. 7 evaluators precisely got the function of each map. • Two evaluators didn’t answer all the map questions • Recommendations  Provide description about each map’s functionality  Experiment using different markers in the city map and the hotel map  Give city name pop-ups when users mouse over the markers in the city map ©2012, Proprietary and Confidential property of Nextag, Inc. 26
    • Key insights: Map• Map problems • One evaluator(P9) mentioned he would like to see more hotels instead of one or three, suddenly he noticed the map was connected to the tumbler above it • One evaluator(P5) complained that the information in the pop- ups was the same with the 3rd tumbler list, and he would rather to have no pop-ups in the map Recommendations Furtherexplore the advantages and defects of using tumblers to navigate the map ©2012, Proprietary and Confidential property of Nextag, Inc. 27
    • Key insights: Map• Map problems • Some evaluators said they would also like to see major attractions, promotions or restaurants in the hotel map • One evaluator said he would like to see public/private beaches in the map when he searched “beach” • Recommendations  Include more information to make the map more useful ©2012, Proprietary and Confidential property of Nextag, Inc. 28
    • AppendixAPPENDIX 29
    • Participants and test procedure• Web site participant attributes • Online usability sessions: o 9 participants: 4(female) + 5 (male) o Ages: 4(18 – 34) + 5(35 – 55) o Income: $40K – $100K o Averaged Web Experience• Usability test procedure • Online web site research conducted via usertesting.com o Evaluators completed several directed searches for hotels in Chicago, then reviewed the findability and usability of Detail View and Filters/NLAs. Evaluators also completed a directed ambiguous search for hotels in places that had beaches, then reviewed the findability and usability of Map. ©2012, Proprietary and Confidential property of Nextag, Inc. 30
    • Test planScenario: Imagine you are looking for hotels for an upcoming vacation. You are considering Chicago because there are rich things to see and do. You arealso thinking about going to beaches to enjoy the sunshine and water. You will be using a travel website to research and book quality hotels with good price.Please use Safari or Chrome for all the tasks.Task 1. Go to [http://travel.nextagqa.com/] and search for a hotel in Chicago. When you’ve found a hotel you like, briefly describe what you found and why youlike it. You do not need to make a booking.1.What is your initial impression of this site?2.Were you able to find a hotel you like?3.What site features, if any, were most useful?4.What site features, if any, were least useful?Task 2. Making sure you are still on the search results page, use the 3 Spinners below the search bar to find a hotel in the Loop neighborhood of Chicago andanother hotel close to an attraction called Navy Pier in Chicago, briefly describe what you found.1.What information does each spinner provide?2.What, if anything, did you like about using the spinners to find the hotels?3.What, if anything, did you dislike?4.How can the process of searching using the spinners be improved?Task 3. You’d like to do more research about the hotel you found near Navy Pier:1.Is there a way to look at detailed information about the hotel?Task 4. If you haven’t done so already, please click on the hotel name in the yellow frame with a green border:1.Is this the information you expected? Please explain.2.Is there a way to look at the rates for this hotel?3.Is there a way to book this hotel?Task 5. Go back to the home page:1.Is there a way to only look at 3 star hotels with pools?2.If you found a way to only see 3 star hotels with pools, were the filter options clear?3.What would you do to improve the process of filtering hotels?Task 6. Imagine you’d like to find a hotel close to a beach. Type ”beach” into the search box, then use the spinners to find a hotel you find interesting. Brieflyexplain what you found.1.Is this what you expected before you started to search? Please explain.2.What, if anything, did you like about using this search method and spinners to find the hotel?3.What, if anything, that bothered you about finding an interesting hotel?4.How could the process of searching using the spinners be improved?Task 7. You’d like to stay at the hotel closest to the beach.1.Is there a way to see the location of the hotels?2.What do you think the two maps are for?3.What do you think the green markers on the map mean?4.What do you think the red markers on the map mean?5.What other information would you like to see on the map? ©2012, Proprietary and Confidential property of Nextag, Inc. 31
    • Testing Video LinksP1 Testing VideoP2 Testing VideoP3 Testing VideoP4 Testing VideoP5 Testing VideoP6 Testing VideoP7 Testing VideoP8 Testing VideoP9 Testing VideoP10 Testing Video ©2011, Proprietary and Confidential property of Nextag, Inc. 32