Webinar:Retail UX Benchmarking: Which BrandProvides the Best Website Experience?                                        Na...
Intros                 Alfonso de la Nuez            Nate Colker                Co-Founder & Co-CEO          UX Researcher...
Overview• Defining UX Benchmarking• Luxury department store Competitive Industry Study• Tips for building your own competi...
Define Benchmarking                                   UX Benchmarking is the process          “Benchmarking is the        ...
Definining UX Benchmarking & Metrics                               Task Completion                                        ...
Define measurable & comparable UX-related KPIs                                       Jeff Sauro,                         ...
Why do we Care about Measuring User Experience?Ordering a Pizza Online (1994)     Introduction        Study Design        ...
Why do we Care about Measuring User Experience?Ordering a Pizza Online (2012)     Introduction        Study Design   Resul...
The World of User Experience Testing  Introduction   Study Design   Results                                          Tips ...
Unmoderated Usability Testing  Introduction   Study Design   Results   Tips
Unmoderated Usability Testing  Introduction   Study Design   Results   Tips
Case Study: 4 Brands  Introduction   Study Design   Results   Tips
Study Summary•   Between subjects design•   50 Participants per brand (200 total)•   2 Tasks + survey questions•   Randomi...
Task DescriptionsTask 1 Objective: measure findability of a common apparel item (“scavenger hunt task”)Task DescriptionPle...
Case Study Preview  Introduction   Study Design   Results   Tips
Task 1 Completion Rates: Find a dress                                                     Task Success Rate        100%   ...
Disaster Rates (Highest Confidence but Failure)  100%                8%                                     4%            ...
Task-Level Ease Overall, how difficult or easy was it to find the price of a grey wool dress?Very Easy                  7 ...
Open-ended responses from participants whothought the task was difficult (1, 2 or 3)Briefly describe why you chose the num...
Open-ended responses from participants whothought the task was difficult (1, 2 or 3)Briefly describe why you chose the num...
Task Time: Completed tasks only              350              300                                    293              250 ...
Post task questionsStrongly Agree                      7                                                               6.3...
Task 2Evaluate the sales sectionPlease take your time to explore the sale section and look for things thatare of interest ...
Sale Section EvaluationHow satisfied are you with the number of items that were on sale?Very Satisfied                    ...
Sale Section EvaluationHow satisfied are you with the sale prices?Very Satisfied                           7              ...
Sale Section EvaluationStrongly Agree                        7                             6.2                            ...
Final Questionnaire:What’s your attitude towards the brand?    7    6    5     4.8 4.9        4.8                         ...
Final Questionnaire:Brand Perception              Nordstrom                       Neiman Marcus                   Pre   Po...
Final Questionnaire                             Nordstrom Neiman       Saks Fifth   Lord &                                ...
Final Questionnaire:Net Promoter Score      NPS           =   % of Promoters                         (9s and 10s)         ...
Final Questionnaire:What did you like the most about the site?                Nordstrom                                   ...
Final Questionnaire:What did you like the least about the site?                Nordstrom                                  ...
We have an excitingannouncement abouta new feature…
Exciting announcement of new feature
Some Final Tips for Creating your own Benchmark Task Selection Standardized design Randomize Conditions Cleaning the D...
Selecting the Right TasksA Balance of User Data and Business Goals    What are users coming to the sites to do (core tasks...
Selecting the Right TasksA Balance of User Data and Business Goals    What are users coming to your site to do (core tasks...
Standardize Tasks Across Competitors  Introduction   Study Design   Results   Tips
Randomize Task Order                                               Task 1         Screener       Pre-task questions   Rand...
Clean the Data                                         Watch out for Cheaters                                         •   ...
Take this with you today:• Triangulate, use multiple metrics• Experimental design• Clean data before analysis• Want to lea...
Thanks so much for your attention!          Time for Q&AFollow us on Twitter @userzoom
Upcoming SlideShare
Loading in …5
×

Webinar: Retail UX Benchmarking

902 views

Published on

In this webinar we covered the results of a recent Retail UX Benchmark Study that included Nordstrom, Neiman Marcus, Saks Fifth Avenue and Lord & Taylor. Nate also gave tips on how to create your own successful competitive benchmarking study.

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
902
On SlideShare
0
From Embeds
0
Number of Embeds
164
Actions
Shares
0
Downloads
15
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Add more specific metrics -
  • In this webinar, we will show you how you can remove face to face moderation, have hundred of users participate simultaneously from different locations in a cost effective and fast way. Userzoom is an on demand research service that will automatically capture and gather both customers attitudes and behaviors.After our demonstration, we will open up to Questions, but let’s start now by diving into UserZoom software as a service.
  • In 1994 it was amazing you could purchase something online. Lot of engineering effort, it was difficult
  • Today it is easy for one person to setup a store
  • Before we get into the details of the case study want to talk a little bit more about the methodology and the design of the study. Here is a diagram guides you in terms of how you collect data based on your research question. As Alfonso mentioned the emphasis is on measuring UX “How many & how much”What people do & What people sayThis chart is a nice what of visualizing the space. How many & how much
  • Just in case you aren’t familiar with the method. Here is an explanation. This is an indirect method, there is no moderation. You participate with your own computer in your own context.
  • If we were to look at her screen we might see this.
  • Add Notes to slides  Objectives/Goals of each taskReorder task # 2=1Task #1 – Exploratory, Measuring Attitudes (quality, desirability of sales section and items)Task #2 – Validated by price, measure behavior, findability, and attitude.
  • https://s.userzoom.com/m/MiBDNTlTMzA5
  • So you’ve decided you want to do a competitive benchmarkPossible image Silo of different sections
  • Create triangle of Metrics (Efficiency, Effectiveness, Satisfaction). Usability Problems
  • Create triangle of Metrics (Efficiency, Effectiveness, Satisfaction). Usability Problems
  • Use Twins
  • Consists ofIntroductory and Final MessagesInitial QuestionnaireTask Descriptions Post Task QuestionsFinal Questionnaire
  • Consists ofIntroductory and Final MessagesInitial QuestionnaireTask Descriptions Post Task QuestionsFinal Questionnaire
  • Consists ofIntroductory and Final MessagesInitial QuestionnaireTask Descriptions Post Task QuestionsFinal Questionnaire
  • Webinar: Retail UX Benchmarking

    1. 1. Webinar:Retail UX Benchmarking: Which BrandProvides the Best Website Experience? Nate Colker| UserZoom 12/13/2012
    2. 2. Intros Alfonso de la Nuez Nate Colker Co-Founder & Co-CEO UX Researcher Twitter Hashtag #uzwebinar Introduction Study Design Results Tips
    3. 3. Overview• Defining UX Benchmarking• Luxury department store Competitive Industry Study• Tips for building your own competitive benchmarking study Introduction Study Design Results Tips
    4. 4. Define Benchmarking UX Benchmarking is the process “Benchmarking is the of comparing a website’s (or any process of comparing digital product) ones business performance, specifically in processes and relation to UX metrics, including performance metrics to among others industry bests or best effectiveness, efficiency, overall practices from other satisfaction and quality industries.” perception, to another that is widely considered to be an industry standard benchmark or best practice Introduction Study Design Results Tips http://en.wikipedia.org/wiki/Benchmarking
    5. 5. Definining UX Benchmarking & Metrics Task Completion Task ConfidenceNumber of Clicks Task Level - Single Ease QuestionNumber of Page Views Test Level - System Usability Scale Net Promoter Score Task Time Satisfaction Introduction Study Design Results Tips
    6. 6. Define measurable & comparable UX-related KPIs  Jeff Sauro, Measuringusability.com Introduction Study Design Results Tips
    7. 7. Why do we Care about Measuring User Experience?Ordering a Pizza Online (1994) Introduction Study Design Results Tips http://articles.chicagotribune.com/1994-08-22/news/9408230031_1_pizza-hut-santa-cruz-operation-internet
    8. 8. Why do we Care about Measuring User Experience?Ordering a Pizza Online (2012) Introduction Study Design Results Tips
    9. 9. The World of User Experience Testing Introduction Study Design Results Tips http://www.useit.com/alertbox/user-research-methods.html Chart by Christian Rohrer
    10. 10. Unmoderated Usability Testing Introduction Study Design Results Tips
    11. 11. Unmoderated Usability Testing Introduction Study Design Results Tips
    12. 12. Case Study: 4 Brands Introduction Study Design Results Tips
    13. 13. Study Summary• Between subjects design• 50 Participants per brand (200 total)• 2 Tasks + survey questions• Randomized task order• Recruitment criteria – Must shop online – Shopped at luxury department stores• Total study length : ~14 minutes• Used a panel vendor• 3-4 days to field the study (11/20-11/23) Introduction Study Design Results Tips
    14. 14. Task DescriptionsTask 1 Objective: measure findability of a common apparel item (“scavenger hunt task”)Task DescriptionPlease find the price of a grey wool dress by Eileen Fisher. Please note down the price, asyou will be asked for it later.Task 2 Objective: Access desirability and perceived quality of the sales section (“open task”)Task DescriptionPlease take your time to explore the sale section and look for things that are of interest toyou. Once you feel you have spent an adequate amount of time exploring this section, clickon the “I’m Finished” button. We will then ask you some questions about your experiencewith the sale section of Nordstrom.com. Introduction Study Design Results Tips
    15. 15. Case Study Preview Introduction Study Design Results Tips
    16. 16. Task 1 Completion Rates: Find a dress Task Success Rate 100% 90% 84% 80% 74% 72% 70% 60% 54% 50% 40% 30% 20% 10% 0% Nordstrom Neiman Marcus Saks Lord & TaylorThe completion rates on Nordstrom, Saks and Lord & Taylor were significantly better than on Neiman Marcus (90% Confidence Level) Introduction Study Design Results Tips
    17. 17. Disaster Rates (Highest Confidence but Failure) 100% 8% 4% 14% 16% 90% 12% 80% 20% 10% 70% 32% 60% 50% Disaster Failure 40% 84% 74% Success 72% 30% 54% 20% 10% 0% Nordstrom Neiman Marcus Saks Lord & Taylor Introduction Study Design Results Tips
    18. 18. Task-Level Ease Overall, how difficult or easy was it to find the price of a grey wool dress?Very Easy 7 6 5.9 6 5 4.8 3.9 4 3 2Very Difficult 1 Nordstrom Neiman Marcus Saks Lord & Taylor Introduction Study Design Results Tips
    19. 19. Open-ended responses from participants whothought the task was difficult (1, 2 or 3)Briefly describe why you chose the number you did.Nordstrom Too many choices. Small print beneath the pictures. I couldn’t find it and wasted a lot of time trying. I never found the grey wool dress therefore I couldnt know the cost. This is obviously a failure at the task asked of the survey. I gave myself the lowest score possible, not for lack of trying thought!Neiman Marcus I could not find the dress. I searched under eileen as the designer. I typed in grey wool dress in the search box and only two childrens dresses came up. I looked under sale items too and found three dresses by this designer non of which were wool. I put Eileen Fisher grey wool dress in search bar and search query came up with "can not find Eileen Fisher grey wool dress" so I put in search Eileen Fisher wool dress and I found dresses none were grey.... I then looked through other pages and did not see a grey dress. Search didn’t work - came back with not found, didn’t have intuitive drop-downs, under designers didn’t have dresses as option. No description said was a grey wool dress, closest was gray 3/4 length dress. Took to long to weed through options looking for dresses by Eileen. Introduction Study Design Results Tips
    20. 20. Open-ended responses from participants whothought the task was difficult (1, 2 or 3)Briefly describe why you chose the number you did.Saks Fifth Avenue The Saks website kept sending me to Saks Direct - it took a number of attempts (7? 8?) to get to dresses, then find the red wool dress assigned. It was very frustrating. Had the site worked well, it should have been easy. Number of steps to complete task It was not a difficult site to navigate, but I would rather narrow down a category rather than to search for a specific item using the search boxLord & Taylor Its the only grey dress that said it was wool. it kept bringing up all different thing unrelated. Because i could not figure out which brand of grey wool dress it was. It took time to find the right dress, and I’m not completely sure it was the right one Introduction Study Design Results Tips
    21. 21. Task Time: Completed tasks only 350 300 293 250 236 200 Seconds 146 150 126 100 50 0 Nordstrom Neiman Marcus Saks Lord & TaylorCompletion time on Lord & Taylor was significantly lower than it was on Nordstrom & Neiman Marcus (90% Confidence Level) Introduction Study Design Results Tips
    22. 22. Post task questionsStrongly Agree 7 6.3 6.2 6.2 6.2 6 5.9 6 5.5 5.2 5.1 5 4.7 4.4 It was easy to navigate the 4.1 website 4 It was clear how to start finding the price of the wool dress 3 I am satisfied with the amount of time it took to complete the taskStrongly Disagree 2 1 Nordstrom Neiman Marcus Saks Lord & Taylor Introduction Study Design Results Tips
    23. 23. Task 2Evaluate the sales sectionPlease take your time to explore the sale section and look for things thatare of interest to you. Once you feel you have spent an adequate amountof time exploring this section, click on the “I’m Finished” button. We willthen ask you some questions about your experience with the sale sectionof Nordstrom.com Introduction Study Design Results Tips
    24. 24. Sale Section EvaluationHow satisfied are you with the number of items that were on sale?Very Satisfied 7 6 5.5 5.2 5.3 4.9 5 4 3 Not at all Satisfied 2 1 Nordstrom Neiman Marcus Saks Lord & Taylor Introduction Study Design Results Tips
    25. 25. Sale Section EvaluationHow satisfied are you with the sale prices?Very Satisfied 7 6 5 3.9 4 3.7 3.3 3 3 Not at all Satisfied 2 1 Nordstrom Neiman Marcus Saks Lord & Taylor Introduction Study Design Results Tips
    26. 26. Sale Section EvaluationStrongly Agree 7 6.2 6.2 6 6 5.7 5.7 5.4 5.5 5.1 It was easy to navigate the sale 5 4.8 4.5 section 4 3.9 4 I found some items that I liked 3 I would consider checking out the sale section again in the future Strongly Disagree 2 1 Nordstrom Neiman Marcus Saks Lord & Taylor Introduction Study Design Results Tips
    27. 27. Final Questionnaire:What’s your attitude towards the brand? 7 6 5 4.8 4.9 4.8 4.8 4.6 4.3 4.4 4.1 4 Before After 3 2 1 Nordstrom Neiman Marcus Saks Lord & Taylor Introduction Study Design Results Tips
    28. 28. Final Questionnaire:Brand Perception Nordstrom Neiman Marcus Pre Post Pre Post 1. Trendy 30% 62% +32% 1. Expensive 60% 80% +20% 2. Good Value 8% 26% +18% 2. Trendy 34% 48% +14% 3. Friendly 12% 30% +18% 3. Trustworthy 24% 18% -6% 4. Exciting 18% 32% +14% 4. Customer 20% 8% -12% Focused Saks Fifth Avenue Lord & Taylor Pre Post Pre Post 1. Trendy 50% 62% +12% 1. Expensive 48% 82% +34% 2. Expensive 80% 88% +8% 2. Trendy 22% 52% +30% 3. Exciting 16% 12% -4% 3. Exciting 8% 26% +18% 4. Trustworthy 28% 12% -16% 4. Good Quality 42% 58% +16% Introduction Study Design Results Tips
    29. 29. Final Questionnaire Nordstrom Neiman Saks Fifth Lord & Marcus Avenue TaylorThe website is easy to use 5.8 5.1 6.2 6.3I feel comfortablepurchasing from this 5.2 4.3 5.1 5.2websiteI find the website to beattractive 5.5 5.3 5.5 5.6The website has a cleanand simple presentation 5.7 5.3 5.1 5.9I will likely visit thiswebsite in the future 4.5 3.6 3.9 4.4On a scale from 1 to 7 Study Design Introduction Results Tips
    30. 30. Final Questionnaire:Net Promoter Score NPS = % of Promoters (9s and 10s) - Detractors (0 through 6) 0 1 2 3 4 5 6 7 8 9 10 NPSNordstrom 10% 0% 2% 6% 6% 16% 14% 12% 18% 10% 6% -38%Neiman Marcus 8% 10% 10% 4% 8% 18% 10% 12% 18% 6% 6% -56% 12% 2% 10% 0% 2% 18% 16% 10% 10% 6% 14% -40%Saks Fifth AvenueLord & Taylor 10% 0% 2% 6% 6% 16% 14% 12% 18% 10% 6% -38% Detractors Promoters Introduction Study Design Results Tips
    31. 31. Final Questionnaire:What did you like the most about the site? Nordstrom Neiman Marcus “I really liked that you can change the colors of “The search function quickly found what I was different items without being forced to open that looking for.” particular item, taking you to another screen.” “I like how the pictures of the apparel would show front and back when I hovered the mouse over “That I could click on the pictures and see the the picture.” back of the garment.” Saks Fifth Avenue Lord & Taylor “The amount of items I could see on one window. “It was easy to navigate and find products I was I hate hitting next over and over again.” looking for.” “Garment being worn by a moving person” “Search options. By Designer By Color By Price. “Easy to find anything. love the search/browse by Very Easy and nice to use.” size, color etc.” Introduction Study Design Results Tips
    32. 32. Final Questionnaire:What did you like the least about the site? Nordstrom Neiman Marcus “Small print beneath the pictures.” “The filters by designer only includes size color “Hard to search compared to other sites I have etc. not dresses, tops etc.” been on.” “Drill through for finding the dress was difficult.” “The models are ultra slim, it can be discouraging “It was difficult to navigate when searching for for someone who isn’t a size 2 or size 0.” something specific.” Saks Fifth Avenue Lord & Taylor “A little too expensive to buy more than one or “The prices were a little steep.” two items.” “They were very expensive. I also felt that for the “The prices, but that is not the website” price you would think they would have more trendy items of clothing. I didn’t find anything that Interested me.” Introduction Study Design Results Tips
    33. 33. We have an excitingannouncement abouta new feature…
    34. 34. Exciting announcement of new feature
    35. 35. Some Final Tips for Creating your own Benchmark Task Selection Standardized design Randomize Conditions Cleaning the Data Introduction Study Design Results Tips
    36. 36. Selecting the Right TasksA Balance of User Data and Business Goals What are users coming to the sites to do (core tasks) ? Are there any tasks that are required to use your site? What drives revenue on your site? Introduction Study Design Results Tips
    37. 37. Selecting the Right TasksA Balance of User Data and Business Goals What are users coming to your site to do (core tasks) ? - Browsing new sale items Are there any tasks that are required to use your site? - Creating an account? What drives revenue on your site? - Completing a purchase Introduction Study Design Results Tips
    38. 38. Standardize Tasks Across Competitors Introduction Study Design Results Tips
    39. 39. Randomize Task Order Task 1 Screener Pre-task questions Randomized Final Questions Task 2Control for ordering effects• Participants become more proficient as they use the site etc.There are exceptions, sometimes tasks must be completed in order Introduction Case Study Results Analysis
    40. 40. Clean the Data Watch out for Cheaters • Very Low task times • 0 Clicks • Nonsense Answers • Enable Quality Controls Introduction Study Design Results Tips http://en.wikipedia.org/wiki/File:Cheating.JPG by Hariadhi
    41. 41. Take this with you today:• Triangulate, use multiple metrics• Experimental design• Clean data before analysis• Want to learn more?
    42. 42. Thanks so much for your attention! Time for Q&AFollow us on Twitter @userzoom

    ×