The Use of an Unmoderated,  Online Usability Testing Service to Test a WebsiteBrandon Kopp & Bill Mockovak           Resea...
TryMyUI• Provides panel of participants that are presented with user  defined tasks• You receive a video screenshot of the...
Many Alternatives                         Userzoom.comUserTesting.com                                            UTest.com...
The BLS.gov Dropdown Usability Test                     • During earlier                       usability                  ...
Prototypes• Test design alternatives for mega-dropdowns             Type of MenuPrototype                         Display ...
Prototype A         • Need to click on           tab to open         • Click on X or           Arrow to close         • Ab...
Prototype B         • Dragging mouse           over menu will           open it         • Menu closes           when you m...
Prototype C         • Current, ‘full’           list of topics         • Menu drops           after delay, with           ...
Prototype D         • Control Condition         • Current menu on           BLS.gov         • Menu drops           immedia...
Method• Define Participant Criteria                            Prototype                                                  ...
Tasks1. Find a publication called the Occupational   Outlook Quarterly. This is an online magazine   about jobs and careers.
Tasks1.   Occupational Outlook Quarterly2.   Mass Layoffs3.   International Unemployment4.   Green Jobs5.   Survey of Occu...
What the Participant Sees    TaskInstructions                     Browser /                    Webpage ViewRecording  Time...
What The Experimenter Sees      Video of         Task IndexParticipant’s Screen
Example Videos   Feedback (A; Task 8)   Feedback (D; Task 6)
Task Success and Time To Complete                                      # of Tasks Successfully Completed                  ...
First Click              Subject   Databases Economic                  ElsewhereTask   Home                               ...
Method• Define Participant Criteria                            Prototype                                                  ...
Open-Ended Questions       • How easy were these tasks to complete? Were they very         easy, easy, neither easy nor di...
Feedback/Recommendations• Final Recommendation:  Prototype C• Several participants were  surprised that the Home tab  had ...
Feedback/Recommendations• Subject Areas menu extended off the screen  on some participants’ browsers requiring  scrolling
Advantages• Cost ($)        Method                CostTraditional, In-Lab    $40 per participantTryMyUI.com            $35...
Advantages• Cost ($)• Cost (Time)           Task                          In-Lab                                   Web    ...
Advantages•   Cost ($)•   Cost (Time)•   Short data collection period•   Participants that can be selected based on criter...
Disadvantages• No follow-up• Cannot correct navigation errors• Limited to 20 minutes*• Task completion time, success, and ...
Discussion • What testing service have you used? • What are the advantages/disadvantages of   that service?          Loop1...
Contact Information      Brandon Kopp       Research PsychologistOffice of Survey Methods Research         www.bls.gov/osm...
Upcoming SlideShare
Loading in...5
×

Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)

322

Published on

Given at UXPA-DC's User Focus Conference, Oct. 19, 2012

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
322
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Feedback A, Task 8 (Students & Teachers): Reacts to Home as a dropdownFeedback D, Task 6 then jump to Task 7 (Injuries & Illnesses): Guy spends 5:20 looking for Workplace Injuries. Doesn’t find it until next task.
  • Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)

    1. 1. The Use of an Unmoderated, Online Usability Testing Service to Test a WebsiteBrandon Kopp & Bill Mockovak Research Psychologist Office of Survey Methods Research UXPA-DC User Focus Conference 19 October 2012
    2. 2. TryMyUI• Provides panel of participants that are presented with user defined tasks• You receive a video screenshot of the participant’s computer with an audio voiceover of their comments 2
    3. 3. Many Alternatives Userzoom.comUserTesting.com UTest.com OptimalWorkshop.comOpenHallway.com Loop11.com
    4. 4. The BLS.gov Dropdown Usability Test • During earlier usability testing, several users had problems with immediately activating dropdown menus • Users would accidentally click menu links and get lost
    5. 5. Prototypes• Test design alternatives for mega-dropdowns Type of MenuPrototype Display Width List of Subject Areas Control Partial list of content, with A Click to show/close Fixed, 3 columns ‘view all’ option B Hover, no delay Fixed, 1 column Compact C Hover, with delay Fixed, 3 columns Current content D Hover, no delay Fixed, 3 columns Current content
    6. 6. Prototype A • Need to click on tab to open • Click on X or Arrow to close • Abbreviated list is initially presented • User clicks View All to see entire list
    7. 7. Prototype B • Dragging mouse over menu will open it • Menu closes when you move mouse off it (no clicking is necessary) • Higher-level categories are displayed
    8. 8. Prototype C • Current, ‘full’ list of topics • Menu drops after delay, with mouseover • Moving off menu makes it disappear quickly
    9. 9. Prototype D • Control Condition • Current menu on BLS.gov • Menu drops immediately (no delay) with mouseover • Moving off menu makes it disappear quickly
    10. 10. Method• Define Participant Criteria Prototype # of Participants Gender: Any Age: 18-55 A 11 Country: U.S. Income: Any B 14 Education: Any Employment: Any C 10 Computer Experience: Beginner-Intermediate or Expert D 10• Participants complete up to 10 tasks while providing verbal feedback
    11. 11. Tasks1. Find a publication called the Occupational Outlook Quarterly. This is an online magazine about jobs and careers.
    12. 12. Tasks1. Occupational Outlook Quarterly2. Mass Layoffs3. International Unemployment4. Green Jobs5. Survey of Occupational Illness and Injury6. Strikes and Lockouts7. Students and Teachers8. NY State Wages
    13. 13. What the Participant Sees TaskInstructions Browser / Webpage ViewRecording Time 14
    14. 14. What The Experimenter Sees Video of Task IndexParticipant’s Screen
    15. 15. Example Videos Feedback (A; Task 8) Feedback (D; Task 6)
    16. 16. Task Success and Time To Complete # of Tasks Successfully Completed Time to Complete Task 8 120 7.4 7 6.6 6.7 Time to Complete Task (in sec.) 96.9# of Tasks Completed (out of 8) 6.2 100 87.8 6 83.8 77.8 80 5 4 60 3 40 2 20 1 0 0 A B C D A B C D
    17. 17. First Click Subject Databases Economic ElsewhereTask Home Publications Areas & Tools Release on page 1 0 3 3 0 32 7 2 3 17 16 7 0 2 3 3 26 9 6 0 1 4 2 33 4 2 0 2 5 1 34 6 1 1 0 6 3 38 3 1 0 0 7 4 25 2 1 4 7 8 1 25 9 1 0 6
    18. 18. Method• Define Participant Criteria Prototype # of Participants Gender: Any Age: 18-55 A 11 Country: U.S. Income: Any B 14 Education: Any Employment: Any C 10 Computer Experience: Beginner-Intermediate or Expert D 10• Participants complete up to 10 tasks while providing verbal feedback• Following testing, participants write responses to 4 open-ended questions
    19. 19. Open-Ended Questions • How easy were these tasks to complete? Were they very easy, easy, neither easy nor difficult, difficult, or very difficult? How easy do you think these tasks would be for an average American citizen using this website? 80% of Participants Selecting Rating 70 67 69 61 60 50 47 39 40 33 30 23 20 20 17 17 10 8 0 0 Easy/Very Easy Neither Easy Nor Difficult Difficult/Very Difficult A B C D
    20. 20. Feedback/Recommendations• Final Recommendation: Prototype C• Several participants were surprised that the Home tab had a menu• No items under Workplace Injuries subcategory on Subject Areas menu
    21. 21. Feedback/Recommendations• Subject Areas menu extended off the screen on some participants’ browsers requiring scrolling
    22. 22. Advantages• Cost ($) Method CostTraditional, In-Lab $40 per participantTryMyUI.com $35 per participantUserTesting.com $39 per participantUTest.com --OpenHallway.com $49-$199 per monthLoop11.com $350 per project
    23. 23. Advantages• Cost ($)• Cost (Time) Task In-Lab Web 20 minutes total; 30 minutes total; Requesting participants explaining criteria to recruiter specifying test groups and criteria 0 minutes; Screening participants 10 minutes per participant done by TryMyUI 0 minutes; Scheduling participants 15 minutes per participant study done at participant convenience 60 minutes total; Preparing for interviews 10 minutes per participant setting up web survey and tasks Total (for 45 interviews) 26.6 hours 1.5 hours 3 weeks; 1 – 2 days; Data collection period based on interviewer schedule Depends on participant criteria
    24. 24. Advantages• Cost ($)• Cost (Time)• Short data collection period• Participants that can be selected based on criteria• Videos can be shared• Participants are skilled/trained – At thinking aloud – At dealing with problems – At evaluating websites• Can get replacements for unusable cases
    25. 25. Disadvantages• No follow-up• Cannot correct navigation errors• Limited to 20 minutes*• Task completion time, success, and other quantitative methods have to be captured manually*• No rating scales*• Panel participants may have selection biases that make them different from a ‘typical’ user * May be different on other testing services
    26. 26. Discussion • What testing service have you used? • What are the advantages/disadvantages of that service? Loop11.com UTest.com Userzoom.comUserTesting.com TryMyUI.com OpenHallway.com OptimalWorkshop.com
    27. 27. Contact Information Brandon Kopp Research PsychologistOffice of Survey Methods Research www.bls.gov/osmr 202-691-7514 kopp.brandon@bls.gov
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×