Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Copyright T. Komischke, H. Strub, T. Sachs, J. Chin 2014
Lect 13: Theory &
Practice of Usability
Studies
User Experience D...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
An broad overview of context
- Who, Anyone
- When, Anytime
- Where, Anywh...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Doctor Who? Who is John Chin?
Education
 Ph.D. in Cognitive Psychology s...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Enough about me, what about you?
Who are you?
 Education and Work experi...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Bottom line goals…
 1) Be neutral and stick to the facts
 2) Advocate f...
Planning a Test
 Who: Low vision or Blind user
 Where: Standing on the sidewalk at a bus stop
 When: During rush hour i...
What are the research questions
 Research questions are not tasks
 Key questions drive focus of the testing and tasks
 ...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Where is project in the product life
cycle?
 Early?
 Are there requirem...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Where is the project in the product life
cycle? (continued…)
 Late?
 UI...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Where is the project in the product life
cycle?
 I prefer early-middle, ...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
What do you know?
 1) The human (in general)
 Topics include:
 Cogniti...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
What do you know?
 2) Your business:
 If a consultancy (or an experienc...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
What do you know?
 2) Your business
 If you’re an employee:
 Get to kn...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
What do you know?
 2) Your business, if you’re an employee, continued
 ...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
What do you know?
 2) Your business, if you’re an employee, continued
 ...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
What do you know?
 2) Your business, if you’re an employee, continued
 ...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
What will you test with? (1 of 3)
 Paper prototype?
 Cheap, usually eas...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
What will you test with? (2 of 3)
 Visio? (or similar, PowerPoint can be...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
What will you test with? (3 of 3)
 Possibly bad addition to cost, time, ...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
How will you record the test?
 Screen critical – see what happened
 Aud...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
How many participants to run? (1 of 3)
 Nielsen’s rules of thumb:
 5 or...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
How many participants to run? (3 of 3)
 Hank’s rules of thumb
 Arrangin...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
How many participants to run? (3 of 3)
 How many customer “demographic g...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
How much time per session?
 Guesstimate: How big/complex does the UI fee...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
When to schedule sessions?
 Easiest: during business hours
 If live obs...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
When not to schedule sessions
 Consider observers as well as participant...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
How many sessions to run per day?
 Don’t make testing days too hard on a...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
How to recruit participants? (1 of 2)
 What are you testing? How special...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
How to recruit participants? (2 of 2)
 Recruiting from your web site
 B...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Who to recruit
 If clear that there’s one demographic (or “persona”), go...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Who to recruit: Common screening
criteria (1 of 2)
 No one in your indus...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Who to recruit: Common screening
criteria (2 of 2)
 Education: I’ve know...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Special participants: Children, Disabled,
Employees
 Children:
 Want a ...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Possible Forms for participants (1 of 2)
 Informed consent form has two ...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Possible Forms for participants (2 of 2)
 NDA (Non-disclosure Agreement)...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Don’t fight the law (but then I’m married
to a lawyer)
 This is probably...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
How much to pay participants?
 Goldilocks rule of thumb:
 Not too much,...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Should I use floaters?
 Floaters are backup participants, usually recrui...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
How to record a test?
 Morae, by Techsmith: The “Honda Fit” of usability...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Recording a test – More on Morae
 Morae also good for portable lab – onl...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Recording a test: Lots of fancier
software, but..
 Pricier, need fancier...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Recording a test: Back to mobile
 How to get a good screen? Apple to HDM...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Other Technology Issues:
 Eye Tracking: compelling to watch, help observ...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
The Moderator’s Guide (1 of 4)
 Introduction
 Why you’re there
 That t...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Moderator’s Guide (2 of 4)
 Think about time. Consider optional tasks to...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Moderator’s Guide (3 of 4)
 Questions to ask: At end of Session
 I like...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
47
The original SUS
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Moderator’s Guide (4 of 4)
 Thoughts on SUS:
 Alternates positive and n...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Cautions on locations (1 of 2)
 Dedicated lab:
 Will you use it enough?...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Cautions on locations (2 of 2)
 Remote Testing
 Success depends on inst...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Screen-sharing software, for remote live
observers
 Easiest to use the c...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Running a Test52
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Be like Johnny Carson
(Used to host The Tonight Show)
 Always genial, fr...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Taking notes during:
 If you follow template strictly and have time, exc...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Things to watch during session55
Copyright T. Komischke, H. Strub, T. Sachs, 2014
What to do if: (1 of 2)
 Participant is no-show
 Ask to arrive 15 minut...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
What to do if: (2 of 2)
 Early in session, can see that participant does...
Copyright T. Komischke, H. Strub, T. Sachs, 2014
After a test
 Debriefs are great. Consider recording them if you don’t t...
Copyright T. Komischke, H. Strub, T. Sachs, J. Chin 2014
Questions?
Upcoming SlideShare
Loading in …5
×

Lect2-3 Theory-Practice_UsabiilityStudies-Oct2014(1)

128 views

Published on

  • Be the first to comment

  • Be the first to like this

Lect2-3 Theory-Practice_UsabiilityStudies-Oct2014(1)

  1. 1. Copyright T. Komischke, H. Strub, T. Sachs, J. Chin 2014 Lect 13: Theory & Practice of Usability Studies User Experience Design III John Chin, Ph.D. jchin@acm.org 973-746-7438 October, 2014 Contact Information
  2. 2. Copyright T. Komischke, H. Strub, T. Sachs, 2014 An broad overview of context - Who, Anyone - When, Anytime - Where, Anywhere - Why, Motivations - What, Task The missing link - How  Along with some depth on many topics 2
  3. 3. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Doctor Who? Who is John Chin? Education  Ph.D. in Cognitive Psychology specializing in  Human Factors  Human-Computer Interaction  University of Maryland, College Park (advisor: Kent Norman) Work Experience  Currently, User Experience Strategist @ Verizon Wireless  User research, Writing UI requirements, Usability testing,  Accessibility, Connected Car, Messaging, 3 patents pending  HiTech – Contractor Homeland Security, TSA  Cognetics – Trizetto Group  Avaya, Alcatel-Lucent, AT&T Bell Labs, Telcordia  Lewis and Clark College 3
  4. 4. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Enough about me, what about you? Who are you?  Education and Work experience  Interests and specialty Why are you here?  Goals and Objectives  Career development, job? How can I help?  Needs and Aspirations  Networking 4
  5. 5. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Bottom line goals…  1) Be neutral and stick to the facts  2) Advocate for your users, all of them  Recruit representative sample of eventual product/system users  Participants – users testing the design/implementation  Audience – observers (possible development team members)  Report findings and possible solutions to readers or stakeholders  Gain better insights and understanding of the issues  Hope to find possible solutions  Few hard and fast answers: mainly tradeoffs 5
  6. 6. Planning a Test  Who: Low vision or Blind user  Where: Standing on the sidewalk at a bus stop  When: During rush hour in the morning  What: Pedestrian getting directions to library  Why: Pick up model created on 3D printer at library  How: Mobile device navigation application 6
  7. 7. What are the research questions  Research questions are not tasks  Key questions drive focus of the testing and tasks  Research:  Can people successfully order products?  Will customers be able to pay their credit card bill?  Will radiologists be able to diagnose cases, without errors, efficiently?  Will consumers like this better than their current camcorder?  Tasks:  Typically tasks should be frequent and/or important  How long will it take customers to find the due date for their current statement? 7
  8. 8. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Where is project in the product life cycle?  Early?  Are there requirements that need to be gathered?  Or are requirements finalized?  What’s the shape of design concepts?  Is this the next version or a new product?  Some will say “too early for usability” since much is undefined  Some will say “Do a (user-centered) focus group instead”  Not yet 100% clear what it does  Bottom line: A lot is vague, testing can have big impact, but make sure you’re exploring relevant issues  Middle of process?  Requirements finalized  Screens rough, or designed for usability  Lots already determined: Use it. [or not, at your own risk]  Bottom line: Testing can have moderate impact, Fast turnaround important. Choose issues where you’ll be listened to. 8
  9. 9. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Where is the project in the product life cycle? (continued…)  Late?  UI is likely designed  Possibly even coded  No leeway on requirements (functionality)  Delivery possibly even scheduled  What is the goal?  If the team will update the UI, could have a meaningful impact  Fitting some request “It needs usability”  No-win if in “police” role  Stakeholder may be looking for “lipstick on the pig”, and endorsement that “Usability says it’s okay”  If it misses the mark, you’re in a no-win situation  Big risk: once it’s delivered, product manager might not want to change UI once existing customers invested time to learn it.  Bottom line: be diplomatic & realistic. 1) Consider risks to company if don’t fix enough, 2) what can be fixed later, and 3) process so won’t repeat this mess next release or product. 9
  10. 10. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Where is the project in the product life cycle?  I prefer early-middle, or early  Beware being brought in late: a no-win game  Especially if project planned without user research.  No resources to create prototype (and no time)  No resources to incorporate results (or time)  No one might care  Even if you have a sponsor, you’re being set up for a battle  Focus on your data. It’s a business decision re what to do.  Be careful – think high level  I’ve been surprised many times – people and organizations that seem to have mature processes  “organizations” – since companies have silos: one or many organizations in a company may be awesome. Be careful every time you meet a new one. 10
  11. 11. Copyright T. Komischke, H. Strub, T. Sachs, 2014 What do you know?  1) The human (in general)  Topics include:  Cognitive processor (and memory)  Perceptual systems  Social interaction (  Emotion  Other topics: Persuasion, attraction, etc.  If you have time to read, get applied books on these topics  For topics you really like, get theoretical books; read research papers.  If you don’t already have a good book list, I’ll have suggestions next week 11
  12. 12. Copyright T. Komischke, H. Strub, T. Sachs, 2014 What do you know?  2) Your business:  If a consultancy (or an experienced department)  How does your firm approach problems?  Why do customers choose it?  Know previous reports – especially with your client  Know how to leverage templates: research plan, test documents, reports  Practically: consider ways to promote future work for colleagues – both your team and other teams [without compromising my integrity] 12
  13. 13. Copyright T. Komischke, H. Strub, T. Sachs, 2014 What do you know?  2) Your business  If you’re an employee:  Get to know others in complementary departments  Market Research (make friends, which also reduces risk of being seen as competition)  Customer Service (Call Center “top 10” lists)  Meet, go to lunch, read reports  VOC (Voice of the Customer) data  Becoming increasingly popular  Whatever you have available.  ForeSee: Survey at end of visits 13
  14. 14. Copyright T. Komischke, H. Strub, T. Sachs, 2014 What do you know?  2) Your business, if you’re an employee, continued  VOC data, continued  OpinionLab  Input possible on any page  Works best with meaningful URLs  Beware:  Length. Tempting to continue asking questions, but not paying customers  Order matters: What goes early late, which questions go before others 14
  15. 15. Copyright T. Komischke, H. Strub, T. Sachs, 2014 What do you know?  2) Your business, if you’re an employee, continued  Analytics (i.e., big data) 15 Copyright M. Tremaine, T. Sachs, A. Melewski 2012 This is a sample, not an Adobe endorsement
  16. 16. Copyright T. Komischke, H. Strub, T. Sachs, 2014 What do you know?  2) Your business, if you’re an employee, continued  Analytics (i.e., big data)  The ability to track…  Beware: What the data makes easy, versus what is of interest  I have been able to track interactions on a page (sometimes every page), but cannot track a user through paths  Who are your company’s customers? Who use online systems?  How do people access your stuff?  What are trends?  Social networking analysis: As many as company follows  Reports: JD Power, Forrester, Keynote, etc.  Data collection services, and reports, (usually not cheap)  Technology trends  The continuing rise of mobile, tablets  The impact of OS, browsers, etc.  Zeitgeist: e.g., Responsive Web Design 16
  17. 17. Copyright T. Komischke, H. Strub, T. Sachs, 2014 What will you test with? (1 of 3)  Paper prototype?  Cheap, usually easy to create, conceptual  Who will create?  Will it be respected internally?  “Photoshop” [i.e., online] paper prototype more “real” than paper  Paper does communicate “conceptual”, room for change  I usually test paper as concepts that extend prototype(s) 17
  18. 18. Copyright T. Komischke, H. Strub, T. Sachs, 2014 What will you test with? (2 of 3)  Visio? (or similar, PowerPoint can be fast)  I’m not a “designer” but I can design fairly well in Visio  If my concepts, I’m in control: can test what I want (or edit myself)  If my concepts, distracting from focusing on testing  Key point: Hyperlinks, so it looks like it’s functioning  More about screens than functionality (won’t function well, i.e., fields cannot be filled in)  Something that really runs?: Html, iRise, Axure, Dreamweaver, Ruby, etc.  Functionality is great addition to realism, believability 18
  19. 19. Copyright T. Komischke, H. Strub, T. Sachs, 2014 What will you test with? (3 of 3)  Possibly bad addition to cost, time, and pain, since someone has to build & support the prototype.  Arranging for someone to create the prototype can be hard, takes time. Don’t assume getting resources will be fast/easy.  For many audiences, many products/systems, this is a necessity  Even more important for mobile UIs – I believe form factor really matters for mobile  What do you think about where mobile is tested? [lab vs. field]  Trend of hybrid device approach: Big addition to complexity  Prototype web, tablet, phone?  iOS ? (pre-7 & 7?), Android (“flavor”?), Windows mobile, Blackberry?  App versus mobile web? 19
  20. 20. Copyright T. Komischke, H. Strub, T. Sachs, 2014 How will you record the test?  Screen critical – see what happened  Audio critical – hear what happened,  High quality audio often undervalued.  Stereo audio recordings can be especially useful, since with headphones you can pick out voices  Very useful: The participant’s face, to see emotions  Still useful – hands on keyboard, on artifacts, etc.  Mobile technology especially challenging to record  Harder to get good screen image  Want to see what hands do, but hands/fingers cover the screen  Still want to see face of the participant 20
  21. 21. Copyright T. Komischke, H. Strub, T. Sachs, 2014 How many participants to run? (1 of 3)  Nielsen’s rules of thumb:  5 or a few more. Go higher if test has truly different groups  Consider more rounds of testing, if prototyping and time are available  I often have trouble getting prototyping for multiple rounds of testing)  Bump up numbers a little if only one round of testing 21
  22. 22. Copyright T. Komischke, H. Strub, T. Sachs, 2014 How many participants to run? (3 of 3)  Hank’s rules of thumb  Arranging a usability test is hard  Unless really simple, run at least 3 days  60 minute sessions: 15 participants  90 minute sessions: 12 participants  Beware going > 4 days at a crack – exhausting, and too much to analyze  Too much for industry people to pay attention to  If need more people, run another series  Or outsource the usability 22
  23. 23. Copyright T. Komischke, H. Strub, T. Sachs, 2014 How many participants to run? (3 of 3)  How many customer “demographic groups” to recruit  Rule of thumb: shoot for at least 3 people per demographic  If differing opinions, still have a tie-breaker  If a no-show, still have 2  If it’s a key customer group (i.e., “prospects”) and site is for them, increase or subdivide that group  No point in testing 12 groups for study with 15 participants 23
  24. 24. Copyright T. Komischke, H. Strub, T. Sachs, 2014 How much time per session?  Guesstimate: How big/complex does the UI feel?  If testing in cafeteria (or some place public) keep sessions really short  To cover more ground, lots of sessions covering different tasks  In lab, I believe no shorter than an hour  Lots of overhead, effort, to recruit, to prep, to get observers, to analyze  I find it hard to go > 1-1/2 hours  Sessions are tiring, for participant, moderator, observers  Hard for many types of participants to get lots of time during business hours – might get more strange people  > 90 minutes, schedule in a break (which lowers efficiency)  I’ve seen 2 and 2-1/2 hour sessions (and no break) in last year – and they worked, but moderator knew domain well 24
  25. 25. Copyright T. Komischke, H. Strub, T. Sachs, 2014 When to schedule sessions?  Easiest: during business hours  If live observers, much better to have straight days & business hours  But will you get enough of “right” people?  Who are target participants? What are their days like?  Many jobs: work day  Physicians: One early (before 9) one at lunch, 1-2 starting at 5 pm  Think about yourself. I rarely schedule before 10 in Manhattan since hard to get there and lots to get lab ready.  If won’t have live observers, can run any time, including a few per day at odd hours  For some groups, consider weekends  But if you have family or other personal issues (like you have a life), get more money to pay participants 25
  26. 26. Copyright T. Komischke, H. Strub, T. Sachs, 2014 When not to schedule sessions  Consider observers as well as participants  Holiday season is hard  If winter, have backup plan  If big storm might arrive during test, have backup plans  Allow time for analysis, and for results to be incorporated  Often have designers responding before discussion. 26
  27. 27. Copyright T. Komischke, H. Strub, T. Sachs, 2014 How many sessions to run per day?  Don’t make testing days too hard on anyone (including the moderator)  Want to be as alert, as good of a moderator for first and last session  Breaks between every session gives flexibility for buggy prototypes, late participants, rebooting computer, trying something new, etc  60 minute sessions: up to 5 per day  Breaks of at least 30 minutes per session (even if just to allow for lateness), 60-90 minutes for lunch  90 minute sessions: up to 4 per day  These are more tiring, make sure you have break time  Schedule lightly on first day of testing, to allow updating of everything – even your discussion guide, and to plan around unexpecteds  When possible, run 1 or 2 participants first day, after lunch 27
  28. 28. Copyright T. Komischke, H. Strub, T. Sachs, 2014 How to recruit participants? (1 of 2)  What are you testing? How specialized the community? What’s your budget? How much time do you have?  Many options, all involve tradeoffs  Professional recruiters: For many types of consumers, and some target audiences (like MDs)  Most say they’ll recruit anybody for any reason  Realistically, they have specialties  Most have “lists” of known people, who may be semi-professional test takers (some seem jaded)  Recruiters aren’t cheap  Lists of customers, you contact yourself  If you want to test improvements to Emerson’s HMI, want people who know today’s Emerson HMI  Beware: lists can take a LONG time to get (6 weeks or more)  Referrals from sales staff, or from technicians who know their “cool” and “boring” customers  Subscriber lists, Professional Society members (especially valuable if want special expertise) 28
  29. 29. Copyright T. Komischke, H. Strub, T. Sachs, 2014 How to recruit participants? (2 of 2)  Recruiting from your web site  But any issue having “customer information”?  Waiting room at medical clinic  Your card is given out by someone trustworthy (like a doctor)  “Mechanical Turk” at Amazon  Craig’s List, or similar ads  Your friends/acquaintances probably not diverse enough.  Beware effects of participants feeling obligated  Children can be especially challenging  You want parents to be nearby so everyone is trusting, but you have to keep them occupied 29
  30. 30. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Who to recruit  If clear that there’s one demographic (or “persona”), go for it  But consider going for sub-demographics as more participants  I usually shoot for up to 3, maybe 4 max, demographic groups.  Product space may be much broader  Target groups that are truly different, in parameters that matter to your UI  Many factors can be targeted independently, with same participants. E.g.:  Ranges for age, education, income  Separate from experience in what you’re testing  Avoid trap of recruiting only the young/well-educated  They look nice on video, have great things to say, but…  Unless all users will be young and well-educated  I learn a lot from diversity 30
  31. 31. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Who to recruit: Common screening criteria (1 of 2)  No one in your industry, or with a household member in your industry  No one in design or usability  Always ask for a range, within whatever criteria.  If open to unemployed or retired, they’re probably easier to find for “business hours” studies than people who work, so “up to 2”  Internet skill: Want people who use web themselves, often, familiar  Age: cover diversity of your user community  I’m pushing for older but not above 69  Many older users, but tend to talk more, and therefore get less done (in my experience)  I’ve had most dishonesty with elders: age (not be too old), internet skills  Consider gender distribution 31
  32. 32. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Who to recruit: Common screening criteria (2 of 2)  Education: I’ve known researchers who screen for minimum of some college for general consumer products.  Getting some without college is more honest test of usability, for those with low literacy skills – who buy and use things  I’m often fascinated with responses of those with least education  Ethnic diversity  Income: As low as appropriate. If you want some truly wealthy people, might need to specially recruit and pay more  But can ask for people whose speech is easy to understand (critical)  Where they live: City vs. suburb; and which city (Philly vs. NY) 32
  33. 33. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Special participants: Children, Disabled, Employees  Children:  Want a witness, at all times  Parent should be nearby  Make it fun  Recruit so you’ll get interested kids  Disabled  Hard enough to get disabled with full cognitive skill  Check literature if recruiting disabled without full cognitive skill  Employees  Is it their choice? Coerced?  Impact of losing time on job  Fear if criticize system, or people, when it’s recorded 33
  34. 34. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Possible Forms for participants (1 of 2)  Informed consent form has two goals:  Inform participants of what they’ll be doing, realistically, any risks (even of embarrassment), and benefits (often, interesting tasks; chance to help change a product; and get paid)  Consent: So it’s clear what they’re consenting to  Media Release:  That you are recording them  How recordings will be used (for study, nothing else)  Can protect you as well as participants: if colleague loves a clip- wants to show at a conference (“Sorry!”)  If session is that good, can go back and pay more, or recreate the scene with actors 34
  35. 35. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Possible Forms for participants (2 of 2)  NDA (Non-disclosure Agreement)  If you’re showing proprietary concepts  Explain NDA too – so understand  I’ve heard that a properly written and executed NDA protects even if people talk – you did your due diligence  Liability release (?)  In case incidentally hurt when coming to your facility  Include conditions that lawyers want  But fight for plain language  Long forms are harder to understand –consider more short forms than fewer long forms 35
  36. 36. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Don’t fight the law (but then I’m married to a lawyer)  This is probably all new for most corporate council  Draft pretty good sample forms before approaching  Shows you know something  Easier to edit than start from scratch  Be able to explain what’s there, why, who and what your intentions are  Provide articles on what you’re after, or books  Negotiate, but with the company’s best interests in mind 36
  37. 37. Copyright T. Komischke, H. Strub, T. Sachs, 2014 How much to pay participants?  Goldilocks rule of thumb:  Not too much, not to little  If doing internal testing on employees, might be limited by corporate rules (participants already missing work during work time  For external participants, ask your recruiter for advice  Consider adding in reasonable extra payment for transportation/parking costs  BTW, if paying a participant more than $500 in a year, check with company HR about possible need to report “salary” to IRS  Only an issue if have same person return 37
  38. 38. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Should I use floaters?  Floaters are backup participants, usually recruited to cover two sessions  Extremely valuable in case of no-show  Also, for when a participant is totally inappropriate  But, who has time to hang out for 2 sessions? Be careful  I use best recruiter I can, request to go for 100% turnout  I’ve had odd birds as floaters 38
  39. 39. Copyright T. Komischke, H. Strub, T. Sachs, 2014 How to record a test?  Morae, by Techsmith: The “Honda Fit” of usability recording:  $1500 for a full license  Records two streams, usually screen and webcam on face, as well as an audio stream  If built-in mic isn’t awesome, get a good external mic  1copy of Morae includes 1 copy of “Observer”, which lets people in another room watch live (delayed ~20 seconds)  Very reliable  Not bad to learn, to set up  Can take notes, and make marks, live (but I don’t see that done)  Will make highlights videos  Pretty efficient storage, so many sessions can go on a DVD 39
  40. 40. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Recording a test – More on Morae  Morae also good for portable lab – only need 2 laptops  Requirements for computers not clear: doesn’t reflect multi-core processors  In my experience, Morae has run fine on machines well below the “minimum recommended” capability  A hack to get 3 images is a hardware video mixer, that will superimpose for you. 40
  41. 41. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Recording a test: Lots of fancier software, but..  Pricier, need fancier computer and equipment, can be testier  I know many big companies use OvoStudios  My personal experience: very nice & helpful, not as reliable as I’d like  But I’ve heard of companies that find it trustworthy  Network to find something that works for your needs.  Consider:  HD looks great, awesome detail  Probably not needed for routine recordings of basic web sites 41
  42. 42. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Recording a test: Back to mobile  How to get a good screen? Apple to HDMI to DVI cable, or Apple TV hack if security allows  License supports 3 images – I think all 3 are important  [Note: target area – ask participants to keep the phone THERE 42
  43. 43. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Other Technology Issues:  Eye Tracking: compelling to watch, help observers stay awake  Beware of heat maps: If interrupt participants, etc.  Magic is uninterrupted, enough people to get good trends, so software can analyze what happened  Web site also needs to support eye tracking: overlays, modals mess things up  Software only knows the URL, not what’s on screen  Emotion tracking: GSR (how emotional) and valence (+/-)  Affectiva is a company to watch 43
  44. 44. Copyright T. Komischke, H. Strub, T. Sachs, 2014 The Moderator’s Guide (1 of 4)  Introduction  Why you’re there  That this is usability so  A) Please speak aloud (we’ll practice),  B) Testing the UI, not testing you,  C) Don’t worry about errors since prototype (not real data or system)  Ask questions (though I might not answer)  Can stop at any time, for any reason  Pay at beginning of session: Show trust (people don’t walk away)  Plan the task list  Based on the research questions  Ideally, based on your knowledge:  What are the true top tasks (consider what Gartner, JD Power say) 44
  45. 45. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Moderator’s Guide (2 of 4)  Think about time. Consider optional tasks to fill sessions for fast participants.  Or have an extra task or two, and prioritize list in pilot sessions  Moderator’s guide: By each task I like to have questions to probe, and suggested things to observe  Participant’s guide just has the tasks – other stuff is deleted  Edit moderator’s guide, either regenerate Participant’s guide, or be careful on version control.  Questions to ask:  Per task: Make sure to ask the goal of the task  Find a value? Perform a task  Otherwise, think about what you’ll learn  Can ask questions like “How quickly do you expect to do this task? How easy do you expect this task to be?  After: How easy was it to perform this task? How often do you perform this task? 45
  46. 46. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Moderator’s Guide (3 of 4)  Questions to ask: At end of Session  I like to ask:  What did you like best about the prototype system?  What did you like least about the prototype system?  [Sometimes in form of “What were 3 best & 3 worst things?”]  The SUS is trendy [SUS = System Usability Scale]  Original by John Brooke, 1986, DEC 46
  47. 47. Copyright T. Komischke, H. Strub, T. Sachs, 2014 47 The original SUS
  48. 48. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Moderator’s Guide (4 of 4)  Thoughts on SUS:  Alternates positive and negative questions, so respondent needs to be awake  Around since the 80’s, lots of experience in the industry  A few common updates [i.e., “website” instead of “system”]  Like school grading: 70 is okay, 80 good, 90 superb  But maybe UI’s are better these days -> I’ve seen mainly upper 80’s or better  Net Promoter Score (NPS): How likely are you to recommend this product/company to a friend?  Trendy, originally published in Harvard B-School Journal  Were participants supposed to learn something?  Though be careful about testing learning in 1-hour long session, unless that’s what system is for  1 hr not much time to learn, especially when many tasks 48
  49. 49. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Cautions on locations (1 of 2)  Dedicated lab:  Will you use it enough? Perhaps try lending it out, but beware…  Sufficient staff support? [reception & waiting area, space for observers who have meetings, snacks & drinks and meals]  Window: Great, but then keep observer room quiet and dark  Need to keep equipment secure  Conference rooms and a portable Morae lab  Still have to keep laptops secure  Will you always get good rooms for test when you need them?  Others’ office  Conference rooms: Will your Morae computers run on their network? If not, how will they communicate?  Fairly safe from interruptions, though.  Cubes: Beware interruptions… Phone will ring & cube’nik might need to pick up; Co-workers don’t recognize you as important  Offices: Probably okay as long as participant respects you 49
  50. 50. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Cautions on locations (2 of 2)  Remote Testing  Success depends on installing web-sharing software, but software, computers and bandwidth are all pretty good  Usually only 5 minutes, sometimes much worse  Better with web cam  Surprisingly to me, rarely interrupted, but phone calls seem to be respected  3rd party facility (i.e., market research company)  Surprisingly pricey – be prepared for unexpected costs  But they handle the details – REALLY nice  When I use a facility, I have them recruit – so one point of failure  Even professionals sometimes use cafeteria, or Starbucks  Best for quick “first impression” studies, lots of people in little time  Can recruit for more in-depth study (follow-up in lab…) 50
  51. 51. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Screen-sharing software, for remote live observers  Easiest to use the corporate standard  It’s IT-endorsed  No firewall issues  Account likely easy to set up, and/or use  If there’s a problem, you used what company wanted  Otherwise, I believe in WebEx & a telephone bridge  Usually reasonably pain-free to install  Performance (delay, quality, frame-rate) pretty good  Not cheapest, but trustworthy is important to me 51
  52. 52. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Running a Test52
  53. 53. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Be like Johnny Carson (Used to host The Tonight Show)  Always genial, friendly  Show was about the guest(s), not him  THEIR thoughts & opinions  Johnny didn’t lead  But he got guests to talk  When there was a problem, he took the blame (never the guest’s fault unless something terrible)  Was incredible at improvising  For more on Johnny, watch for rebroadcast of PBS documentary 53
  54. 54. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Taking notes during:  If you follow template strictly and have time, excel template for all key questions and observation points is neat  Time consuming to set up  Not convenient to use when with participant  Can take notes with computer  Not best for establishing rapport  Take notes at a constant pace (lots of them)  So don’t communicate what you care about  So writing when participant thinks they’ve said something important  Compatriot’s notes (ideally, not in sight of participant) can be great, but everyone looks for different things  Who will be lead on the report? 54
  55. 55. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Things to watch during session55
  56. 56. Copyright T. Komischke, H. Strub, T. Sachs, 2014 What to do if: (1 of 2)  Participant is no-show  Ask to arrive 15 minutes early: time for forms, for bathroom, and to call recruiter before session time  Call recruiter – check on reminders  Backup slots available?  If remote, can’t get to internet  Common to get weird firewall issues  Have a local copy  Prototype fails  Computer fails  Participant arrives quite late  Running really behind. Participant’s a talker, trouble, started late  Participant wants bathroom break, but already behind 56
  57. 57. Copyright T. Komischke, H. Strub, T. Sachs, 2014 What to do if: (2 of 2)  Early in session, can see that participant doesn’t fit the study?  It appears that participant lied to recruiter re something key? (After all, these studies pay well and economy is bad)  You give cash or check to participant at start of session, and he/she gets up and leaves?  Participant admits to having committed a crime on your video recording? (“When I do my taxes, I don’t put in all the receipts”)  You have strong evidence that participant is victim of or commits abuse, or is considering suicide? 57
  58. 58. Copyright T. Komischke, H. Strub, T. Sachs, 2014 After a test  Debriefs are great. Consider recording them if you don’t take good notes.  Write down first impressions of results while they’re fresh in your mind.  Get invoices in promptly: * participants *, recruiter, and any vendor you might use again  Return borrowed equipment promptly (before damaged, lost)  Listen to sessions as “background music”. You will hear things  Push self for rapid turn-around.  Consider rehash/highlights meeting immediately afterwards, recorded, to get impressions of observers  Tight deadline for report doesn’t allow forgetting  But beware getting slammed if rest of your job fell behind due to time away for testing 58
  59. 59. Copyright T. Komischke, H. Strub, T. Sachs, J. Chin 2014 Questions?

×