Your SlideShare is downloading. ×
0
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
ETL May 2010
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

ETL May 2010

1,596

Published on

The official slides from our day-long workshop, Escape The Lab

The official slides from our day-long workshop, Escape The Lab

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,596
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
32
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Please stand by ESCAPE THE LAB WILL BEGIN AT 9:00 Friday, May 7, 2010
  • 2. ESCAPE THE LAB LET’S GET STARTED MMKAY? Friday, May 7, 2010
  • 3. Friday, May 7, 2010
  • 4. I’m Cyd He’s Nate Friday, May 7, 2010
  • 5. #ETLAB Friday, May 7, 2010
  • 6. remote USABILITY Friday, May 7, 2010
  • 7. oh Friday, May 7, 2010
  • 8. and Friday, May 7, 2010
  • 9. also Friday, May 7, 2010
  • 10. NO Friday, May 7, 2010
  • 11. time Friday, May 7, 2010
  • 12. travel Friday, May 7, 2010
  • 13. OR Friday, May 7, 2010
  • 14. lasers Friday, May 7, 2010
  • 15. HOW TO BE graceful When All Your TECHNOLOGY BREAKS Friday, May 7, 2010
  • 16. PLEASE DON’T TRIP ON THE CABLES Friday, May 7, 2010
  • 17. Follow the buddy system Friday, May 7, 2010
  • 18. How about you all? Best remote research story Most important thing to get out of today? Biggest fear or skepticism about remote research? Friday, May 7, 2010
  • 19. Go TO escapethelab.com NOW Friday, May 7, 2010
  • 20. Friday, May 7, 2010
  • 21. Remote Research In person Remote Definition: Research where the participant is not sharing physical space with the researcher. % of qualitative research done remotely* *Source: BP’s best guess Friday, May 7, 2010
  • 22. What we’ll cover today Recruiting Moderated Unmoderated Results Pitfalls Friday, May 7, 2010
  • 23. We shall begin... with a little experiment Friday, May 7, 2010
  • 24. EXERCISE JUMP IN THE POOL Table groups Pick one moderator and one participant (they need to be in different locations) 10 minutes of moderating Friday, May 7, 2010
  • 25. MODERATOR Instructions Your participant has a sheet of paper and a link Goal: find out what’s hardest about presenting physical task instructions online Hint: ask them to give you a play by play of what they are doing Follow along with your own paper NO helping, even if you could. Friday, May 7, 2010
  • 26. It’s A RACE First participant to make an origami giraffe is the winner Friday, May 7, 2010
  • 27. Friday, May 7, 2010
  • 28. Surprises? Friday, May 7, 2010
  • 29. When does remote win? Page 10-15 Friday, May 7, 2010
  • 30. Native Environment Cheap ethnography, not cheap lab. Friday, May 7, 2010
  • 31. Real Time Allows self-motivated tasks. Friday, May 7, 2010
  • 32. Involved Observers Easier to collaborate in real time. Friday, May 7, 2010
  • 33. Geography Friday, May 7, 2010
  • 34. When does remote fail? Friday, May 7, 2010
  • 35. Bandwidth Yes, it’s still an issue. Friday, May 7, 2010
  • 36. Translation It’s expensive and difficult to do live. Friday, May 7, 2010
  • 37. Minors Consent is a bear. Especially if they’re under 13. Friday, May 7, 2010
  • 38. Faces We’ve all got webcams today. Most users don’t. Webcams are invasive. Friday, May 7, 2010
  • 39. CASE STUDY: Epic FAIL Friday, May 7, 2010
  • 40. What went wrong Coins were way more valuable than we expected People mis-assessed their English levels Bandwidth was really a problem Nobody plays Habbo when their parents are home to consent Friday, May 7, 2010
  • 41. making the most Friday, May 7, 2010
  • 42. TIME-AWARE Tasks Told to find purple pumps. Has to wear this for sister’s wedding this weekend. Doesn’t care. Whole different interaction. Friday, May 7, 2010
  • 43. Real Time OBservers More on page 109-111 Friday, May 7, 2010
  • 44. Change on the fly “No one seems engaged in the “Let’s add questions about who they laptop customizer” get recommendations from” Friday, May 7, 2010
  • 45. Designing Remote Studies Friday, May 7, 2010
  • 46. Friday, May 7, 2010
  • 47. Friday, May 7, 2010
  • 48. plan these things Friday, May 7, 2010
  • 49. plan these things Method Recruiting Logistics Friday, May 7, 2010
  • 50. MODERATED must-haves Friday, May 7, 2010
  • 51. MODERATED must-haves Consent Friday, May 7, 2010
  • 52. MODERATED must-haves Consent Tech set-up Friday, May 7, 2010
  • 53. MODERATED must-haves Consent Tech set-up The beef Friday, May 7, 2010
  • 54. MODERATED must-haves Consent Tech set-up The beef De-installation Friday, May 7, 2010
  • 55. MODERATED must-haves Consent Tech set-up The beef De-installation Thanks & incentive Friday, May 7, 2010
  • 56. UNMODERATED must-haves Friday, May 7, 2010
  • 57. UNMODERATED must-haves Expectation setting Friday, May 7, 2010
  • 58. UNMODERATED must-haves Expectation setting Instructions Friday, May 7, 2010
  • 59. UNMODERATED must-haves Expectation setting Instructions The beef Friday, May 7, 2010
  • 60. UNMODERATED must-haves Expectation setting Instructions The beef Thanks & incentive Friday, May 7, 2010
  • 61. Recruiting Friday, May 7, 2010
  • 62. Recruiting Same: Good “talkers” Same: Demographic fits Friday, May 7, 2010
  • 63. Recruiting Same: Good “talkers” Same: Demographic fits Different: Relevant current tasks Different: Compatible tech Friday, May 7, 2010
  • 64. Logistics: What & How Provide test target Access Credentials Friday, May 7, 2010
  • 65. Logistics: Who & When How to connect with the user (we’ll discuss in Tools) Managing time zones Session length Friday, May 7, 2010
  • 66. Paying Incentives Collect minimal personal info Deliver quickly Appropriate for location Friday, May 7, 2010
  • 67. Advanced planning: hybrid studies Why use only one method? Some methods don’t tell you the why 3-5 remote interviews is all you need Friday, May 7, 2010
  • 68. Portable Research Friday, May 7, 2010
  • 69. BREAK Friday, May 7, 2010
  • 70. Recruiting for Remote Friday, May 7, 2010
  • 71. We all need people Relevant tasks Compatible tech Good talkers Demographic fits In the moment Friday, May 7, 2010
  • 72. Gold Standard Page 52 Live recruiting Get as close as you can Friday, May 7, 2010
  • 73. Recruiting Options Ethnio Links or house ads to online forms Recruiting agencies Customer email lists Panels Craigslist/online ads Friday, May 7, 2010
  • 74. use Ethnio when Website access Some budget Quotas are relatively simple Friday, May 7, 2010
  • 75. use forms when You can place a link but not code Low budget Quotas are relatively complex Options: Google Docs Wufoo Friday, May 7, 2010
  • 76. Google docs Friday, May 7, 2010
  • 77. use Agencies when No website access Higher budget Online recruiting failed Friday, May 7, 2010
  • 78. Customer lists Sub-segments Ownership requirements No website access Friday, May 7, 2010
  • 79. PANELS Friday, May 7, 2010
  • 80. Craigslist & forums When other methods don’t work No dough Takes a while Embarrassing to admit Friday, May 7, 2010
  • 81. the Gold Standard again Live recruiting Get as close as you can Friday, May 7, 2010
  • 82. Live Recruiting Math (for interviews) Target: 6 recruits/hour Allow for: 1% response (low average) You Need: 8-10,000 uniques/day Friday, May 7, 2010
  • 83. Page 56 Friday, May 7, 2010
  • 84. Screener constraints Page 60-63 Keep it short Different from surveys Who gets paid? Friday, May 7, 2010
  • 85. Remember the Point Tasks that align with research goals Technical compatibility Control demographics Friday, May 7, 2010
  • 86. EXERCISE Friday, May 7, 2010
  • 87. Friday, May 7, 2010
  • 88. Your best questions Friday, May 7, 2010
  • 89. GO EAT LUNCH Famished Friday, May 7, 2010
  • 90. exercise We recommend: Split Pea Seduction, 6th & Minna Custom Burger, 7th & Natoma Tony Baloney’s, 7th & Howard Tin Vietnamese, Howard & 6th Harvest Market, Natoma & 8th Sightglass Coffee, 7th & Folsom Friday, May 7, 2010
  • 91. exercise We recommend: Split Pea Seduction, 6th & Minna Custom Burger, 7th & Natoma Tony Baloney’s, 7th & Howard Tin Vietnamese, Howard & 6th Harvest Market, Natoma & 8th Sightglass Coffee, 7th & Folsom Friday, May 7, 2010
  • 92. Please stand by ESCAPE THE LAB WILL RESUME AT 1:15 Friday, May 7, 2010
  • 93. welcome back Friday, May 7, 2010
  • 94. Time for a contest! Tweet why you hate remote research #ETLAB Win a copy of Silverback Friday, May 7, 2010
  • 95. Remote Research Toolkit MODERATED most of Chapter 8 Friday, May 7, 2010
  • 96. 5 easy pieces Talk to the user See what they’re doing Take notes (optional) Involve observers (optional) Record audio & video (optional) Friday, May 7, 2010
  • 97. Communication phone Skype Reliable Cheap or free Can be expensive Harder for users for international Can combine with Conference calls screensharing to needed for stress user’s observers connection Friday, May 7, 2010
  • 98. Screensharing Standalone Integrated Adobe Connect UserVue (RIP) LiveLook FuzeMeeting (new) WebEx* GoToMeeting *WebEx also has mobile versions Friday, May 7, 2010
  • 99. Adobe Connect Friday, May 7, 2010
  • 100. FuzeMeeting(New) Friday, May 7, 2010
  • 101. Livelook Friday, May 7, 2010
  • 102. Webex Friday, May 7, 2010
  • 103. GoToMeeting Friday, May 7, 2010
  • 104. Taking Notes Second monitor if possible Excel Hack (for time codes) Friday, May 7, 2010
  • 105. How to rig your notes What the formulas look like Auto timestamp! Leave one column (A) empty for notes. Set the cells of the next column (B) to record the clock time of each entry in A. =IF(A2=“”,“”,IF(B2=“”,NOW(),B2)) Set the cells of the next column (C) to record the difference between the first entry and the current entry =IF(ISERROR(B2-$B$2),“”,B2-$B$2) Fill down (Ctrl-D). Expect to use up to 120 rows for a session. Don’t Hide column B. You’ll never need to see it write this down, it’s on page 104 Friday, May 7, 2010
  • 106. Observer Involvement IM Web Chat Rooms Integrated Chat Passing Notes Friday, May 7, 2010
  • 107. Recording Tools Hardware Software Phone Patch Camtasia (Windows) Cables iShowU HD (Mac) Friday, May 7, 2010
  • 108. Phone patch set up Friday, May 7, 2010
  • 109. Friday, May 7, 2010
  • 110. Simple Set-up Friday, May 7, 2010
  • 111. Intermediate Set-up Friday, May 7, 2010
  • 112. Advanced Set-up Friday, May 7, 2010
  • 113. Friday, May 7, 2010
  • 114. exercise Roll your own What will be your base set-up? Write it down Friday, May 7, 2010
  • 115. Friday, May 7, 2010
  • 116. Your version? Friday, May 7, 2010
  • 117. Winners!! Friday, May 7, 2010
  • 118. Moderating for Remote Friday, May 7, 2010
  • 119. Is it really any different? (Short answer: some.) Page 94-99 Friday, May 7, 2010
  • 120. Question from the BP inbox yesterday It would have actually helped to know how many users the authors recommend for a tar* type of test. In a regular user test , many of us have seen how findings repeat a er 5 -8 users of each user type are observed. However, here we don't always know which task a user will be doing when we interrupt them. So in cases where we really are following users along as they complete what they came to do, how many useres should we target ? Given that the authors have been doing this for a while a recommendation from them on how many users to target and their notes on the topic would have been really interesting.  *Time-Aware Research Friday, May 7, 2010
  • 121. Making passionate tasks work Friday, May 7, 2010
  • 122. Making passionate tasks work Recruit for relevant current tasks Script for customizable tasks Friday, May 7, 2010
  • 123. Re-script for remote Friday, May 7, 2010
  • 124. Re-script for remote “Ok, I’d like you to start from the homepage, pretend you are shopping for groceries for a party, and buy some paper plates.” Friday, May 7, 2010
  • 125. MODERATED must-haves Also see 37-43 Friday, May 7, 2010
  • 126. MODERATED must-haves Consent Also see 37-43 Friday, May 7, 2010
  • 127. MODERATED must-haves Consent Also see Tech set-up 37-43 Friday, May 7, 2010
  • 128. MODERATED must-haves Consent Also see Tech set-up 37-43 The beef Friday, May 7, 2010
  • 129. MODERATED must-haves Consent Also see Tech set-up 37-43 The beef De-installation Friday, May 7, 2010
  • 130. MODERATED must-haves Consent Also see Tech set-up 37-43 The beef De-installation Thanks & incentive Friday, May 7, 2010
  • 131. Chances are they’ve never done this before Friday, May 7, 2010
  • 132. What they worry about Friday, May 7, 2010
  • 133. Who are you and can I trust you? Friday, May 7, 2010
  • 134. If you can see my screen, can you see my face? Friday, May 7, 2010
  • 135. Will you be able to take control of my computer? Friday, May 7, 2010
  • 136. Will you help me get the plug-in off my computer at the end? Friday, May 7, 2010
  • 137. Will this work if I have a firewall/proxy/strict IT department? Friday, May 7, 2010
  • 138. Is this going to end up on YouTube? Friday, May 7, 2010
  • 139. Will I really get the incentive if I do this? Friday, May 7, 2010
  • 140. How long will it take? Friday, May 7, 2010
  • 141. Is it OK if I have to take a quick call in the middle? Friday, May 7, 2010
  • 142. Is it OK if my kids/dog/fratmates are making noise in the background? Friday, May 7, 2010
  • 143. OXYMORON? Remote empathy Friday, May 7, 2010
  • 144. Phone Interviewing Specifics Friday, May 7, 2010
  • 145. Things sound leading Pitfalls Friday, May 7, 2010
  • 146. Body language is huge Actual language is important too Make sure you’re comfortable Friday, May 7, 2010
  • 147. EXERCISE Practice screensharing Moderator, Participant, at least one observer Use the script in your packet as a starting point Get your participant all set up and ready to start an interview Friday, May 7, 2010
  • 148. Friday, May 7, 2010
  • 149. How did it go? Friday, May 7, 2010
  • 150. Advanced Techniques High Emotions Friday, May 7, 2010
  • 151. Advanced Techniques One to many moderation Multi-threading Friday, May 7, 2010
  • 152. Future Frontiers Friday, May 7, 2010
  • 153. BREAK Friday, May 7, 2010
  • 154. Remote Research Toolkit UNMODERATED Friday, May 7, 2010
  • 155. 3 parts Provide test target Record user behavior Capture comments (optional) Friday, May 7, 2010
  • 156. AUTOMATED Tools Friday, May 7, 2010
  • 157. What do you want to do? Get freeform data cheaply Quantify user behaviors Conceptual & IA testing Friday, May 7, 2010
  • 158. This stuff is pretty cheap • Usabilla $0-950/yr • Optimal Workshop $559-1350/yr • Loop11 $350/study • OpenHallway $19-199/mo • UserTesting.com $39/user • Feedback Army $10/10 users Friday, May 7, 2010
  • 159. UserTesting.com Friday, May 7, 2010
  • 160. UserTesting.com Friday, May 7, 2010
  • 161. UserTesting.com Friday, May 7, 2010
  • 162. UserTesting.com Friday, May 7, 2010
  • 163. Using your own recruits Set up an account Email support@usertesting.com with: the URL where your target lives the scenario (500 char limit) tasks (2000 char limit) up to 4 questions to be answered in writing at the end Request a UserTesting URL for your study Friday, May 7, 2010
  • 164. Loop11 Friday, May 7, 2010
  • 165. Loop11 Friday, May 7, 2010
  • 166. Loop11 Friday, May 7, 2010
  • 167. Loop11 Friday, May 7, 2010
  • 168. Loop11 Friday, May 7, 2010
  • 169. Usabilla Friday, May 7, 2010
  • 170. Optimal Workshop Treejack Chalkmark Friday, May 7, 2010
  • 171. EXERCISE Choose one of these tools Sign up for a test account Create a 15-minute test for any website you like Friday, May 7, 2010
  • 172. Friday, May 7, 2010
  • 173. RESULTS Friday, May 7, 2010
  • 174. Session Videos Heatmaps Notes Completion Rates Card Sort Groups Lions, Tigers, Bears... Friday, May 7, 2010
  • 175. Telling a story If you have time-coded notes, you can make highlight videos If you have user comments, you can do tag analysis or make word clouds If you have numbers, you can compare to benchmarks If you have card groups you can do cluster analysis Friday, May 7, 2010
  • 176. Case Study: Data Mismatch Friday, May 7, 2010
  • 177. Qualitative Results 16 remote interviews Worst findability BP has ever observed Recommendation to burn it down and start over Friday, May 7, 2010
  • 178. Quantitative Data EFFECTIVENESS RATIOS EFFICIENCY Your task is to make an Success users Only appointment with a pediatric Time (mm:ss) arrhythmia Mean time per task 01:36 specialist at Standard deviation 00:47 Children’s Hospital. Once you have found it, please remember Clicks or write down the 6 phone number you Mean clicks per task (exact value: 6) would call to make 4 an appointment by Standard deviation (exact value: 4.4) phone, as you will be asked for that information later on. Answers distribution ??? ! The phone number search has been... *[1=Extremely difficult / 7=Extremely easy] 1 2% (1) 2 2% (1) 3 10% (5) 4 18% (9) 5 18% (9) 6 24% (12) 7 26% (13) N 50 Mean* 5.2 Friday, May 7, 2010
  • 179. What happened? Friday, May 7, 2010
  • 180. 32 page report, distilled Navigation Search ease Hospital ease of use of use reputation 1 2 3 4 5 6 7 terrible neutral UserZoom’s red excellent flag minimum Friday, May 7, 2010
  • 181. What’s different IRL? Friday, May 7, 2010
  • 182. Recording 1 more button to press Friday, May 7, 2010
  • 183. Observer input One more stream on your screen (Possibly 2 or 3) Friday, May 7, 2010
  • 184. Notes Tap, tap, tapping away Friday, May 7, 2010
  • 185. So, what could go wrong? Friday, May 7, 2010
  • 186. 3 points of failure Technology yours participant’s observer’s Recruiting Test Design Friday, May 7, 2010
  • 187. Control what you can control Which is, your set up. You wouldn’t go backpacking without checking your gear. Do it 24 hours in advance. Or else. Page 240-241 Friday, May 7, 2010
  • 188. Prepare for what you can’t control User’s dog trips over power cord Internet affected by local weather User can’t install Doesn’t know how to install Friday, May 7, 2010
  • 189. Mid-Session Tech Problems Friday, May 7, 2010
  • 190. Live Recruiting Problems What if they gave a study and nobody came? Friday, May 7, 2010
  • 191. Increase exposure When you’re not getting enough views When the wrong type of users are responding Friday, May 7, 2010
  • 192. Increase incentives When it’s cheaper When you’re almost getting enough When segments are tough Friday, May 7, 2010
  • 193. Switch methods When base traffic level isn’t high enough Friday, May 7, 2010
  • 194. Script Problems Cut the BS. Friday, May 7, 2010
  • 195. The bottom line If they’re not engaged, something’s wrong. Friday, May 7, 2010
  • 196. Burn the script Find out what matters. Friday, May 7, 2010
  • 197. QUESTIONS Friday, May 7, 2010
  • 198. @boltpeters CYD@BOLTPETERS.COM NATE@BOLTPETERS.COM Friday, May 7, 2010
  • 199. kthxbai. Friday, May 7, 2010

×