Your SlideShare is downloading. ×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Best Practices on Recruiting Participants for Remote User Testing

1,572
views

Published on

Recruiting the right participants is often the most difficult and challenging aspect of conducting research studies. A well-managed recruitment process can be pivotal to the success of achieving your …

Recruiting the right participants is often the most difficult and challenging aspect of conducting research studies. A well-managed recruitment process can be pivotal to the success of achieving your research objectives.

Join JonDelina (“JD”) Buckley, Manager of User Experience Research at Kelley Blue Book, and Aaron Rager, Client Development Manager at Research Now, to learn about some of the best practices on recruiting the right participants for remote unmoderated user testing.

Published in: Technology, Business

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,572
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
11
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Alfonso speak.
  • .
  • Phrase questions in a general way, and then look for specific responses. (instead of saying “How often does the advertising on XYZ website confuse you?” and then providing a list of options such as frequently, sometimes or never, it may be better to include questions like “What ONE thing about your experience on XYZ website would you change?” and then provide a list of several options including an “advertising” option
  • As mentioned each panelist has to undergo extensive screening (vs. 300+ screening criteria) prior to joining our panels.We also employ Dynamic profile enrichment—allowing us to continually engage respondents to update their profile due to “lifestyle” changes.Our incentive practices are routed in what makes the respondent most happy (as they are given a choice to select from) and range from loyalty points, frequent flyer miles and/or cash.Our respondent are paid for their time—even if they don’t qualify for a survey. This is a primary driver of our overall value, retention and response rates.We limit the number of surveys a respondent can take—which downsizes fatigue/burnout and again, drives retention. (We’ve the ability to change this based upon categorical research needs.)To ensure respondent happiness /eliminate frustrations, we also employ “seemless” routing systems—sending a respondent seemlessly from a survey they may not qualify for to another. This allows us not to burn through sample, which maximizes our panel use and keep our costs at bay.
  • Transcript

    • 1. Webinar: Best Practices on Recruiting Participants for Remote User Testing JonDelina “JD” Buckley, Kelley Blue Book Aaron Rager, Research Now Alfonso de la Nuez, UserZoom #uzwebinar
    • 2. Speakers: JD Buckley Manager of User Experience Kelley Blue Book Speaker Aaron Rager Client Development Sales Manager Research Now Speaker Alfonso de la Nuez Co-Founder and Co-CEO UserZoom Moderator
    • 3. Quick Housekeeping • Chat box is available if you have any questions • There will be time for Q&A at the end • We will be recording the webinar for future viewing • All attendees will receive a copy of the slides/recording • Twitter hashtag: #uzwebinar www.userzoom.com
    • 4. Agenda for Today 1. Introductions 2. Recruiting for Remote User Research JD Buckley, Kelley Blue Book 3. Capabilities Overview Aaron Rager, Research Now 4. Q&A www.userzoom.com
    • 5. About UserZoom  All-in-one Enterprise software solution that helps Businesses costeffectively test, measure and improve UX over websites & mobile apps.  Offer online or remote user research & testing solutions, saving UXers time, money, effort, and a lot of actionable insights  UX Consultants since ‟01, SaaS since „09 Product Suite:  Unmoderated Remote Usability Testing  In Sunnyvale (CA) Manchester (UK), Munich (DE) and Barcelona (Spain)  Remote Mobile Usability Testing  Online Surveys (web & mobile)  Online Card Sorting  90% renewal rate, 50% revenue growth rate in the last 3 years  Tree Testing  Screenshot Click Testing  Screenshot Timeout Testing (5-sec test)  Web VOC  Mobile VOC www.userzoom.com
    • 6. Recruiting For Remote User Research JD Buckley Manager, User Experience Research Kelley Blue Book
    • 7. Agenda • Who I Am & What I Do • A Valuable Tool In The Researchers Toolkit • Types of Remote User Research • Recruiting Options • Determining Your Research Participants • Developing a Screener – Best Practices • Tips and Tricks for Remote Recruiting
    • 8. Agenda • Who I Am & What I Do • A Valuable Tool In The Researchers Toolkit • Types of Remote User Research • Recruiting Options • Determining Your Research Participants • Developing a Screener – Best Practices • Tips and Tricks for Remote Recruiting
    • 9. Who I Am, What I Do JD Buckley – Manager of User Experience Research – Kelley Blue Book Education •BA – Communications/Marketing •MA – Human Factors Psychology Companies •Ad Agencies - J. Walter Thompson, Lord Dentsu & Partners, Dailey & Assoc’s,TBWAChiat/Day •Technology Companies – DirecTV, Yahoo! •Technology Incubators & Startups – Idealab, Dank-Muller & Co, InsiderPages.com, Picassa.com, Evolution Robotics, Omnilux •Health and Medical – Kaiser, Herbalife •Automotive – Kelley Blue Book
    • 10. Agenda • Who I Am & What I Do • A Valuable Tool In The Researchers Toolkit • Types of Remote User Research • Recruiting Options • Determining Your Research Participants • Developing a Screener – Best Practices • Tips and Tricks for Remote Recruiting
    • 11. User research has traditionally been conducted like this
    • 12. User Researchers have traditionally recruited like this uxforthemasses.com
    • 13. In the last several years, remote user research has become a popular addition to the researchers toolkit: Benefits include: • Opportunity to collect data from a larger sample size compared to in-lab research • Potential to reach special audiences not immediately available • Ability to create a diverse test panel with broader geographical distribution • Ability to simulate more realistic behavior in real-world context • Allows collection of reliable quantitative as well as qualitative data
    • 14. Agenda • Who I Am & What I Do • Valuable Tool In The Researchers Toolkit • Types of Remote User Research • Recruiting For Remote User Research • Determining Your Research Participants • Developing a Screener – Best Practices • Tips and Tricks for Remote Recruiting
    • 15. Moderated Remote Research: Participants perform certain tasks remotely and the researcher observes and asks questions from another location using a screen-sharing application http://www.flickr.com/photos/boltpeters/3918152527/
    • 16. Un-moderated/Automated Remote Research: Participants perform certain tasks and answer survey-like questions, researcher and participant are separated over time and space Userzoom.com
    • 17. Remote Self-Moderated Participant perform tasks while their interactions and audio are recorded as they talk “aloud” without a moderator present Usertesting.com
    • 18. Diary Studies Participants chronicle their daily activities or interactions with a product for a specific period of time, and share that information remotely with a researcher http://www.hcii.cmu.edu/M-HCI/2011/BOA-PlanningTools/
    • 19. Agenda • Who I Am & What I Do • Valuable Tool In The Researchers Toolkit • Types of Remote User Research • Recruiting Options • Determining Your Research Participants • Developing a Screener – Best Practices • Tips and Tricks for Remote Recruiting
    • 20. Similar to traditional in-lab research, user researchers have two options for remote recruiting: 1. Have someone else do it – (e.g. recruitment companies) 2. Do it yourself – (e.g., recruit in house)
    • 21. External recruiters can provide multiple options for remote recruiting Intercept intercept participants from desktop or mobile websites Keynote.com Company Email List Research Panel Custom Recruit Invite participants from opt-in email list leverage panel of individuals who represent a specific audience create a custom panel of individuals with targeted profiles
    • 22. External Recruiting – Pros and Cons Pros Cons • Frees the researcher to focus on the study • Can be more expensive than self recruiting • Can leverage multiple sources to access your target audience • Requires coordination with the external recruitment agency • Handle incentives and quota fulfillment • Study design complexity and recruiter involvement may increase costs • May help with screener development • Quality of panelists data differ across panels & companies
    • 23. Self-recruiting can offer some similar and additional options Intercept intercept participants from desktop or mobile websites Keynote.com Company Email List Invite participants from opt-in or rented email list Social Media Online Posting leverage social media options to source target audience Post an ad on a blog, classifieds site, newsletter or via SEM
    • 24. Self-recruiting – Pros and Cons Pros • Can be less expensive than using an external resource • Allows the researchers to control the process and/or quality of participants Cons • Can be time consuming • Requires strong project management/organization skills from the researcher • Limited options/back-up if anything goes wrong
    • 25. Recruitment Method and Remote Research Type Cheat Sheet Remote Moderated Remote Unmoderated Remote Self Moderated Diary Studies Intercept Email Lists Intercept Research Research Type Types Email Lists Custom Social Research Recruit Media Panel Recruitment Methods Custom Recruit Social Media Research Panel Online Postings Online Postings
    • 26. Agenda • Who I Am & What I Do • A Valuable Tool In The UX Researchers Toolkit • Types of Remote User Research • Recruiting Options • Determining Your Research Participants • Developing a Screener – Best Practices • Tips and Tricks for Remote Recruiting
    • 27. Every product or service has a target audience
    • 28. You need to understand the experience of the people who are actually going to want to use, understand and buy your product. Anyone else’s experience will be of marginal use, or even deceptive. Observing the User Experience, 2nd Edition – Goodman, Kuniavsky and Moed
    • 29. 90percentofeverything.com
    • 30. Whether you‟ve decided to recruit for your research by using segments, profiles or persona‟s, you should have a clear idea of:
    • 31. Demographics data: Includes physical characteristics (age, gender, ethnicity) employment, education and household income information Freewaregenius.com/quantcast.com
    • 32. Channel Usage: (Web, Mobile or Tablet) http://www.lukew.com/ff/entry.asp?1436
    • 33. Technology Usage Google-Multiscreenworld
    • 34. The presence or absence of key behaviors, needs and triggers that help you narrow in on your appropriate research audience Melissa - 33, married, 3 children “After the accident, we decided to find something very safe with room for our family.” • • • • Car for family Other people’s opinions Influencer / Influenced Urgent need
    • 35. Agenda • Who I Am & What I Do • Valuable Tool In The Researchers Toolkit • Types of Remote User Research • Recruiting Options • Determining Your Research Participants • Developing a Screener – Best Practices • Tips and Tricks for Remote Recruiting
    • 36. Essentially, a screener is a script that allows you to filter the general population to an audience that reflects your specific criteria
    • 37. The screener is probably the most important part of the recruitment process. It’s a script that filters out people who will give you good responses from the ones who merely match your basic criteria. Observing The User Experience 2nd Edition – Goodman, Kuniavsky, Moed
    • 38. Screener Best Practices: Limit to 20 questions and a completion time of 10 minutes max
    • 39. QX. Within the next 2 weeks, how likely are you to purchase coco pops? Extremely Likely A COMPLETE: PLACE IN GROUP Somewhat Likely B DO NOT RECRUIT Not very likely C DO NOT RECRUIT Not sure D DO NOT RECRUIT Screener Best Practices: Start with the questions that will screen people out
    • 40. Screener Best Practices: Every question should have a purpose
    • 41. Screener Best Practices: Be specific, clear and never use jargon
    • 42. 1/15/12 4:42am 6lbs 14oz Screener Best Practices: Ask for exact dates, times, and quantities
    • 43. Screener Best Practices: Make sure your questions are not leading or biased
    • 44. Example: Email Invite To Screener Your thoughts are important us! We would like to invite you to participate in a website evaluation that will take approximately 20 - 30 minutes. This website evaluation is different from other surveys you may have taken in the past. You will be asked to view designated website(s), perform simple tasks, and report your experience with the site(s). Your opinions and experiences are very important, and will be used for future website development efforts. Screener Best Practices: Clearly state the format of the research
    • 45. Screener Best Practices: Build in flexibility
    • 46. Agenda • Who I Am & What I Do • Valuable Tool In The UX Researchers Toolkit • Types of Remote User Research • Recruiting Options • Determining Your Target Audience • Developing a Screener • Tips and Tricks for Remote Recruiting
    • 47. Tips and Tricks for Remote Research Recruiting • Clarify the research objectives, target audience, metrics for success, budget and timeline • Use the Right Tool • Recruit based on behavior first; demographic criteria second • Be careful not to back yourself into a corner by “over-determining” your target audience • Give people a reason to participate in your research beyond the incentive • Over recruit if possible • Make mistakes and learn from them
    • 48. References Goodman, E., Kuniavsky, M., Moed, A., Observing the User Experience, A Practitioner’s Guide to User Research, Elsevier, Morgan Kaufmann, 2012, ISBN: 978-0-12-384869-7 Albert, B., Tedesco, D., Tullis, T., Beyond the Usability Lab, Conducting Large-Scale Online User Experience Studies, Elsevier, Morgan Kaufmann, 2010
    • 49. A Capabilities Overview Research Now social. mobile. online. Aaron Rager Client Development Manager June 6, 2013
    • 50. Who We Are Research Now is the leading provider of global market research services.  Experienced. Market leader in powering the insights industry, completing 2M+ interviews across 3000+ projects each month  Innovative. Advanced digital research services including Social, Mobile, and Online  Leading. Proprietary technologies and the largest online panels in the industry: 6M+ members across 38 countries  Reliable. Over 2000 clients globally  Consultative. Recognized as #1 in customer service annually since 2007*  Worldwide. 1200+ employees in over 20 cities across four continents, with offices in Dallas, New York, Los Angeles, Toronto, San Francisco, Chicago, London, Paris, Frankfurt, Hamburg, Athens, Madrid, Dubai, Sydney, Melbourne, Auckland, Shanghai, Singapore, New Delhi  Trusted. Work closely with CASRO, ESOMAR, and the MRA 50 *marketresearchcareers.com
    • 51. Partial Client List 51
    • 52. Our Panel Recruitment Methodology “By-Invitation-Only” Online Panel Recruitment – 52 A method of inviting only pre-validated individuals to enroll into a market research panel through a controlled process.
    • 53. Research Now: Global Reach 53
    • 54. On-Going Panel Quality Management Optimizing the panel member experience • Robust panel profiling – 300+ profiling attributes • Socio-demographic, life stage, interests and hobbies, shopping habits, purchase intent, etc. • • Dynamic Profile Enrichment® Business Panel - 40+ firmographic attributes » Industry type, annual revenue, title, functional role, purchasing role, etc. • Proven incentive methodology – Pay-all – Fair-value exchange – Respondent reward choice – Cash & Online currency • Managed Participation Levels – Survey frequency limiters – Member managed invitation maximums • Intelligent Respondent Routing – Scout & Sample Deployment (SDS) Systems – Completed surveys routed to update profile – Survey terminates routed to different survey Approximately 80% Member Retention rate since inception • 54
    • 55. Example of Panel Reach USA (adults 18+) Grand Total Sub Total Adults 18 to 34 18 to 24 25 to 34 Sub Total Adults 35 to 54 35 to 44 45 to 54 Sub Total Adults 55 to 64 55 to 64 Sub Total 65+ 65+ 55 © 2012 Research Now TOTAL 2,074,994 670,745 182,848 487,897 870,673 439,254 431,419 360,701 360,701 172,875 172,875 Dated Panel Counts North East Mid-West South 404,613 526,530 700,359 136,131 164,756 225,382 40,551 44,747 59,672 95,580 120,009 165,710 165,496 224,270 299,671 80,060 111,069 155,240 85,436 113,201 144,431 71,356 96,236 115,935 71,356 96,236 115,935 31,630 41,268 59,371 31,630 41,268 59,371 West 443,492 144,476 37,878 106,598 181,236 92,885 88,351 77,174 77,174 40,606 40,606 Estimated Completes @ 50% Incidence 15 minute Study TOTAL North East Mid-West South West 103,750 20,231 26,327 35,018 22,175 33,537 6,807 8,238 11,269 7,224 9,142 2,028 2,237 2,984 1,894 24,395 4,779 6,000 8,286 5,330 43,534 8,275 11,214 14,984 9,062 21,963 4,003 5,553 7,762 4,644 21,571 4,272 5,660 7,222 4,418 18,035 3,568 4,812 5,797 3,859 18,035 3,568 4,812 5,797 3,859 8,644 1,582 2,063 2,969 2,030 8,644 1,582 2,063 2,969 2,030
    • 56. Expertise in Mobile Data Collection The Leader in Mobile Engagement For Research Currently reaching and engaging mobile users in 12 key ways:  Push notification  Geo fencing  Geo targeting  Behavioral targeting based on usage of:  apps, content, music, camera, browser, phone and text  Venue trigger (foursquare integration)  QR code trigger  Diary studies - calendar trigger  Self-opted via short code SMS  Interviewer mode for F2F / CAPI  Email invites  SMS invites  In-app intercepts & mobile web intercept 56 © 2012 Research Now
    • 57. http://www.linkedin.com/company/userzoom https://www.facebook.com/userzoom @userzoom Questions & Answers JD Buckley Aaron Rager Alfonso de la Nuez Manager of User Experience Kelley Blue Book Client Development Sales Manager Research Now Arager@researchnow.com 214-365-3451 Co-Founder and Co-CEO UserZoom