Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

UXPA2019 Scaling research without scaling the team

122 views

Published on


Digital product teams are moving faster and the desire to learn directly from users is increasing. When research headcount hasn’t caught up, others in the organization are taking matters into their own hands. How can we shift our role from the research bottleneck in the product development process to the coaches who empower more decision-makers to speak to users directly?
We’ll share the strategy we took to successfully scale the amount and impact of research happening at a fast-growing medium-sized startup as a team of three researchers.

You’ll learn how we:

Empowered stakeholders to answer their own tactical questions through research training workshops and self-serve resources.
Initiated a knowledge base that allows stakeholders to synthesize fresh insights from across past studies, increasing the lifespan of past and new research findings.
Slowed down and invested in research operations in order to move faster later.

  • Be the first to comment

UXPA2019 Scaling research without scaling the team

  1. 1. SCALING RESEARCH WITHOUT SCALING THE TEAM Cordelia Hyland | Jordan Berry | Erik Olesund Thumbtack UXPA 2019
  2. 2. Why did you choose this talk?
  3. 3. Let’s begin with a story.
  4. 4. Research team Thumbtack
  5. 5. SOLO SMALL BIG
  6. 6. Erik Olesund Leads research on how customers search and discover the right pros to work with Cordelia Hyland Leads research that helps pros manage the services they offer and their conversations with customers Experience Researchers at Thumbtack Jordan Berry Leads research that helps pros match with the right customers
  7. 7. SCALING RESEARCH WITHOUT SCALING THE TEAM 1 2 3 Help each researcher achieve more by building scalable processes for repeat activities Increase the value and relevance of existing research by making past insights easy to discover Make our colleagues stronger research partners through trainings and coaching
  8. 8. Our Pros Hundreds of thousands of Pros available in every county in the U.S. Top categories 1. Handyman 2. General Contractor 3. Cleaner 4. Painter 5. Photographer Our Customers Millions of customers each year, starting projects across nearly 500 categories. Largest regional markets Atlanta, Washington DC, Denver, New York, Virginia Beach
  9. 9. Scale the researchers How we built research processes that scale the reach and impact of ourselves 1
  10. 10. Ways we’ve scaled the researchers ● Team branding ● Templates ● The pro online community ● Partnered with customer support for phone surveys
  11. 11. ● Stickers! ● Slack emojis ● Custom report templates Team branding
  12. 12. Templates ● Study plans ● Reports ● Observer note taking doc ● Group debrief framework
  13. 13. Framework for facilitating group synthesis
  14. 14. Thumbtack Community
  15. 15. In review ● Build a team brand ● Create templates ● Leverage your customer forums
  16. 16. Partnered with customer support for phone surveys
  17. 17. The tiger team
  18. 18. Who are they?
  19. 19. What do they do?
  20. 20. Why did a professional recently churn? Why did a professional decide that a lead wasn’t the right fit? What are the types of questions we ask? 1 2
  21. 21. How do we analyze the data? 1 2 Use a form that does double duty—as a script and call notes Analyze the calls as you would a normal study
  22. 22. How does this help scale our research? ● Research happens in the background, concurrent with other studies ● Scales non-researchers, too
  23. 23. If you choose to do this ● Be sensitive towards the team’s time ● Provide real-time feedback to agents ● Clarify the purpose of the call for the user
  24. 24. What are existing resources you can leverage?
  25. 25. SCALING RESEARCH WITHOUT SCALING THE TEAM 1 2 3 Help each researcher achieve more by building scalable processes for repeat activities Increase the value and relevance of existing research by making past insights easy to discover Make our colleagues stronger research partners through trainings and coaching
  26. 26. Scale existing research How to make past insights discoverable and relevant today 2
  27. 27. More research. More data.
  28. 28. “Where is that video from the customer we met in New York?”
  29. 29. “Didn’t we ask a similar question in your study two months ago?”
  30. 30. “Can you send me any research reports we have about X?”
  31. 31. How do we make past insights discoverable and relevant today?
  32. 32. What we did.
  33. 33. Is this the right approach for us? Who is going to populate the database? How do we develop a consistent yet flexible tagging taxonomy? How do we determine what qualifies as an insight and what doesn’t? Will our research partners be able to find the relevant “nuggets”? What about participant privacy?
  34. 34. Make insights discoverable.
  35. 35. Make insights discoverable. For ourselves.
  36. 36. Make insights discoverable. For ourselves. And for our teams.
  37. 37. For ourselves For our teams +
  38. 38. Data repository ● Store recordings of qualitative research ● Automatic transcription ● Tag and search
  39. 39. “What do we know about [x]?” How are pros managing their budgets? How do customers use reviews? What features do pros value most? How are customers searching? How is [x] feature perceived?
  40. 40. The Research Archive makes past insights discoverable to the rest of Thumbtack
  41. 41. Overview of completed research projects Each entry has ● Link to report ● Keywords ● Metadata Anyone can filter/search the archive
  42. 42. Impact Just in time answers.
  43. 43. One tool doesn’t rule them all Organizing your assets are worth the hassle Make it easy to search 1 2 3 What we learned from making past insights easy to discover
  44. 44. What are you doing to keep past research insights alive?
  45. 45. SCALING RESEARCH WITHOUT SCALING THE TEAM 1 2 3 Help each researcher achieve more by building scalable processes for repeat activities Increase the value and relevance of existing research by making past insights easy to discover Make our colleagues stronger research partners through trainings and coaching
  46. 46. Scaling the contribution of others Making our colleagues stronger research partners through trainings and coaching 3
  47. 47. The challenge.
  48. 48. Execute ongoing research to support product teams Conduct proactive, strategic research Research operations Scale research What does the experience research team do?
  49. 49. Execute ongoing research to support product teams Conduct proactive, strategic research Research operations Scale research What does the experience research team do? Where we wanted to be
  50. 50. Execute ongoing research to support product teams Conduct proactive, strategic research Research operations Scale research What does the experience research team do? Where we wanted to be What we needed to do
  51. 51. How do we raise the quality bar of all research that happens at Thumbtack? 1
  52. 52. How do we help non- researchers know when to do their own research and when to partner with research? 2
  53. 53. TRAINING PROGRAM
  54. 54. workshops TRAINING PROGRAM
  55. 55. workshops 1:1 coaching TRAINING PROGRAM
  56. 56. Workshop content ● What research is (and isn’t) ● How to work with us ● Moderating practice ● Cognitive biases
  57. 57. Cognitive biases 101
  58. 58. Biases that impact our interpretation of what we heard Confirmation bias Clustering illusion Recency bias Biases that impact how the participant responds Observer-expectancy Social desirability False recall
  59. 59. Moderating practice research question ≠ interview question[ ]
  60. 60. Interview a participant to better understand behavior around a specific action 1 moderator Be yourself and answer truthfully 1 participant Use the checklist to analyze how well the moderator is doing and provide feedback 1 observer ACTIVITY OVERVIEW
  61. 61. Interview your participant to understand more about their sleep. You want to understand their habits, values, and how they fall asleep in order to inform the design and messaging of a new sleep assistant app. PROMPT
  62. 62. What we learned.
  63. 63. Break them down to build them up
  64. 64. Biases are sticky, really sticky
  65. 65. “I noticed that I often ask leading questions and fall into the trap of confirmation bias. So this was extremely helpful for me to be more aware of! – workshop participant
  66. 66. Cater the content to the audience
  67. 67. Some training is best left for 1:1 coaching
  68. 68. The impact.
  69. 69. 180 participants 4 teams 14 workshops
  70. 70. 67 studies across Thumbtack In the last eight months 33 researcher-led 34 researcher-advised 21 individual non-researchers with a tiny team of researchers plus guerrilla research, product feedback at monthly events, customer advisory council calls, community forum posts, tiger team calls
  71. 71. “User research is key to making a great product and I love that the UXR team is empowering teams to conduct their own research and welcoming opportunities to partner on studies. – workshop participant
  72. 72. SCALING RESEARCH WITHOUT SCALING THE TEAM 1 2 3 Help each researcher achieve more by building scalable processes for repeat activities Increase the value and relevance of existing research by making past insights easy to discover Make our colleagues stronger research partners through trainings and coaching
  73. 73. THANK YOU
  74. 74. What’s working for you to scale research?
  75. 75. Q&A Jordan Berry jordanb@thumbtack.com @jordanberry Cordelia Hyland cordelia@thumbtack.com @cordelia_hyland Erik Olesund erikolesund@thumbtack.com @olesund
  76. 76. You? + +
  77. 77. We need help feeding the Thumbtack monster!
  78. 78. YES, WE’RE SCALING THE TEAM Interested in joining us? bit.ly/thumbtackresearchishiring or scan the QR code Jordan Berry jordanb@thumbtack.com @jordanberry Cordelia Hyland cordelia@thumbtack.com @cordelia_hyland Erik Olesund erikolesund@thumbtack.com @olesund
  79. 79. EXAMPLE SLIDES BELOW
  80. 80. SOCIALIZING
  81. 81. 67 studies across Thumbtack In the last eight months,
  82. 82. 67 studies across Thumbtack In the last eight months, 33 researcher-led
  83. 83. 67 studies across Thumbtack In the last eight months, 33 researcher-led 34 researcher-advised
  84. 84. 67 studies across Thumbtack In the last eight months, 33 researcher-led 34 researcher-advised 21 individual non-researchers
  85. 85. 67 studies across Thumbtack In the last eight months, 33 researcher-led 34 researcher-advised 21 individual non-researchers plus guerrilla research, product feedback at monthly events, customer advisory council calls, community forum posts, tiger team calls
  86. 86. +
  87. 87. How do we scale the impact of research without scaling the team?
  88. 88. 1. Summarizing a program of research 2. Documenting State of X 3. Reframing to answer specific questions
  89. 89. Take the time to compare findings to past studies. 1
  90. 90. IF A BOTTLE FELL ON THE FLOOR AND NO ONE WAS AROUND TO HEAR IT, DID IT MAKE A SOUND? Image source: Time Magazine
  91. 91. SCALING RESEARCH WITHOUT SCALING THE TEAM Cordelia Hyland | Jordan Berry | Erik Olesund Thumbtack UXPA 2019
  92. 92. ● Org/leadership ○ Promise to move faster ○ Empower their team members to answer their own questions ○ Trainings curated by team and their needs ● IC teammates ○ Referencing current questions their team is asking ○ Show buy-in from leadership How did we get buy-in for the trainings?
  93. 93. FAST SLOW
  94. 94. PACE IMPACT

×